Absolute position embedding

This note is a summary of Master Positional Encoding: Part I

$$\begin{array}{r}P{E}_{(pos,2i)}=sin(pos/10000^{2i/d_{\mathrm{model}}})\\ P{E}_{(pos,2i+1)}=cos(pos/10000^{2i/d_{\mathrm{model}}})\end{array}$$

Think about the position encoder first as our normal binary encoding: 000, 001, 010, 011, etc. That’s not continuous. It cannot be interpolated, and has jump. So that’s why there is sin there. Why do we have both cosine and sin though? That’s to make sure there’s a nice property:

$$\mathrm{PE}(x+\Delta x)=\mathrm{PE}(x)\cdot \mathbf{T}(\Delta x)$$

and with both sin and cos, we have this:

$$(\begin{array}{c}\cos (\theta +\varphi )\\ \sin (\theta +\varphi )\end{array})=\left(\begin{array}{cc}\cos \varphi & -\sin \varphi \\ \sin \varphi & \cos \varphi \end{array}\right)\left(\begin{array}{c}\cos \theta \\ \sin \theta \end{array}\right)$$

and we can find such a $T(\Delta x)$ as

$$\begin{array}{rl}{\mathbf{v}}^{(i)}& =[\cos ({\omega }_0{x}_i),\sin ({\omega }_0{x}_i),\dots ,\cos ({\omega }_{n-1}{x}_i),\sin ({\omega }_{n-1}{x}_i)]\\ T(\Delta x)& =\left(\begin{array}{ccc}\left[\begin{array}{cc}\cos ({\omega }_0\Delta x)& -\sin ({\omega }_0\Delta x)\\ \sin ({\omega }_0\Delta x)& \cos ({\omega }_0\Delta x)\end{array}\right]& \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & \left[\begin{array}{cc}\cos ({\omega }_{n-1}\Delta x)& -\sin ({\omega }_{n-1}\Delta x)\\ \sin ({\omega }_{n-1}\Delta x)& \cos ({\omega }_{n-1}\Delta x)\end{array}\right]\end{array}\right)\end{array}$$