This article is only about understanding the implementation of positional embeddings, but to really understand positional embeddings you have to understand the theory. Positional embeddings are group representations. This is how you find them, and this is also how you parameterize them to make them differentiable/learnable. To give a few examples:
1. Rotary positional embeddings are representations of the translation group. The n-dimensional translation group ℝⁿ is abelian, so its irreducible representations are all one-dimensional ρ: ℝⁿ → ℂ, specified by a momentum vector m: ρₘ(x) = exp(i⟨m, x⟩). You can get a multi-dimensional representation by choosing several m, which are your learnable parameters.
2. If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant).
3. If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to [0, 1]. Moving up/down/to a neighbor in the quadtree requires an affine transformation of the local coordinates, which compose, so we can label every node by its affine transformation to the root (x, d): y → x + 2⁻ᵈy, where x is the node's translation in the image and d is its depth in the tree. The finite-dimensional irreducible representations are parameterized by momentum vectors m as well as a modulus M and basis vectors {e₁, e₂, ..., eₖ} where k = ord₂(M) and look like ρₘ,ₑ(x, d) = DPᵈ where D is the diagonal matrix with entries Dⱼⱼ = exp(2πi⟨2ʲm, x⟩ / M) and P permutes the basis vectors.
As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks.
From what I usually see, ML researches just throw everything at the wall and see what sticks. Whatever approach turns out to work better, gets used more often, and sometimes theoretical background is developed for why it works. RoPE works not because it’s good math, just it turned out to work better than some other equally arbitrary approaches at some point, and then other approaches have also been developed that work better than RoPE.
Most people who publish papers in ML are alchemists, but please do not assume all of us are. The original authors of the RoPE paper knowingly designed their embedding to be translation invariant. I do not know if they were aware it was a group representation, but what they were trying to design was in essence a group representation.
I dont think the target audience for the blog is abstract algebra grad students
However would be good to add that the solution can be structural(like in NoPE), then you dont need explicit representations and it just falls out of your architecture.
When I see a title "Understanding..." I assume it will be about understanding theory not implementation or the theory of implementation. When I see a title "Surveying..." or "Understanding Implementation..." I assume it will be about surveying the implementation of that theory. There was just a title mismatch so I felt it was prudent to fill in the gap in understanding the blog did not cover.
Also, I expect the target audience of the blog is ML researchers, since he goes into some of the math, and umm... the kind of ML researcher that is not at least aware of the basics of abstract algebra is closer to an alchemist than a researcher.
At the quantum level, nature seems to use the half spin ireps, but I fail to see if they are useful for 2D images or maybe 3D shapes as inputs for LLMs. Any ideas?
It is called representation theory, and once you have seen even the basics of representation theory it is obvious the authors of the original RoPE paper were trying to rederive a representation for the translation group.
The reason you want to know the general theory behind positional embeddings is it lets you work with other kinds of relative positions. For example, quad/octtrees are useful for faster attention on large images/videos, or a Gaussian splat might want 3D translation + rotation invariance.
I believe you can also go further and start adding in gauge groups and whatnot, but I just thought about that off the top of my head.
1. Rotary positional embeddings are representations of the translation group. The n-dimensional translation group ℝⁿ is abelian, so its irreducible representations are all one-dimensional ρ: ℝⁿ → ℂ, specified by a momentum vector m: ρₘ(x) = exp(i⟨m, x⟩). You can get a multi-dimensional representation by choosing several m, which are your learnable parameters.
2. If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant).
3. If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to [0, 1]. Moving up/down/to a neighbor in the quadtree requires an affine transformation of the local coordinates, which compose, so we can label every node by its affine transformation to the root (x, d): y → x + 2⁻ᵈy, where x is the node's translation in the image and d is its depth in the tree. The finite-dimensional irreducible representations are parameterized by momentum vectors m as well as a modulus M and basis vectors {e₁, e₂, ..., eₖ} where k = ord₂(M) and look like ρₘ,ₑ(x, d) = DPᵈ where D is the diagonal matrix with entries Dⱼⱼ = exp(2πi⟨2ʲm, x⟩ / M) and P permutes the basis vectors.
From what I usually see, ML researches just throw everything at the wall and see what sticks. Whatever approach turns out to work better, gets used more often, and sometimes theoretical background is developed for why it works. RoPE works not because it’s good math, just it turned out to work better than some other equally arbitrary approaches at some point, and then other approaches have also been developed that work better than RoPE.
It's always like this because people can't admit ML is just spray and pray. So they superimpose theory.
However would be good to add that the solution can be structural(like in NoPE), then you dont need explicit representations and it just falls out of your architecture.
Also, I expect the target audience of the blog is ML researchers, since he goes into some of the math, and umm... the kind of ML researcher that is not at least aware of the basics of abstract algebra is closer to an alchemist than a researcher.
I don't expect a blog post to use the same phrasing as an academic article in several regards, including what you've described.
The reason you want to know the general theory behind positional embeddings is it lets you work with other kinds of relative positions. For example, quad/octtrees are useful for faster attention on large images/videos, or a Gaussian splat might want 3D translation + rotation invariance.
I believe you can also go further and start adding in gauge groups and whatnot, but I just thought about that off the top of my head.