On attention heads and bilinear forms

3 Balanced bimodal spectra and simplex faces

For \(m\ge 1\), let \(\Lambda _m\) consist of the unit vectors \(\lambda _1\le \cdots \le \lambda _{2m}\) with \(\lambda _m{\lt}0{\lt}\lambda _{m+1}\). Set

\[ (\lambda _+)_j=\lambda _{2m+1-j},\qquad (\lambda _-)_j=-\lambda _j\quad (1\le j\le m). \]

Define

\[ \mathcal I=\lVert \lambda _+\rVert ^2-\lVert \lambda _-\rVert ^2,\quad C=\langle \lambda _+,\lambda _-\rangle ,\quad \mathcal P=1-\lVert \lambda _+-\lambda _-\rVert ,\quad (\iota \lambda )_i=-\lambda _{2m+1-i}. \]

For \(\lambda \in \Lambda _m\),

\[ \lVert \lambda _+\rVert ^2=\tfrac 12(1+\mathcal I),\qquad \lVert \lambda _-\rVert ^2=\tfrac 12(1-\mathcal I),\qquad 2\lVert \lambda _+\rVert \lVert \lambda _-\rVert =\sqrt{1-\mathcal I^2}, \]
\[ \lVert \lambda _+-\lambda _-\rVert ^2=1-2C,\qquad \mathcal P=1-\sqrt{1-2C},\qquad \sum _j((\lambda _+)_j+i(\lambda _-)_j)^2=\mathcal I+2iC. \]

Moreover, \(0\le 2C\le 1\) and \(0\le \mathcal P\le 1\).

Proof

The two lobes partition a unit vector, so their squared norms sum to one. Combine this with their difference \(\mathcal I\), expand the squared difference and the complex square, and use nonnegativity of the lobe entries and of squared norms for the bounds.

The map \(\iota \) is an orthogonal, self-adjoint linear involution, preserves \(\Lambda _m\), and exchanges \(\lambda _+\) and \(\lambda _-\). Consequently \(\mathcal I\circ \iota =-\mathcal I\), \(C\circ \iota =C\) and \(\mathcal P\circ \iota =\mathcal P\). For \(\lambda \in \Lambda _m\),

\[ \lVert \lambda -\iota \lambda \rVert ^2=2\lVert \lambda _+-\lambda _-\rVert ^2,\qquad \mathcal P(\lambda )=1-\frac1{\sqrt2}\lVert \lambda -\iota \lambda \rVert . \]

In particular, \(\mathcal P(\lambda )=1\) if and only if \(\iota \lambda =\lambda \).

Proof

Reversing the coordinates twice gives the identity; reversing and negating preserves inner products and the ordering and sign pattern defining \(\Lambda _m\). Reindex the finite sums to obtain the transformation rules. Split the displacement sum into its two halves to identify both with the squared distance between the lobes.

Let \(L\ne 0\) and \(\pi (L)=(a,b,c,d)\). The coordinates are nonnegative and sum to one. Their faces and vertices have the following fibers.

  1. \(a=0\) exactly when \(L\) is symmetric, and \(a=1\) exactly when \(L\) is antisymmetric.

  2. \(b=0\) exactly when \(S\) is semidefinite, and \(b=1\) exactly when \(L\) is symmetric and \(\alpha =\beta \).

  3. \(d=0\) exactly when \(\alpha _j\ge \beta _j\) for every \(j\), equivalently \(S=H+P\) with \(H\) symmetric with spectrum symmetric about zero and \(P\) positive semidefinite. The matrices may be chosen to commute with \(S\) and with each other, with \(\lVert P\rVert ^2=c\lVert L\rVert ^2\). Also, \(d=1\) exactly when \(L\) is symmetric negative semidefinite.

  4. \(c=0\) exactly when \(\beta _j\ge \alpha _j\) for every \(j\), equivalently \(S=H-P\) with the same conditions on \(H,P\). They may be chosen to commute with \(S\) and with each other, with \(\lVert P\rVert ^2=d\lVert L\rVert ^2\). Also, \(c=1\) exactly when \(L\) is symmetric positive semidefinite.

Proof

The spectral theorem and the energy partition give nonnegativity and sum one. Vanishing squared norms characterize the \(a\) face and the coordinatewise inequalities for \(c,d\). The lobe inner product vanishes exactly when one lobe is zero, giving the \(b\) face. When \(\alpha \ge \beta \), pair positive and negative eigenvectors to construct \(H\) with paired eigenvalues; the remainder \(P\) is positive semidefinite. Both are diagonal in an eigenbasis of \(S\), giving commutation and the norm formula. Conversely, adding a positive semidefinite matrix increases the ordered eigenvalues, so comparison with the paired spectrum of \(H\) gives lobe dominance. Apply the argument to \(-L\) for the other face. The vertices follow from the partition of one.