On attention heads and bilinear forms

3 Profiles of bilinear forms

All matrices are real. Matrix norms are Frobenius norms. The Lean development uses their squares to avoid unnecessary square roots.

For \(L\ne 0\), let \(\alpha \) and \(\beta \) be the decreasing lists of positive eigenvalues of \(S\) and absolute values of its negative eigenvalues, padded with zeros to equal length. Define \(\pi (L)=(a,b,c,d)\) by

\[ a=\frac{\lVert T\rVert ^2}{\lVert L\rVert ^2},\qquad b=\frac{2\langle \alpha ,\beta \rangle }{\lVert L\rVert ^2},\qquad c=\frac{\lVert (\alpha -\beta )_+\rVert ^2}{\lVert L\rVert ^2},\qquad d=\frac{\lVert (\alpha -\beta )_-\rVert ^2}{\lVert L\rVert ^2}. \]

Positive and negative parts are taken entrywise. Lean pads both lists to length \(N\) and includes the denominator in each definition.

If the symmetric part \(S\) of \(L\) is nonzero, \(\pi (L)=(a,b,c,d)\) and \(\pi (S)=(0,b_S,c_S,d_S)\), then \((b,c,d)=(1-a)(b_S,c_S,d_S)\).

Proof ▼

Both triples are the same expressions in the eigenvalues of \(S\), divided by \(\lVert L\rVert ^2\) and by \(\lVert S\rVert ^2\) respectively, and \(\lVert S\rVert ^2/\lVert L\rVert ^2=1-\lVert T\rVert ^2/\lVert L\rVert ^2=1-a\).

If \(L\) has rank one and unit norm, then with \(t=\operatorname {tr}(L)\),

\[ \pi (L)=\Bigl(\frac{1-t^2}2,\frac{1-t^2}2,(t_+)^2,(t_-)^2\Bigr). \]

For \(N\ge 2\) the set of profiles of rank-one matrices is \(\Theta =\{ a=b,\ cd=0\} \subset \Delta _3\). (For \(N=1\) only the two vertices \(c=1\) and \(d=1\) occur, so the set statement needs \(N\ge 2\).)

Proof ▼

Write \(L=uv^T\) with unit vectors, so \(t=\langle u,v\rangle \); the profile is invariant under orthogonal changes of coordinates. If \(u=\pm v\), bring \(u\) to the first basis vector. Otherwise \(e=(u+v)/\sqrt{2+2t}\) and \(f=(v-u)/\sqrt{2-2t}\) are orthonormal; in a basis extending them \(L\) is the block \(\tfrac 12\bigl(\begin{smallmatrix} 1+t & \sqrt{1-t^2} \\ -\sqrt{1-t^2} & t-1 \end{smallmatrix}\bigr)\), whose symmetric part is \(\operatorname {diag}((1+t)/2,(t-1)/2)\). Read off the profile. Every point of \(\Theta \) arises from \(u=e_1\), \(v=te_1+\sqrt{1-t^2}\, e_2\).

Let \(X{\gt}0\) have finite nonzero second moment, fix \(u,v{\gt}0\) and \(\rho =u/v\). Let \(N_n\ge 2n\) and let \(S_n\in M_{N_n}(\mathbb {R})\) be symmetric with eigenvalues \(vX_1,\ldots ,vX_n\), \(-uX'_1,\ldots ,-uX'_n\) and zeros, where the \(X_i,X'_i\) are independent copies of \(X\). Then almost surely

\[ \pi (S_n)\longrightarrow \frac1{1+\rho ^2}\bigl(0,\, 2\rho ,\, (1-\rho )_+^2,\, (\rho -1)_+^2\bigr)\in p(\Theta ), \]

where \(p(a,b,c,d)=(0,a+b,c,d)\). As \(\rho \) ranges over \((0,1]\) the limit traces the edge \(\{ d=0\} \), and over \([1,\infty )\) the edge \(\{ c=0\} \).

Proof ▼

Write \(\beta _n=\rho \alpha _n+\varepsilon _n\) for the sorted lobes; by Lemma 6, \(\delta _n=\lVert \varepsilon _n\rVert /\lVert \alpha _n\rVert \to 0\) almost surely. By Cauchy–Schwarz, the triangle inequality and the \(1\)-Lipschitz property of \(x\mapsto x_\pm \), the ratios \(\langle \alpha ,\beta \rangle /\lVert \alpha \rVert ^2\), \(\lVert \beta \rVert ^2/\lVert \alpha \rVert ^2\) and \(\lVert (\alpha -\beta )_\pm \rVert /\lVert \alpha \rVert \) lie within \(\delta _n\), \(\delta _n^2+2\rho \delta _n\) and \(\delta _n\) of \(\rho \), \(\rho ^2\) and \((1-\rho )_\pm \). Divide by \(\lVert S_n\rVert ^2/\lVert \alpha _n\rVert ^2=1+\lVert \beta _n\rVert ^2/\lVert \alpha _n\rVert ^2\). On each edge, \(b(\rho )=2\rho /(1+\rho ^2)\) is a strictly monotone bijection onto \((0,1]\).