2 Random matrices

These results give the random baselines for the symmetric energy of an attention head. The distinction between a ratio of expectations and the expectation of a ratio matters: the latter uses Gaussian symmetry.

Lemma 5 Symmetric and skew energy identities

✓

For any \(L\in M_N(\mathbb {R})\),

\[ \lVert L\rVert ^2=\lVert S\rVert ^2+\lVert T\rVert ^2,\qquad \lVert S\rVert ^2=\tfrac 12\lVert L\rVert ^2+\tfrac 12\operatorname {tr}(L^2),\qquad \lVert T\rVert ^2=\tfrac 12\lVert L\rVert ^2-\tfrac 12\operatorname {tr}(L^2). \]

For \(L\ne 0\), division gives

\[ \frac{\lVert S\rVert ^2}{\lVert L\rVert ^2}=\frac12+\frac{\operatorname {tr}(L^2)}{2\lVert L\rVert ^2}, \qquad \frac{\lVert T\rVert ^2}{\lVert L\rVert ^2}=\frac12-\frac{\operatorname {tr}(L^2)}{2\lVert L\rVert ^2}. \]

Proof ▼

Expand the squared Frobenius norms entrywise, using invariance under transpose and \(\sum _{i,j}L_{ij}L_{ji}=\operatorname {tr}(L^2)\). Adding the two identities gives the orthogonal decomposition of the energy.

Corollary 1 Iid square random matrix baseline

✓

Let the entries of \(L\in M_N(\mathbb {R})\) be independent, mean zero, and square-integrable with common variance \(\sigma ^2\). Then

\[ \mathbb {E}\lVert L\rVert ^2=N^2\sigma ^2,\quad \mathbb {E}\lVert S\rVert ^2=\frac{N(N+1)}2\sigma ^2,\quad \mathbb {E}\lVert T\rVert ^2=\frac{N(N-1)}2\sigma ^2. \]

For \(N{\gt}0\) and \(\sigma \ne 0\), the ratios to \(\mathbb {E}\lVert L\rVert ^2\) are \((N+1)/(2N)\) and \((N-1)/(2N)\) respectively. If the entries are iid nondegenerate centered Gaussian, then also

\[ \mathbb {E}\left[\frac{\lVert S\rVert ^2}{\lVert L\rVert ^2}\right]=\frac{N+1}{2N}, \qquad \mathbb {E}\left[\frac{\lVert T\rVert ^2}{\lVert L\rVert ^2}\right]=\frac{N-1}{2N}. \]

Proof ▼

Independence and centering kill the off-diagonal cross terms, giving \(\mathbb {E}\operatorname {tr}(L^2)=N\sigma ^2\). Apply the energy identities. In the Gaussian case, symmetry of the joint law makes the expected normalized squared coordinate equal to \(1/N^2\), while sign symmetry cancels the remaining normalized cross terms.

Corollary 2 Random QK-product baseline

✓

Let \(W_K,W_Q\in M_{n\times N}(\mathbb {R})\) have jointly independent, mean-zero, square-integrable entries, with variances \(\sigma _K^2\) and \(\sigma _Q^2\) respectively. For \(L=W_K^TW_Q\),

\[ \mathbb {E}\lVert L\rVert ^2=nN^2\sigma _K^2\sigma _Q^2,\qquad \mathbb {E}\lVert S\rVert ^2=\frac{nN(N+1)}2\sigma _K^2\sigma _Q^2,\qquad \mathbb {E}\lVert T\rVert ^2=\frac{nN(N-1)}2\sigma _K^2\sigma _Q^2. \]

For positive dimensions and nonzero variances, the ratios of the last two expectations to the first are \((N+1)/(2N)\) and \((N-1)/(2N)\).

Proof ▼

Expand \(L_{ab}=\sum _r(W_K)_{ra}(W_Q)_{rb}\). Independence gives \(\mathbb {E}L_{ab}^2=n\sigma _K^2\sigma _Q^2\) and \(\mathbb {E}\operatorname {tr}(L^2)=nN\sigma _K^2\sigma _Q^2\). Sum the entries and use Lemma 5.

Proposition 6 Expected symmetric energy for random low-rank QK products

✓

Let \(1\le n\le N\) and let \(W_K,W_Q\in M_{n\times N}(\mathbb {R})\) have jointly independent centered Gaussian entries, iid within each matrix, with respective nonzero variances \(v_K,v_Q\). For \(L=W_K^TW_Q\),

\[ \mathbb {E}\left[\frac{\lVert S\rVert ^2}{\lVert L\rVert ^2}\right]=\frac{N+1}{2N}, \qquad \mathbb {E}\left[\frac{\lVert T\rVert ^2}{\lVert L\rVert ^2}\right]=\frac{N-1}{2N}, \]

independently of the head dimension \(n\).

Proof ▼

First take equal variances. Sign changes and permutations of Gaussian columns preserve the joint law. Sign symmetry cancels the off-diagonal terms of \(\operatorname {tr}(L^2)/\lVert L\rVert ^2\); permutation symmetry makes the expected normalized squared entries equal. Thus its expectation is \(1/N\), and Lemma 5 gives the result. The denominator is nonzero almost surely. Rescale \(W_K\) to reduce unequal variances to the equal-variance case; the normalized energies are unchanged by this rescaling.

Lemma 6 Proportional lobes

✓

Let \(X{\gt}0\) almost surely, with finite nonzero second moment, and fix \(u,v{\gt}0\), \(\rho =u/v\). Let \(\alpha _n\) be the decreasing sorting of \(vX_1,\ldots ,vX_n\) and \(\beta _n\) the decreasing sorting of \(uX'_1,\ldots ,uX'_n\), where the two sequences together consist of independent copies of \(X\). Then

\[ \frac{\lVert \beta _n-\rho \alpha _n\rVert }{\lVert \alpha _n\rVert } \longrightarrow 0\qquad \text{almost surely as }n\longrightarrow \infty . \]

Proof ▼

Reduce to two samples with the same scale. For sorted nonnegative lists, a layer-cake identity expresses their inner product using the minimum of two empirical survival counts. The strong law controls these counts and the empirical second moments. Fatou’s lemma bounds the limiting inner product from below by the common second moment. Expanding the squared difference gives convergence to zero after division by \(n\); the squared denominator divided by \(n\) converges to the strictly positive second moment. Restore the scales \(u,v\).