Dependencies

Let the entries of \(L\in M_N(\mathbb {R})\) be independent, mean zero, and square-integrable with common variance \(\sigma ^2\). Then

For \(N{\gt}0\) and \(\sigma \ne 0\), the ratios to \(\mathbb {E}\lVert L\rVert ^2\) are \((N+1)/(2N)\) and \((N-1)/(2N)\) respectively. If the entries are iid nondegenerate centered Gaussian, then also

Let \(W_K,W_Q\in M_{n\times N}(\mathbb {R})\) have jointly independent, mean-zero, square-integrable entries, with variances \(\sigma _K^2\) and \(\sigma _Q^2\) respectively. For \(L=W_K^TW_Q\),

For positive dimensions and nonzero variances, the ratios of the last two expectations to the first are \((N+1)/(2N)\) and \((N-1)/(2N)\).

A smooth map \(a\colon C^k\to C\) is an attention function if there are a smooth \(\omega \colon C^k\to \Delta _{k-1}\), a smooth convex \(\phi \colon \mathbb {R}^k\to \mathbb {R}\) and a polynomial map \(\sigma \colon C^k\to \mathbb {R}^k\) with

The data \(\omega ,\phi ,\sigma \) are carried along with \(a\). A polynomial map is one whose components are polynomials in the coordinates of \(x_1,\ldots ,x_k\).

For \(m\ge 1\), let \(\Lambda _m\) consist of the unit vectors \(\lambda _1\le \cdots \le \lambda _{2m}\) with \(\lambda _m{\lt}0{\lt}\lambda _{m+1}\). Set

Define

An attention function whose scoring function has the form \(\sigma (x)=(P_{k-1}(x_1,x_k),\ldots ,P_0(x_k,x_k))\) for polynomials \(P_d\colon C^2\to \mathbb {R}\) of bidegree at most \((1,1)\) is biaffine; it is bilinear if every \(P_d\) is a bilinear form.

For attention functions \(a_k\colon C^k\to C\), \(k=2,\ldots ,\ell \), the causal attention head is \(A(x_1,\ldots ,x_\ell )=(x_1,a_2(x_1,x_2),\ldots ,a_\ell (x_1,\ldots ,x_\ell ))\).

For a vector \(\lambda \) with \(|\lambda _i|{\lt}1\), define

For \(L\in M_N(\mathbb {R})\), put

Then \(L=S+T\), with \(S\) symmetric and \(T\) antisymmetric.

For \(L\ne 0\), let \(\alpha \) and \(\beta \) be the decreasing lists of positive eigenvalues of \(S\) and absolute values of its negative eigenvalues, padded with zeros to equal length. Define \(\pi (L)=(a,b,c,d)\) by

Positive and negative parts are taken entrywise. Lean pads both lists to length \(N\) and includes the denominator in each definition.

Let a bilinear attention function on a head space \(H\) have forms \(B_{k-1},\ldots ,B_0\), let \(L_j\colon H\to H^\vee \), \(L_j(v)(u)=B_j(u,v)\), assume \(B_0\) nondegenerate, and set \(T_d=L_0^{-1}L_d\). The function is RoPE-like if \(T_{d+e}=T_dT_e\) whenever \(d,e\ge 0\) and \(d+e{\lt}k\).

For a head \(A\), a linear value map \(V\colon C\to C\) and a map \(M\colon C\to C\), the single-head transformer is \(T=T_M\circ T_A\) with \(T_A(x)_i=x_i+V(A(x)_i)\) and \(T_M(y)_i=y_i+M(y_i)\).

Every polynomial \(P\colon C\times C\to \mathbb {R}\) of bidegree at most \((1,1)\) has a unique decomposition \(P(u,v)=B(u,v)+L(u)+R(v)+D\) with \(B\) bilinear, \(L,R\) linear and \(D\in \mathbb {R}\), namely

For \(\lambda \in \Lambda _m\),

Moreover, \(0\le 2C\le 1\) and \(0\le \mathcal P\le 1\).

For any \(L\in M_N(\mathbb {R})\),

For \(L\ne 0\), division gives

For \(W_K,W_Q\in M_{n\times N}(\mathbb {R})\), set

The symmetric part of \(W_K^TW_Q\) is \(S=\tfrac 12M^TJM\). For every \(k\ge 1\),

If \(M:\mathbb {R}^N\to \mathbb {R}^{2n}\) is surjective and \(n\ge 1\), the nonzero spectrum of \(S\), with multiplicity, is the spectrum of \(\tfrac 12JG\). Its sorted normalized nonzero spectrum \(\lambda \) exists uniquely and belongs to \(\Lambda _n\). For \(H=JG/(2\lVert S\rVert )\),

The map \(\iota \) is an orthogonal, self-adjoint linear involution, preserves \(\Lambda _m\), and exchanges \(\lambda _+\) and \(\lambda _-\). Consequently \(\mathcal I\circ \iota =-\mathcal I\), \(C\circ \iota =C\) and \(\mathcal P\circ \iota =\mathcal P\). For \(\lambda \in \Lambda _m\),

In particular, \(\mathcal P(\lambda )=1\) if and only if \(\iota \lambda =\lambda \).

Every \(\lambda \in \Lambda _m\) has \(|\lambda _i|{\lt}1\). Writing \(n=2m\),

and both series converge absolutely. Moreover,

If a symmetric matrix \(A\) has spectrum \(\lambda \), with multiplicity, then

Let \(S\) be symmetric, let \(\lambda _\pm \) be its sorted positive eigenvalues and absolute values of negative eigenvalues, padded with zeros to equal length, and let \(\mathcal P=1-\lVert \lambda _+-\lambda _-\rVert /\lVert S\rVert \). If \(\lambda _+\ne 0\) and \(\rho =\lVert \lambda _-\rVert /\lVert \lambda _+\rVert \), then

with equality if and only if \(\lambda _-=\rho \lambda _+\).

If the symmetric part \(S\) of \(L\) is nonzero, \(\pi (L)=(a,b,c,d)\) and \(\pi (S)=(0,b_S,c_S,d_S)\), then \((b,c,d)=(1-a)(b_S,c_S,d_S)\).

Let \(X{\gt}0\) almost surely, with finite nonzero second moment, and fix \(u,v{\gt}0\), \(\rho =u/v\). Let \(\alpha _n\) be the decreasing sorting of \(vX_1,\ldots ,vX_n\) and \(\beta _n\) the decreasing sorting of \(uX'_1,\ldots ,uX'_n\), where the two sequences together consist of independent copies of \(X\). Then

If \(L\in M_N(\mathbb {R})\) has rank at most \(n\), the symmetric part of \(L\) has at most \(n\) positive and at most \(n\) negative eigenvalues, counted with multiplicity.

Let \(\widehat C=C\oplus \mathbb {R}e\) and \(\iota (x)=x+e\). For every single-head QKV transformer \(T\colon C^\ell \to C^\ell \), with scores \(\langle Q(x_k+p_k),K(x_j+p_j)\rangle /\sqrt n\), there is a single-head bilinear transformer \(\widehat T\) on \(\widehat C\) with the same potentials such that \(\widehat T(\iota x_1,\ldots ,\iota x_\ell )=\iota ^\ell (T(x_1,\ldots ,x_\ell ))\).

Let \(1\le n\le N\) and let \(W_K,W_Q\in M_{n\times N}(\mathbb {R})\) have jointly independent centered Gaussian entries, iid within each matrix, with respective nonzero variances \(v_K,v_Q\). For \(L=W_K^TW_Q\),

independently of the head dimension \(n\).

Fix a norm on \(C\) and put \(\lVert x\rVert _\infty =\max _k\lVert x_k\rVert \). If \(\lVert M(y)\rVert \le L\lVert y\rVert \) for all \(y\) and \(V=\sum _i\lVert V_i\rVert \) (operator norms), then \(\lVert T(x)-x\rVert _\infty \le (L+V+LV)\lVert x\rVert _\infty \).

If \(L\) has rank one and unit norm, then with \(t=\operatorname {tr}(L)\),

For \(N\ge 2\) the set of profiles of rank-one matrices is \(\Theta =\{ a=b,\ cd=0\} \subset \Delta _3\). (For \(N=1\) only the two vertices \(c=1\) and \(d=1\) occur, so the set statement needs \(N\ge 2\).)

With \(B_0\) nondegenerate, the function is RoPE-like if and only if there is \(S\in \operatorname {End}(H)\) with \(T_d=S^d\) for all \(0\le d{\lt}k\). When \(S\) exists it is \(T_1=L_0^{-1}L_1\).

Let \(X{\gt}0\) have finite nonzero second moment, fix \(u,v{\gt}0\) and \(\rho =u/v\). Let \(N_n\ge 2n\) and let \(S_n\in M_{N_n}(\mathbb {R})\) be symmetric with eigenvalues \(vX_1,\ldots ,vX_n\), \(-uX'_1,\ldots ,-uX'_n\) and zeros, where the \(X_i,X'_i\) are independent copies of \(X\). Then almost surely

where \(p(a,b,c,d)=(0,a+b,c,d)\). As \(\rho \) ranges over \((0,1]\) the limit traces the edge \(\{ d=0\} \), and over \([1,\infty )\) the edge \(\{ c=0\} \).

Let \(Q,K\colon C\to H\) with \(\dim H=n\) and \(\dim C=N\). If the stacked map \(J=(Q,K)\colon C\to H\oplus H\) is surjective, the symmetric form \(S(x,y)=\tfrac 12(\langle Kx,Qy\rangle +\langle Qx,Ky\rangle )\) has signature \((n_+,n_-,n_0)=(n,n,N-2n)\). In Lean the signature is read off the Gram matrix of \(S\) in an arbitrary basis of \(C\).

Let \(L\ne 0\) and \(\pi (L)=(a,b,c,d)\). The coordinates are nonnegative and sum to one. Their faces and vertices have the following fibers.

1. \(a=0\) exactly when \(L\) is symmetric, and \(a=1\) exactly when \(L\) is antisymmetric.

2. \(b=0\) exactly when \(S\) is semidefinite, and \(b=1\) exactly when \(L\) is symmetric and \(\alpha =\beta \).

3. \(d=0\) exactly when \(\alpha _j\ge \beta _j\) for every \(j\), equivalently \(S=H+P\) with \(H\) symmetric with spectrum symmetric about zero and \(P\) positive semidefinite. The matrices may be chosen to commute with \(S\) and with each other, with \(\lVert P\rVert ^2=c\lVert L\rVert ^2\). Also, \(d=1\) exactly when \(L\) is symmetric negative semidefinite.

4. \(c=0\) exactly when \(\beta _j\ge \alpha _j\) for every \(j\), equivalently \(S=H-P\) with the same conditions on \(H,P\). They may be chosen to commute with \(S\) and with each other, with \(\lVert P\rVert ^2=d\lVert L\rVert ^2\). Also, \(c=1\) exactly when \(L\) is symmetric positive semidefinite.