CS Paper Radar

Phase Space Attention:A Hairer Lift Circumvents the Single-Layer Induction Obstruction

Kingsuk Maitra, Shagun Sood Morteza Hosseini, Suman Gunnala, Vikram Gupta

cs.LG cond-mat.mtrl-scics.AI ·submitted 2026-09-26 07:29 UTC · v1
theoryunusual

Abstract

We circumvent the Sanford-Hsu-Telgarsky (SHT) single-layer induction obstruction within a linear, one-step, causal, bilinear, symplectically consistent design class on the post-RoPE substrate, by lifting attention onto a symplectic phase space, mirroring Hairer's lift of Stormer-Verlet. The lift exits the premise of the SHT counting argument rather than the bound itself. The reframing exhibits the obstruction as a filter-order gap: a one-layer bilinear score realises a $z$-transform of joint order $(0,0)$, whereas the induction discriminator requires key-side order $\geq 1$. Applying the symplectic upper shear $M_γ:(q,p)\mapsto(q+γp,p)$ to the post-RoPE query and key streams closes it. We prove this lift is unique within the factorised subclass, exactly symplectic at operator level, and requires post-RoPE placement; and in an explicit $T_4$-only Gaussian reduction we derive a closed-form two-branch induction phase transition, held out at $r=0.9876$ with zero fitted parameters. That law is an analytically solvable limit, not a robust prediction: restoring the $T_3$ channel moves $γ_c$ at $d_k=64$ from 1.030 to 0.569 and removes the crossover. Deployability follows by exact derivation: the KV cache is unchanged, prefix reuse and speculative decoding are preserved, overhead is $6d$ FLOPs per token per layer, INT8 headroom grows by at most $\log_2(1+2γ)$ bits, fused kernels are unmodified, and no parameters are added. At 91.3M parameters a supercritical sweep locates an emergence band: induction forms 3/3 seeds at $γ=0.80$ in a mean of 717 steps, against 2/3 seeds and 2700 steps at $γ=0$. Adverse results are reported as directly: a key-only half-lift reaches 0.949 against 0.811 for the symmetric operator, so if induction accuracy is the objective, the half-lift is the better construction. Forty-one notebooks and result files ship as ancillary material.

Closest papers

Nearest neighbours by meaning across the whole index.

HySPE: Positional Encoding via Symplectic Dual Shears

cs.CL ·2026-10-07 ·Zhongping Ji
similarity 0.60
Abstract · 11 pages

We introduce Hyperbolic Symplectic Positional Encoding (HySPE), grounding positional attention in non-compact symplectic transformations. While canonical Rotary Position Embedding (RoPE) parameterizes the compact, elliptic branch of $\Sp(2,\R)$ via rotations, HySPE operationalizes its hyperbolic branch via a damped symmetric composition of dual shears, yielding a conformally symplectic contraction with two spectral decay rates per channel pair. To eliminate the exponential representation drift inherent to naive absolute factorizations, we diagonalize the operator in its invariant eigenbasis and introduce blockwise coordinate rebasing with adaptive centered execution. This guarantees length-independent numerical bounds while matching cached RoPE forward latency (7.21\,ms on an RTX 4090). On TinyShakespeare, HySPE-UltraLong maintains an invariant perplexity of 4.810 up to $16\times$ zero-shot extrapolation ($L=4096$), whereas RoPE degrades to 131.198. Scaled to a 51M-parameter subword Transformer on WikiText-103 ($L_{\text{train}}=512$), HySPE closely matches RoPE in-domain while robustly extrapolating to length 8192, reducing tail perplexity by 83.9\% over RoPE. While these controlled experiments establish HySPE's extrapolation robustness and numerical stability, evaluating its scaling behavior on large-scale foundation models remains an important direction for future investigation.

0.26

Integrable building blocks for symplectic neural networks: learned-type layers, fiber-preserving layers, and what a single layer can do

cs.LG math.NAmath.SG ·2026-10-18 ·Andrew Qing He
similarity 0.51
Abstract · 27 pages, 28 figures

Exactly symplectic neural networks are built almost exclusively from shear maps and gain expressivity only through depth. We ask what else a single explicit symplectic layer can be. Observing that every layer in use is the exact flow of a Hamiltonian that is a function of commuting integrals, we obtain three families of layers: a learned-type layer, in which every conjugate pair is rotated, squeezed or sheared by a state-dependent amount and the type is found by training; moving-plane shears, residual symplectic maps outside the known ridge-function class; and fiber-preserving layers, which combine an unconstrained point transformation with a fiber translation at the cost of one linear solve. We characterize residual symplectic layers $x \mapsto x + J\nabla H(x)$: in one degree of freedom they are exactly the shears along a line, while from two degrees of freedom on a strictly larger class exists. Conjugating a learned-type layer by a symplectic network yields an exact one-parameter group with an exactly conserved learned Hamiltonian. In experiments on sixteen systems with one to forty degrees of freedom, against SympNets, HénonNets and learned-plane shear networks, layers built on nonlinear integrals are far more accurate when the Hamiltonian contains functions of those integrals. On nonlinear Schrödinger lattices with six to forty sites, twist layers alternating with linear symplectic maps are over a hundred times more accurate than every shear-based network, and with an on-site potential of learned form three to four orders of magnitude more accurate. On chains of Tao oscillators with the step size as an input, stretch layers are 45 to 70 times more accurate than learned-plane shears and recover the Hamiltonian to 1%. On generic non-separable systems, fiber layers improve on learned-plane shears by at most a factor of two, and widening the networks does not help any architecture.

0.25

Grassmann--Plücker Parametrization of Convolutional Filter Subspaces: Regularity and Closed Embeddings

cs.LG math.AG ·2026-09-03 ·Hongyu Yuan, Huaiqing Zuo
unusual similarity 0.51
Abstract · 24 pages; includes a symbolic computational example using Singular

We propose a geometric parametrization of the filters in a single convolutional layer: the parameter is no longer an ordered family of filter vectors, but a fixed-dimensional subspace of the filter space. For one-dimensional finite-stride convolution, the filter-to-convolution-operator correspondence gives an injective linear map $\mathcal{C}:\mathcal{K}\to H$. This map sends filter subspaces in $\mathrm{Gr}(q,\mathcal{K})$ to operator subspaces in $\mathrm{Gr}(q,H)$; composing it with the Plücker embedding yields a projective parametrization $Φ:\mathrm{Gr}(q,\mathcal{K})\to\mathbb{P}(\bigwedge^q H)$. Using $T_U\mathrm{Gr}(q,\mathcal{K})\cong\mathrm{Hom}(U,\mathcal{K}/U)$, we compute the differential of the induced Grassmannian map and show that the differential of $Φ$ is injective at every point. We then use the vanishing equations for Plücker coordinates and standard affine coordinates on a Grassmannian to prove that $\mathrm{Gr}(q,\mathcal{C}(\mathcal{K}))\hookrightarrow\mathrm{Gr}(q,H)$ is a closed embedding, and hence that $Φ$ is a closed embedding. Consequently, the parameter space is isomorphic to its projective image, the parametrization is finite and birational onto its image, every fiber is a singleton, and the resulting projective neural variety is smooth. For $k=4$ and $q=2$, we also use Singular to recover the image ideal and check its dimension, degree, chart rank, and smoothness. This computation illustrates, rather than replaces, the general proof. Finally, we discuss possible connections with filter redundancy and low-rank convolution, while distinguishing the proved geometric results from application proposals requiring numerical validation.

0.31

Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention

cs.LG math.AGmath.RT ·2026-08-31 ·Tīkun Ông, Georg Bökman
theory similarity 0.50
Abstract · v2: added experiments. 29 pages, 10 figures

We give a complete characterization of equivariant multi-head self-attention (MHSA): if an MHSA layer is equivariant to a symmetry group $G$, then $G$ can only act by permuting head-clusters, with QK and OV matrices satisfying an equivariance constraint tied to the group action. As a consequence, we prove that any fixed MHSA architecture that achieves exact equivariance by polynomially parameterizing unconstrained MHSA parameters inevitably leads to expressivity loss within the class of equivariant maps: the equivariance locus of unconstrained MHSA forms a union of extremely many Zariski-irreducible components in a reduced parameter space, and any single architecture covers at most one. For $G=D_4$ acting on $C$ copies of the regular representation as the token feature space, we show that there are $Ω(C^{64})$ components for eight attention heads. We illustrate this with experiments on artificial tasks and show that the predicted head-cluster structures can arise naturally when the attention layer is pressured to become equivariant.

0.42

Symplectic Normal Coordinates Flow for Generative Modeling of Hamiltonian Systems

cs.LG physics.acc-ph ·2026-11-08 ·Christopher M. Pierce
top venuecode/data similarity 0.48
Abstract · Accepted at NeurIPS 2026. 10 pages, 6 figures plus appendices. Code: https://github.com/electronsandstuff/Normal-Coordinates-Flow

In conditioned generative models of physical systems, the densities at different values of the conditioning parameters are often related through symplectic transformations. For stable linear Hamiltonian systems, there exists a symplectic transformation to "normal coordinates" such that the dynamics is reduced to rotations in each phase-space plane. This picture extends to the nonlinear case where, away from resonances, the Birkhoff normal form again consists of rotations in phase space, now with amplitude dependence. In this work, we introduce an invertible symplectic flow layer consisting of amplitude-dependent rotations in learned normal coordinates. We derive the condition under which the nonlinear rotation layer is symplectic and detail an architecture that guarantees it. Experiments are carried out on three conditional generative modeling problems: a synthetic chain of coupled nonlinear oscillators and two problems from accelerator physics. We show empirically that normalizing flows with the new layers achieve competitive or superior negative log-likelihoods compared to baseline flows, with gains that grow with the strength of the nonlinearity on the synthetic system.

0.51

Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras

cs.LG ·2026-09-01 ·Jiming Feng, Junliang Li
similarity 0.48
Abstract · 14 pages, 2 figures. Preprint

We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators $T=OV^\top$ nearly closes under composition, $T^2\approxαT$. Across six pretrained endpoints spanning 2.8B--235B parameters, 3.98--8.00% of heads reach squared closure alignment $\mathcal{P}\geq0.9$, while no matched within-layer O/V mismatch does. An exact principal-coordinate factorization, $T=Q_OKQ_V^\top$ and $T^2=Q_O(KDK)Q_V^\top$, separates within-support transport from read--write return geometry. Across all 7,304 heads in nine MHA/GQA models, scrambling only the orientation of $K$ while preserving singular values, norms, factor spans, and principal angles reduces median closure from 0.336 to $1.04\times10^{-4}$; trained orientation wins for 98.64% of heads and in every layer. Constructive searches show that high closure is feasible in every surveyed layer, but usually not attained. Retrospective trajectories in three independently trained lineages further separate broadly available capacity from the orientations attained by final strong heads. Under exact value sharing, headwise closure extends to a right-action algebra, $T_iT_j=α_jT_i$. Seven-model experiments verify the approximate law and reveal distinct oblique projections with a shared value-defined kernel. These results characterize scaled idempotence as a sparse trained orientation within broadly available geometric capacity and show how value sharing extends a headwise relation into a local operator algebra.

0.40

Ancestor Lifting of Quasi-Cyclic Additive Codes and Certified Quantum Codes

cs.IT quant-ph ·2026-10-27 ·Liangdong Lu, Ran Xu, Ruipan Yang, Qiang Fu
theory similarity 0.46
Abstract

Additive codes over $\Ffour$ that are self-orthogonal under the trace inner product correspond to binary quantum stabilizer codes. We study 1-generator $(1,2)$-quasi-cyclic (QC) additive codes and their exact quantum distances. For composite lengths $n=\ell m$, an ancestor-lifting theorem transports dual words of the folded ancestor to the length-$n$ code. A two-layer decomposition for $\ell=2$ and a CRT splitting for odd $\ell$ yield dimension laws and explain equality, escape, and mixed-branch behavior in the certified examples. A palindromic criterion ensures symplectic self-orthogonality, while reversal pairing and a symplectic Gram rank formula describe broader construction families. Exact distances are established by information-set decoding screening, Brouwer--Zimmermann computation on the additive dual, and independent witness verification. We certify twenty-one record-breaking binary quantum codes across twenty entries of Grassl's code table, with twenty-eight certified members in total. Propagation improves thirty-four further entries, and explicit constructions are supplied for three table-tying GV entries marked Code Missing Construction. Six additional candidates are reported only with rigorous upper bounds and heuristic screening evidence. The results connect divisor-lattice structure in additive quasi-cyclic codes with reproducible certification of exact quantum distances.

0.10

Singer-Difference-Set Qudit Stabilizer Codes from Non-Degenerate Quadrics in PG(d,q)PG(d,q): Construction, Structural Theorems, and Monte-Carlo Performance

cs.IT quant-ph ·2026-10-01 ·Michel Kulhandjian, Lajos Hanzo
publishedtheory similarity 0.46
Abstract · Accepted for publication in IEEE Open Journal of the Communications Society

We propose Q(q,d), a family of q-ary non-CSS qudit stabilizer codes with prime power q and dimension d >= 2, constructed from Singer difference sets and non-degenerate quadrics in PG(d,q). The parity check H = (A | M_Q A) couples the Singer-circulant incidence matrix A with the quadric circulant M_Q, and a Cancellation Lemma uses the identity (k - lambda) = q^{d-1} = 0 mod p to make H symplectic-isotropic over F_q. Five structural theorems carry the novelty: a cross-correlation modular-constancy identity nu = 1 mod q at q odd, d >= 3, with a two-value pattern that proves a 2-torsion distance ceiling; a CRT zero-count identity; an explicit Clifford bridge identifying Q(q,2) as a graph-state encoding of the trivial Singer CSS code, with the 2021 low-density-spreading (LDS) sign-flip pattern realised by a symmetric circulant; a paired column-rigidity equality d_min = R_P + 1; and a lower bound d_min(Q(7,2)) >= 8 from an exhaustive Singer-Frobenius search. A Hamada-Smith specialisation yields the closed-form logical dimension k(p,d) = theta_d(p) - C(p+d-1, d) - 1, and the flagship Q(p,2) = [[p^2 + p + 1, p(p+1)/2, d_min]]_p attains asymptotic rate one-half. We conjecture d_min(Q(p,2)) = p + 1 for odd primes, proven at p = 5 and lower-bounded to >= 8 at p = 7. Instances include Q(3,2) = [[13,6,3]]_3, Q(5,2) = [[31,15,6]]_5, Q(7,2) = [[57,28, d>=8]]_7, and Q(3,4) = [[121,105,3]]_3. Monte-Carlo over 1.5 x 10^6 trials shows Q(5,2) achieves a 70-fold per-logical-qudit improvement over Steane [[7,1,3]]_2 at depolarising rate 10^-3 with zero silent logical errors. A textbook bounded-distance decoder is used; Berlekamp-Massey-style cyclic decoding is left as the natural sequel.

0.29