CS Paper Radar

Beyond Embedding Transfer: Component Roles in Grokking Transfer and Stability

Zeyu Jia

cs.LG ·submitted 2026-09-16 03:28 UTC · v3

Abstract

Warm-start transfer accelerates delayed generalization (grokking) on algorithmic tasks, yet how internal blocks, residual-stream depth, group non-commutativity, and continual optimization govern transferred circuits remains unclear. Across ten scale-matched batteries ($>1{,}800$ trajectories) on abelian $\mathbb{Z}_{113}$ ($L \in \{1,2,3\}$) and non-abelian $S_5$, transferring internal blocks alongside boundary carriers lifts early accuracy by $5.46$ percentage points and cuts validation confirmation latency by $616.7 \pm 129.0$ steps in 1-layer models ($p_{\rm Holm}=0.004$). In 2-layer models, full transfer compresses trigger onset $8.1\times$ ($+704.7 \pm 84.3$ integral units), whereas omitting donor embeddings incurs a $-4{,}475.6 \pm 76.8$-unit interface deficit. On ternary $S_5$ conjugation ($aba^{-1}$), two-token architectures stall in a unary conjugacy-class shortcut ($\mathrm{Cl}(aba^{-1})=\mathrm{Cl}(b)$, $34.48\% \pm 6.19\%$ peak accuracy). Combining three-token causal routing ($[a,b,a]$) with Peter--Weyl width ($d_{\rm model}=256 > \sum_λd_λ^2 = 120$) reaches $97.97\% \pm 1.01\%$ ($80.08\% \pm 7.81\%$ at equal $427\text{k}$ parameters) with $3.2\times$ faster onset than scratch. Under continued AdamW optimization, unshielded circuits relapse by $11.85\%\text{--}80.58\%$ drawdown, and depth-$L$ stability follows a graded $(L-1)$-prefix pinning hierarchy ($75.99\% \to 31.26\% \to 0.32\% \to 0.04\%$ drawdown at $L=3$). Finally, \textbf{Closed-Loop Curvature-Anchored Consolidation} ($η_k = η_0(1 - A_{\rm val}(k))$ with monotonic AMSGrad second moments) holds saturated drawdown to $0.00\%\text{--}0.12\%$ and lifts pre-saturation terminal accuracy by $+1.67\text{--}5.37$ points over fixed-budget micro-stepping ($87.06\% \pm 0.94\%$ vs.\ $81.69\% \pm 0.90\%$ on $S_5$, $p_{\rm Holm}=0.002$).

Comment
21 pages, 6 figures, 12 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

cs.AI cs.LG ·2026-09-09 ·Anish Kataria
similarity 0.55
Abstract

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on \emph{why} this transition occurs, the quantitative structure of \emph{when} it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time: $T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}$ ($R^2 = 0.732$; $0.821$ with interactions). The exponent hierarchy reveals that data complexity ($D^{-2.04}$) is the dominant driver of regime transition, not model capacity ($H^{-0.27}$): doubling data accelerates generalization by ${\sim}4\times$, while doubling width yields only ${\sim}1.2\times$. A sharp phase boundary at weight decay $λ\gtrsim 1.0$ separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

0.49

ReKV: Overlapping Quantized KV Transfer and Suffix Recomputation

cs.DC ·2026-11-05 ·Guangjian Dong
benchmark similarity 0.51
Abstract · 5 pages, 3 figures, 3 tables

Prefill--decode disaggregation moves a prompt's key--value (KV) cache between workers before the first token can be generated. Reducing this handoff to either cache compression or recomputation misses a useful choice: which layers to transmit, at what precision, and which layers to execute again on the decode worker. ReKV transfers a quantized prefix KV cache and one BF16 boundary state, then reconstructs a contiguous suffix while the prefix is in flight. In a model-shaped synthetic benchmark on cross-node H100s, the Q8 configuration meeting separately evaluated quality criteria reduces 8K handoff latency by $2.36\times$, $2.84\times$, and $3.65\times$ over BF16 transfer for Qwen2.5-7B, Qwen2.5-3B, and Qwen3-8B. It is also faster than the best measured transfer-only or recompute-only action at those points. A separate real-weight BF16 recomputation experiment reduces prefill-to-token-return time by $1.16$--$1.63\times$. The preferred action changes with prompt length and link bandwidth.

0.30

When do machine-learned exchange-correlation improvements inherit into density-functional tight binding?

cs.LG cond-mat.mtrl-sciphysics.chem-phquant-ph ·2026-08-14 ·Can Polat, Mustafa Kurban, Erchin Serpedin, Hasan Kurban
similarity 0.50
Abstract

Machine-learned exchange-correlation functionals correct band gaps at near-semilocal cost, while density-functional tight binding reaches the $10^3$-$10^6$-atom regime; combining them assumes that a better parent yields a better parameterization, but we show it does not. Current-generation functionals are orbital-dependent generalized Kohn-Sham operators, whereas the parameterization channel is built on a multiplicative potential, preventing exact representation. Using the transfer ratio, the surviving fraction of a parent-level change, we find anti-transfer: coherently negative ratios across four covalent semiconductors move the gap in the wrong direction, consistent with a molecular proxy and an r$^2$SCAN control. The minimal-basis overgap is dominated by the on-site convention rather than basis incompleteness; correcting the on-site block removes most of it, while one $d$-polarization shell closes a further $16$-$40%$, depending on the placement of the empty $d$ level, which no free-atom eigenvalue uniquely fixes. Occupied-manifold enhancements, ionic and closed-shell repulsive potentials, and rocksalt-oxide gaps inherit, whereas elemental and III-V covalent networks inherit neither gaps nor repulsive potentials and oxide networks inherit only the latter. We screen 23 elements and release the parameter sets, showing that the transfer ratio provides a cheap pre-test before any parameterization campaign.

0.04

Measuring Optimal Transport in Transformer Depth

cs.CL ·2026-09-01 ·Alexandre Quemy
published similarity 0.50
Abstract · Submitted at GDDL Workshop @ NeurIPS 2026

A transformer carries each token's state from layer to layer, and the whole vocabulary carried together forms a cloud that moves with depth. We ask whether a trained network moves this cloud the way optimal transport would: at the cheapest cost, and along the map that pairs each token with its optimal destination. We measure both on Pythia-160m and Pythia-410m, with an exact assignment between consecutive layer clouds, a measured sampling floor, calibration on couplings known to be optimal, and a split of the cost into the common shift of the cloud and the token-specific moves. At the last layer, both models move their tokens where the optimal-transport map sends them, at the optimal cost for Pythia-410m and slightly above it for Pythia-160m. At the first layer they do not. In between, single layers can be judged on cost at only two of ten transitions, and blocks of several layers move the cloud at close to the optimal cost. The agreement at the last layer is much weaker at initialisation (0.64 against 0.86) and grows with training.

0.48

What Should We Freeze? Guarded Freezing: Connectivity Shapes the Fine-Tuning of Pretrained Models

cs.LG ·2026-09-26 ·Leonel Aguilar
similarity 0.49
Abstract

When adapting pre-trained models through fine-tuning, freezing weights alone might not preserve performance, as updates elsewhere can change the inputs to the frozen core, ultimately affecting overall performance. We first analyse the case where a selected frozen core can be isolated and propose removal-value, a capacity-based score that approximates HOPE's removal cost averaged over removal orders. We show that in VGG-8, cutting paths from trainable neurons into a frozen core makes selection using this score useful: $70\%$ frozen preserves $5.22\pm0.51$ percentage points more old-task accuracy than DEFT at similar new-task accuracy. In transformers, shared residual streams leave paths into frozen neurons open. For this case, we derive drift-value, a forward-only proxy for the output disturbance from updating each weight entry under a local update model. In language models, at 40 epochs, this policy exceeds adapted Wanda and RIA freezing scores in settings with substantial retention loss, while its differences from Fisher remain unresolved. After 160 epochs on Qwen2.5-1.5B, it retains $0.0433\pm0.0102$ more than static Fisher. In DINOv3 vision-transformer adaptation to point clouds, drift-value retains $0.440$ image accuracy versus $0.187$ for a random mask of the same count. These results motivate Guarded Freezing: select by removal-value when incoming paths are cut, and by drift-value when they remain.

0.27

A Pre-Specified Construction-Confirmation Test of Operation-Level Causal Transfer Across Finite Isomorphic Symbolic Domains

cs.LG ·2026-08-16 ·Xinyi Shan
similarity 0.49
Abstract · 20 pages, 5 figures, and 4 ancillary CSV files

Behavioral accuracy, linear decodability, and successful activation interventions do not by themselves show that a model carries an operation-level structure from one symbolic domain to another. We ask a narrower question in finite isomorphic state spaces: if the hidden-state difference between two operations is estimated separately for each source input, does adding that difference to a mapped recipient input move the model toward the corresponding recipient answer? The design compares this input-specific intervention with wrong-operation, norm-matched random, and no-op controls, and separates candidate construction from an independently isolated confirmation split. On a frozen Qwen2.5-7B-Instruct model at layers 20--21, one route--domain--operation candidate from a family pre-specified and frozen before confirmation access, transparent | integer_mod16--letters16 | successor->predecessor, passed both PyVene splits; its confirmation intersection--union p-value was 0.000198 and its 36-family Holm-adjusted p-value was 0.006943. A subsequent NNsight 0.7.0 experiment, pre-specified and frozen before its confirmation access, tested only this selected prompt route, without candidate or layer reselection. It reproduced all 12 confirmation effect estimates, confidence intervals, and exact sign-flip p-values numerically; its 36-family Holm-adjusted p-value was 0.007141. The result is therefore limited to one prompt route and one candidate, replicated across two intervention implementations on one model revision and one layer interval. It does not establish cross-model generalization, full-family backend independence, domain-general transfer, or algebraic invariance.

0.10

CacheBridge: Efficient Cross-Model KV Transfer for Multi-Model LLM Serving

cs.AI ·2026-09-01 ·Xingyu Qu, Siyuan Lu, Zhiyu Chen, Sheng Wang, Tao Lin
highly cited similarity 0.48
Abstract

Multi-model LLM serving uses routers, cascades and agent workflows to move requests across models with different costs and capabilities. Existing serving systems reuse KV caches only within one model, so every switch forces the target to re-prefill history the source already processed, adding time to first token (TTFT) that grows with history length and target size. Cross-model KV transfer can replace this prefill, yet prior methods are slower than re-prefill end to end. We present CacheBridge, the first serving system to accelerate heterogeneous multi-model workloads through cross-model KV transfer. It co-designs a compact, attention-calibrated KV mapper with systematic optimization for online multi-model serving, which makes model switch efficient on latency and memory cost. In Qwen3 14B$\rightarrow$32B agent serving on $τ$-bench, it reduces mean TTFT by $30$ to $98\%$ versus re-prefill at $1$ to $24$ QPS with comparable task success. Across five model pairs from three families, it uses $8$ to $12\times$ less mapper memory than Closed-Form Mapper and retains on average $97.94\%$ and $102.94\%$ of re-prefill task quality on chat/reasoning and RAG workloads, respectively. The promising results have opened new opportunities for efficient multi-model LLM serving through cross-model KV reuse.

0.42

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

cs.AI cs.CLcs.LG ·2026-09-29 ·Zehao Jin, Ruixuan Deng, Junran Wang
code/data similarity 0.48
Abstract

Pretrained transformers use little of their depth to follow references in context. Thirteen base models reliably follow only 1.4-3.6 lines, and extra pretrained loops add little. A task-trained rank-8 LoRA at one early layer extends this computation with all model weights frozen. Qwen3-8B improves from 15.5% to 99% exact accuracy on 24-line chains; a longer-trained LoRA reaches 50 lines. Ouro-1.4B reaches 60 lines after four loops and at least 160 after eight. The LoRA starts a relay: program lines pass on their chain identity through a short range of middle layers. Frozen heads read progressively further up the chain, and removing parent-line attention stops the relay. A frozen-model measurement locates the last useful intervention layer within tolerance in three of four held-out models. Task-specific LoRAs also improve MuSiQue. Default answers therefore understate the computation accessible through a tiny edit. Code and an interactive demo are available at https://lunamos.github.io/stop-thinking-too-early/

0.33