CS Paper Radar

Reusing Calibration Gains across Model Representations

Xu Cao, Wenxuan Xiao

cs.LG cs.DB ·submitted 2026-10-16 00:21 UTC · v1

Abstract

Scale calibration learns an update for one model representation. We study how much of that update can be reused by another representation of the same pretrained model. Our transfer procedure extracts positive group-wise gains and installs them in the recipient's weights or native quantization scales without recipient-side optimization. On a quantized recipient, it preserves integer codes, zero points and storage layout. For a Qwen3-1.7B floating-point/GPTQ INT3 pair, bidirectional transfer retains 76.9-90.5% of the text-loss improvement from recipient-specific calibration on fresh WikiText and C4 cohorts excluded from project development. Shuffling the same gain values within each projection worsens both recipients, showing that their learned group assignment matters after transfer. A Llama-3.2-3B control keeps the initial native tensors at all four calibrated projections identical and changes nine projections elsewhere. Cross-installation retains 98.6-99.3% of the text-calibration benefit, alongside a small, resolved preference for each recipient's own gain. These results quantify both the adaptation carried by a reusable calibration update and the remaining benefit of recipient-specific training.

Closest papers

Nearest neighbours by meaning across the whole index.

ResCom: Activation Residual Compensation for Large Language Model Quantization

cs.LG cs.AI ·2026-10-25 ·Ning Yang, Bohan Cui, Zhuowen Liu, Yuting Wang
similarity 0.62
Abstract · 25 pages (10 pages main text + appendix), 9 figures, 20 tables

Low-bit weight--activation quantization reduces the memory and computation needed to deploy large language models, but activation rounding can still degrade accuracy after calibration. Reconstruction from rounded inputs cannot distinguish activations within the same quantization bin. We propose ResCom, a low-rank correction that uses the activation rounding residual to compensate for the resulting layer-output error. An output-aware calibration procedure selects residual directions by their contribution to output reconstruction and computes the correction in closed form, without gradient training or updates to the calibrated weights. On Llama-3-8B with 4-bit weights and activations, rank-32 attention corrections reduce WikiText-2 perplexity from 7.708 to 7.147, improve MMLU by 2.89 percentage points, and raise five-shot GSM8K accuracy from 23.96 to 36.16 percent. Evaluations on models from 1B to 70B parameters and on held-out C4 test the method beyond this setting. The evaluated FP32 implementation adds 2.25 percent to 4K prefill time; a separate compact-storage implementation retains a 27.7 percent peak-memory reduction relative to FP16.

0.22

Activation-Aware Weight Tensorization: A Calibration-Time Preconditioner for Tensor-Network LLM Compression

cs.LG cs.AI ·2026-10-07 ·Alessandro Beatini, Marco Maronese, Emanuele Rodolà
similarity 0.61
Abstract

Post-training tensor-network compression replaces Transformer linear layers with Tensor Train (TT) or Tree Tensor Network (TTN) operators, but standard decompositions minimize weight-space Frobenius error rather than functional error under the layer's activation distribution. We propose Activation-aware Weight Tensorization (AWT), a training-free calibration wrapper that preconditions each weight matrix with a diagonal activation-derived scale before an unchanged TT/TTN solver and deploys the result with only an input-side elementwise rescaling. Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B, AWT consistently improves vanilla TT/TTN tensorization at 2-6 times compression: under single-operator replacement, AWT closes 12-35% of the WikiText perplexity gap to the dense baseline across the three model families and 2-6 times compression settings; while under multi-operator Llama suffix replacement it closes 27-60% across attention-group and all-seven-matrix settings. The gains also transfer to downstream HellaSwag and ARC-Challenge evaluations. We further show that diagonal preconditioning is a robustness-modularity tradeoff rather than a diagonal-covariance assumption: a dense full-covariance oracle wins its own weighted objective in 80/81 cases, yet diagonal AWT gives better held-out functional fidelity in 53/81 cases. Together, these results position AWT as a principled, modular preconditioner for improving functional fidelity in fixed TT/TTN compression pipelines without modifying the decomposition solver.

0.29

UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists

cs.AI ·2026-08-21 ·Ye Chen, Weining Zhang
benchmark similarity 0.60
Abstract · 23 pages, 12 tables, 3 figures

Organizations maintain task-specific adapters for open-weight language models, and each new base-model release forces a migration decision: retain existing specialists, port adapters, refresh from preserved behavior, or retrain. Prior transfer work evaluates isolated model pairs, without studying these choices across real model release sequences. We present UpgradeBench, a decision-driven longitudinal benchmark covering four consecutive Qwen releases, one continuation checkpoint, six tasks, and two model sizes, augmented by OLMo checkpoints with known training lineage. The benchmark disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether specialization assets transfer across versions, and what recovery resources are usable. We observe upgrade gains differ across task-scale-release episodes: some retrained baselines improve while others stay within training noise, with durability ranging from under one release interval for text-to-SQL to over fourteen months for intent classification. Direct adapter copying depends neither on architecture nor model family: on OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens; annealing and model souping introduce no extra harm, with portability decaying with continued-pretraining distance. Given preserved input data, teacher relabeling recovers target-base specialists without fresh gold annotations, though compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross-version adapter portability (Spearman 0.74 across eight model pairs). We release per-example predictions, cost logs, split manifests, and evaluation code.

0.33

Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

cs.CL cs.AIcs.LGcs.PF ·2026-08-21 ·Bakbergen Ryskulov, Iker García-Ferrero, David Montero, David Jansen, Ali Hashemi +3 more
similarity 0.59
Abstract · Patent Application Number: 26382838.6 / P202602102EP

Serving large language models cheaply increasingly means shipping models that are both structurally compressed to a fraction of their parameters and quantized to 4 bits. Together these steps degrade reasoning, mathematics, coding, and long-context behavior enough to require a recovery, or healing, stage before deployment. The default recipe, quantization-aware training (QAT), re-fits the compressed, quantized model to hard labels; in our pipeline it converged slowly and collapsed past its peak. We adopted Quantization-Aware Healing (QAH) instead. Because a structurally compressed model is never independently trained at full precision, its bfloat16 checkpoint is a distillation-recovered approximation of the original; QAH distills the 4-bit student directly from the original, uncompressed model. On a GPT-OSS 120B to 60B to MXFP4 pipeline, the QAH student matches or beats its bfloat16 source on 7 of 9 benchmarks at roughly 4 times less weight memory and half the teacher's parameter count, and is released open-weight as Hypernova-60B. Against a matched QAT baseline it reaches a comparable peak about 7 times faster and stays stable under continued training, without hand-tuned early stopping. We also report deployment lessons, including a large, reproducible quality gap between distributed-training backends. Our aim is a recipe deployable without a multi-week hyper-parameter search.

0.14

What a Weight Update Buys: Leverage Across Training Stages

cs.LG cs.CL ·2026-10-27 ·Shikhar Shiromani
code/datahot direction similarity 0.58
Abstract

Post-training moves a language model's weights by about a thousandth of their norm and changes its behavior sharply; pretraining moves them by a third and buys a slow drift. We introduce leverage, an exact measure of what a unit of weight change buys: the Fisher quadratic form of an update, computed with one forward-mode pass per direction, divided by the same form for random directions of the same size at the same checkpoint. Leverage orders released Olmo-3, Tülu-3, Zephyr, and OLMo-2 checkpoints over four orders of magnitude: pretraining windows sit at the random level on prose, SFT five times above it, RL-Zero and RLVR at 400-1,500, and DPO at 400-23,000. With this instrument we find that pretraining accumulates into noise: windows of 30k to 1M steps are no more consequential on prose than random directions of their size, as a noisy quadratic model of SGD predicts. We find that training precision changes what an update looks like: training bf16 parameters without an fp32 master copy freezes 36-99% of coordinates, lowers leverage up to sevenfold, and creates an alignment between update magnitudes and the diagonal Fisher that fp32 training does not show. We derive and verify a law for reuse: the sign-flip null predicts the relative damage of random drop-and-rescale (DARE) at keep rate $p$ as $(1-p)/(p\,ρ_{\mathrm{sign}})$, within a factor of two of measurement wherever DARE succeeds. Two controls make these readings reliable: every controlled SFT and DPO trajectory with fp32 master weights loses leverage as it moves, and two re-runs of a released DPO stage from its anchor with its published recipe land within 1% of the released displacement and 4% of its leverage, where a plain recipe reads five to six times lower; leaving one component out traces the level to the length-normalized loss and its batch size of 128 together.

0.36

When Do Task Vectors Interfere? Mapping the Validity Boundaries of Weight-Space Composition

cs.LG ·2026-08-10 ·Chencheng Zhu, Xiaoyang Li, Taotao Cai
benchmark similarity 0.58
Abstract · 12 pages, 3 figures; includes references and an expanded appendix

Task arithmetic composes skills by adding weight displacements, and merged models are then judged on benchmark suites. We measure when that composition is functionally additive, and find that the answer depends as much on how the model is prompted as on which tasks are merged. Across two-dimensional composition surfaces -- five model settings from 0.5B to 8B, two families, LoRA and full fine-tuning -- pairwise non-additivity is real, seed-stable, and transfers in coarse order to unseen task pairs: all eight preregistered sign predictions held. But it is input-conditioned everywhere we measured: the same merged model that shows a six-point interaction contrast on code prompts shows none on math prompts, and wrapping the identical code prompts in the instruction template the adapters were trained on collapses the contrast twenty-fold, from +6.9 to +0.3 points -- while re-serializing them in an untrained chat template leaves it intact (+12.5), falsifying our own preregistered prediction. Execution benchmarks (pass@1) inherit the training-format wrapper's blindness. Weight-space composition therefore supports coarse, input- and format-conditioned functional statements -- not a universal merging-performance predictor, and not one that training-format evaluations can see.

0.40

Few Bits, One Law: Toward W2A4KV2

cs.AI cs.CLcs.LG ·2026-10-06 ·Kai Yi, Tarek Elgamal, Sruthikesh Surineni, Vignesh Vivekraja, Soumyadeep Ghosh +1 more
similarity 0.58
Abstract

Extreme low-bit LLM compression is most challenging when weights, activations, and KV caches are quantized together: their distributions differ, and quantization errors interact throughout the network. We introduce CanonQ, a unified quantization-aware training framework that addresses these challenges by separating source canonicalization from task-aware adaptation. Fixed rotations and energy normalization map heterogeneous tensor sources to canonical coordinates, enabling frozen Gaussian-reference codebooks to be reused across layers and models. Joint training then adapts the network to the coupled errors of weight, activation, and cache quantization within a common scalar/vector interface. We bound frozen-codebook transfer error and local task loss, and derive an exact normalization-aware straight-through Jacobian that links quantization distortion to gradient bias. The strongest gains arise under joint W2A4KV2 compression: across LLaMA3-1B/3B/8B, CanonQ-Omni achieves up to 14.28x lower WikiText-2 perplexity and up to 57.9% higher mean zero-shot accuracy than prior state-of-the-art and representative quantization baselines. The benefits extend to Qwen3-1.7B, code generation, and mathematical reasoning: on instruction-tuned MobileLLM-Pro-1B at W2A16KV16, CanonQ achieves relative improvements of 41.7% in HumanEval pass@1 and 39.1% in GSM8K exact match over the strongest evaluated quantization baseline.

0.23

When PTQ Calibration Helps or Hurts: Dose-Driven Collapse and Validation-Gated Abstention in Quantized Text Embedders

cs.AI ·2026-10-19 ·Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun Jeon
similarity 0.57
Abstract · 9 pages, 4 tables, 2 figures

Learned post-training quantization (PTQ) calibration fits quantizer parameters on a small calibration set, and for a text embedder the natural supervision is retrieval relevance. We ask two questions that are usually conflated: how far to calibrate, and whether to calibrate at all. Across three embedders (e5-base-v2, BGE-M3, EmbeddingGemma-300m) at 4-bit weight quantization, 161 calibration runs, the outcome of a fixed procedure ranges from a gain to catastrophic degradation, and the driver is the dose of the update, measured as the functional displacement D_func of the quantized model from its uncalibrated start. On EmbeddingGemma the same learning-rate grid gives +1.47 to -71.53 points of in-domain NDCG@10 change; a label-free cosine objective collapses at the same dose (-69.52 vs -71.63), so the collapse is a property of learnable calibration rather than of retrieval supervision. Objective sensitivity is heterogeneous: on e5, ranking calibration is substantially more robust than cosine at matched displacement (+14.16 +/- 4.34 pp, 95% t-CI [+3.37, +24.96], three paired slates), whereas on BGE-M3 the ordering is directionally opposite but unresolved ([-5.18, +2.38]). Dose is the main axis but not a sufficient statistic: equal D_func reached along different paths differs by up to 1.61 pp. We therefore treat calibration as an optional intervention. A pre-registered rule that accepts a checkpoint only when a per-query paired confidence interval on a 25% calibration-validation split clears the uncalibrated baseline was run on 6 independent slates and five arms. It never selected a checkpoint below its slate's baseline (0 harmful selections in 30 decisions), retained 8 of 11 beneficial updates, and paid a mean oracle regret of 1.52 pp with one abstention that forfeited 11.13 pp, a safety-opportunity trade-off, not a free improvement. We report what remains open, including label efficiency.

0.24