CS Paper Radar

TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training

Zhipeng Xia, Haotian Xu, Siyu Yun, Liqi Lin, Hu Liu, Yu Li, Cheng Zhuo

cs.LG ·submitted 2026-08-31 13:33 UTC · v1

Abstract

LLM training is increasingly vulnerable to silent data corruption (SDC), yet existing protection methods largely treat Transformer computations uniformly because their vulnerability remains poorly understood. We present the first systematic characterization of SDC vulnerability across major computation interfaces in both the forward and backward passes of Transformer training. Our analysis reveals two distinct error propagation mechanisms: forward-pass vulnerability is highly location dependent, with faults on the Q/K path producing persistent training deviations, whereas backward-pass vulnerability is largely governed by gradient exponent distributions rather than computation locations. Motivated by these observations, we propose TrainSDC, a characterization-guided protection framework consisting of Q/K-path recomputation, residual-gain monitoring, and exponent-aware gradient scaling. Experiments on Llama 3.2-1B and Qwen3-0.6B show that TrainSDC maintains training behavior close to fault-free execution under both sparse and dense fault injection while introducing only 1.65%-6.76% runtime overhead.

Comment
12 pages, 5 figures, and 6 tables. Includes an appendix with additional experiments

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond the Diff - VulSpot: A Vulnerability Dataset Where No Vulnerability Goes Unlocalized

cs.CR cs.SE ·2026-11-11 ·Dyna Soumhane Ouchebara, Stéphane Dupont
benchmark similarity 0.64
Abstract · 5 pages, under review

A vulnerability detector is only as trustworthy as the benchmark that scores it, and today's real-world datasets tend to flatter detectors. They are often built once and never refreshed, so they quickly miss new vulnerabilities, their labels are either noisy or heavily filtered, and their random splits let models train on near-copies of test code and on future vulnerabilities. Above all, when a fix only adds code, such as a missing NULL check, they do not point at where the vulnerability was, preventing detectors from learning to localize some of the most common C/C++ weaknesses. We present VulSpot, a living benchmark for detecting, classifying, and localizing vulnerabilities, with over 500k functions in 21 languages from 24k registered CVEs, including over 15k vulnerable C/C++ functions. By linking the code added by a patch to the existing statement it protects, VulSpot locates the vulnerability in 99.3\% of the functions that previous diff-based datasets used to leave without a location. Rather than discarding uncertain labels, it tells users how reliable each one is, and it comes with leak-free splits, complementary code context, and hierarchical CWE labels. An automated pipeline refreshes VulSpot every month, so that each release brings vulnerabilities no model has seen yet.

0.12

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

cs.LG ·2026-09-29 ·Hadi Reisizadeh, Jiajun Ruan, Sijia Liu, Mingyi Hong
code/data similarity 0.63
Abstract

Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable content. In this work, we show that such evaluations can create an illusion of forgetting: even when output-level leakage is eliminated, sensitive information can remain encoded in the model's hidden representations. We first provide a theoretical analysis establishing a fundamental separation between output suppression and representational erasure. Specifically, we show that the decoder can be made arbitrarily insensitive to sensitive directions, driving output-level leakage to zero, while the hidden representations retain the underlying information. To empirically validate this phenomenon, we train generative probe decoders on hidden states across transformer layers, enabling layer-wise measurement of information leakage. Across three widely used benchmarks, TOFU, MUSE, and WMDP, and state-of-the-art unlearning methods, we find that substantial sensitive information remains recoverable from hidden representations, even when standard output-level metrics indicate successful unlearning. To address this gap, we propose Probe-Adversarial Representation Suppression (PARS), an unlearning objective that adversarially minimizes the extractable information from hidden representations. PARS directly targets representational leakage and provides significantly stronger guarantees of erasure under adversarial probing and relearning attacks, outperforming all evaluated baselines. Our results highlight a fundamental limitation of existing unlearning paradigms and suggest that true forgetting in LLMs requires controlling not only model outputs, but also the information encoded in hidden representations. Codes are available at https://github.com/OptimAI-Lab/HiddenStateUnlearning.

0.27

zkLLMPoT: Efficient Zero Knowledge Proof of Training for Large Language Models

cs.AI ·2026-10-06 ·Junkai Liang, Zhanpeng Guo, Pengfei Wu, Qingni Shen, Jiaheng Zhang +3 more
similarity 0.63
Abstract · 18 pages, 4 figures, Preprint

Auditing the claimed outcomes of large language model (LLM) training is challenging when model weights and training data are private, while cryptographically proving the full training process is prohibitively expensive at Transformer scale. We present zkLLMPoT, a zero-knowledge framework that certifies auditor-defined properties of a trained checkpoint through forward evaluation rather than verification of its optimization trajectory. zkLLMPoT includes 2 phases: 1) The trainer fixes the architecture and the model weights are committed. Then the auditor selects challenge sequences, preventing the trainer from modifying the checkpoint in response to the audit data. 2) Then the trainer proves the objective value attained by the committed model on those sequences. This formulation makes the certification cost independent of the number of training iterations, without revealing model weights or requiring access to private training data. We build on sumcheck- and lookup-based arguments to certify Transformer computations, while supporting next-token loss and task-specific audit objectives. Across four model families, operator-level benchmarks yield proving times of 41-59 seconds for 1.1-1.5B-parameter models and 131 seconds at 13B for the covered operators, with verification below half a second at a sequence length of 512.

0.38

Follow the Gradient: Tracing Safety Degradation Back to Training Data

cs.LG cs.AIcs.CL ·2026-10-27 ·Saket Reddy, Amro Abbas, Anshuman Suri, Ari Morcos, David Schwab +1 more
community pick similarity 0.63
Abstract

Fine-tuning Large Language Models (LLMs) on instruction data can quietly erode their safety, even when no training example looks harmful. Because this safety-degrading data looks benign, standard content filters struggle to catch it. We tackle this problem by developing SimpleLoRIF, a scalable gradient embedding method that identifies fine-tuning examples whose gradients align with unsafe behavior. By simplifying a low-rank attribution method and restricting backpropagation to the final four transformer blocks, SimpleLoRIF can embed datasets on the order of a million examples, boosting gradient attribution from a small-scale analysis tool into a practical filter for full post-training datasets. We first validate it on emergent misalignment, where we find that gradients can connect insecure-code training data to the broadly harmful behaviors that appear after fine-tuning, with AUC as high as 0.865 in the gradient space of misaligned models. We then use SimpleLoRIF to build SafeTulu, a version of Tulu 3 with the 20% of examples most gradient-aligned with unsafe responses removed. Training on SafeTulu reduces GCG attack success rate by up to 22% across Qwen, Llama, and Olmo, outperforming both LLM-as-a-judge and prior gradient-based filtering, while largely preserving or even improving capability. Finally, we analyzed the removed data and identify possibly under-studied sources of safety degradation, such as tool-execution failures and narrative arithmetic. We release SafeTulu along with a Hugging Face Space that lets researchers build their own customized versions of SafeTulu to fine-tune on, such as by changing the removal fraction or filtering only certain subsets.

0.29

Bait-and-Recover: Poisoning Internal Refusal Signals to Defend LLMs against White-Box Editing Jailbreaks

cs.CR cs.AIcs.CL ·2026-09-05 ·Tian Gao, Zhipeng Xie, Yuhao Wu, Junhua Liu, Xin Fang
publishedcode/data similarity 0.63
Abstract · 13 pages, 3 figures. Code: https://github.com/SparkShieldLab/bait-and-recover

Open-weight large language models face a low-cost white-box threat from representation engineering attacks. Attackers can estimate refusal directions and search for projection-matrix edits that suppress safety alignment while preserving general capabilities, within minutes on a single GPU and without gradient-based training. We propose Bait-and-Recover, a weight-level defense that places a bait adapter where attackers read activations and a paired recovery adapter at the subsequent layer. Trained via gradient routing, this decouples the observation path from the behavior path. By actively poisoning the residual signal used for measurement, Bait-and-Recover disrupts the attacker's edit search, while the recovery layer restores clean downstream computation. Across four open-weight models, our defense raises the minimum refusal rate against white-box edit searches from 16.25% to 71.75% under a strict behavior-preservation budget (KL <= 0.10), with negligible impact on general benchmarks. By invalidating the core measurement assumption of these attacks, observation-path poisoning offers a practical complement to behavior-level safety training.

0.48

Revisiting Cauchy-Schwarz Norm Bounds for SDC Detection in LLM Inference

cs.DC ·2026-10-13 ·Anton Altenbernd, Dylan Everingham, Philipp Wiesner, Odej Kao
similarity 0.62
Abstract

Large Language Models (LLMs) execute billions of floating-point operations per output token across many hardware accelerators. At this scale, hardware-induced faults can bypass system-level detection mechanisms and cause Silent Data Corruption (SDC). Algorithm-Based Fault Tolerance (ABFT) is the state-of-the-art technique for protecting matrix multiplication from such faults, but it incurs non-negligible computational overhead and requires thresholds to distinguish hardware-induced errors from floating-point roundoff. In this paper, we revisit Cauchy--Schwarz Norm Bounds (CSNB) as a low-overhead alternative for detecting SDC during LLM inference. We compare CSNB against ABFT through controlled fault injection and evaluate the runtime overhead of both verification and threshold determination across multiple model sizes, batch sizes, and generation lengths. Our results show that, on average, CSNB retains 99% of the fault detection capability of ABFT for the evaluated linear layers in dense LLMs, while reducing verification overhead by 87% and threshold determination overhead by 67%. For Mixture-of-Experts (MoE) router gate computations, however, CSNB retains only 55% of the fault detection capability of ABFT. These findings suggest that CSNB is a promising low-overhead alternative for protecting linear layers in dense LLMs, while checksum-based verification remains preferable for the evaluated MoE router gate.

0.13

Embedding Drift in Code Vulnerability Models Under Intended Behaviour-Preserving Transformations

cs.SE ·2026-09-18 ·Hasti Ghaneshirazi, Tahsin Reza, Ladan Tahvildari
similarity 0.61
Abstract

Code edits that preserve program behaviour can shift frozen code embeddings across classifier decision boundaries, causing correctly detected vulnerabilities to be predicted as benign. Such instability is problematic because semantically neutral changes, including comment removal, insertion of unreachable code, variable renaming, and loop rewriting, should not alter a model's security judgment. We investigate this issue using 15,000 C/C++ functions organized as 7,500 vulnerable-patched pairs from the Big-Vul dataset. Four mutation tracks are applied to both vulnerable and benign samples: comment removal, unreachable code insertion, variable renaming, loop rewriting, and a combined setting that applies all transformations. Using frozen microsoft/codebert-base embeddings with six downstream classifiers, we evaluate the robustness of vulnerability predictions and introduce a train-only defence that projects clean and mutated representations closer together while preserving vulnerability-class information. Under combined mutations, the baseline Vulnerable Flip Rate (VFR) ranges from 35.55% to 42.15%. The proposed defence reduces the mean common-set VFR from 22.60% to 11.32% and improves mean mutated accuracy from 55.33% to 57.10%. For Logistic Regression, VFR decreases by 17.29 percentage points, although benign-to-vulnerable flips increase by 11.95 percentage points. These results indicate that the defence improves robustness to semantics-preserving code transformations but introduces a false-positive trade-off and does not fully eliminate prediction instability.

0.15

The Role of Quantization in Loss Spikes

cs.LG cs.AI ·2026-10-28 ·Zaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer +2 more
similarity 0.60
Abstract

Loss spikes in language model pretraining can waste substantial compute and derail otherwise healthy training runs. A primary driver of these instabilities is the growth and divergence of output logits in the final language modeling (LM) head. In this work, we study a particularly costly form of this instability associated with output logit growth: delayed, terminal divergences that emerge only after prolonged training, which we call time-bomb spikes. We find that these instabilities are preceded by rapidly growing quantization error in the LM head, which eventually culminates in a loss spike. Crucially, reducing numerical precision accelerates time-bomb spikes, while increasing precision improves stability. We then propose a simple yet effective mitigation that replaces AdamW on the LM head with SGD with momentum, suppressing logit growth and improving stability over long training horizons. Finally, we find that existing mitigation techniques from the literature (along with our own) all reduce quantization error, suggesting a shared numerical mechanism behind their effectiveness. Together, our results identify finite precision error as a hidden driver of time-bomb spikes and show that suppressing this error is key to stable long-horizon pretraining.

0.21