CS Paper Radar

Auditable Unlearning for LLMs

Piyush Tiwary, Anushka Srivastava, Prathosh A. P

cs.AI ·submitted 2026-11-04 02:58 UTC · v1
hot direction

Abstract

Machine unlearning aims to remove a user's data from a trained model after a deletion request. Existing methods are majorly evaluated on three axes - how well they forget, how much capability they retain, and how robust they are to membership inference attacks, but not on whether the provider can offer the user any evidence that the deletion occurred. To fill this gap, we introduce auditability as a fourth axis, defined as the separation between requested (forget set) and retained (retain set) data under a statistic the provider computes. We show that across eleven unlearning methods on TOFU, MUSE and RWKU no method satisfies all four axes, with different methods falling short on different axes. To address this, we propose a two stage method that can be used on top of any unlearning method, in the the first stage we propose ${GapWiden}$ which uses a hinge loss to increase the difference between audit statistics of forget set and retain set. In the second stage we propose ${GapVeil}$, an inference time rejection-sampler which uses the base model as the drafter and re-samples based on the audit statistics of the drafted response. We observe that using our method, auditability and leakage reach oracle parity for every unlearning method, and forget performance converges to the oracle; retain performance improves for nine of eleven methods but remains short of parity. Our results show that evidence of deletion or auditability can be constructed by design, and that treating it as an explicit requirement changes the viability of existing unlearning methods.

Closest papers

Nearest neighbours by meaning across the whole index.

Counterfactual Certification for Sequential Unlearning with Long-Run Performance Guarantees

cs.LG stat.ML ·2026-10-13 ·Hanna Benarroch, Jamal Atif, Olivier Cappé
similarity 0.69
Abstract

Certified machine unlearning ensures the proximity between the unlearned model and a model retrained without the data to be forgotten. Extending this framework to sequential deletion requests, typically arriving as groups of variable size (e.g., all records of one user or of coordinated users), is challenging. For unlearning methods based on fine-tuning on the retain sets, this leads to either degrading performance or increasing computation as the number of unlearning requests grows. For independent successive requests, we instead certify each model release against a counterfactual unlearning trajectory in which the group being deleted never took part. With a fixed fine-tuning budget per deletion, we show that, for smooth strongly convex objectives, this approach keeps the performance loss bounded, uniformly in the number of requests. Experiments with multiclass logistic regression on FEMNIST show that the resulting algorithm needs fewer corrective steps than existing sequential methods and outperforms training-time differential privacy, particularly when group sizes are heterogeneous.

0.17

Forget the Data, Keep the Capability: Switchable Unlearning for Retrieval

cs.CV ·2026-10-27 ·Jingong Chen, Kyle Gao, Guangyuan Li, Chulwoo Pack, Sijia Liu +1 more
theorycode/data similarity 0.68
Abstract

Privacy compliance requires re-identification models to selectively forget individuals while preserving their ability to recognize others. For this, we present MoME, a switchable framework that can adapt pre-trained backbones for privacy compliance, allowing the same model to switch between Full mode for standard inference and Core mode for unlearned inference. MoME splits one backbone into core parameters that preserve retrieval and zero-initialized parasitic parameters that carry the full-data knowledge, and training them under gradient isolation so the core never receives a forget-set gradient. Disabling the branch yields Core, which satisfies a request to delete a person's training data, and enabling it yields Full, which retains what the deleted data taught, so honoring the request is a flag flip rather than a retraining run. We prove Core independent of the forget set up to an audited set of coupling channels and introduce the rank-based Forgetting-Rank statistic to audit that independence in a trained model. On Market-1501, VeRi-776 and RegDB, Core reads at a retrained reference under three membership attacks while retrieving above it, and the retrieval advantage carries to ResNet and CLIP backbones. Code and weights are available at https://github.com/Jingong-Chen/MoME.

0.22

Client-Side Probing of Deleted Ridge Statistics in Federated Unlearning

cs.CR cs.LG ·2026-09-03 ·Yijun Quan, Giovanni Montana
similarity 0.67
Abstract

Federated unlearning aims to remove a client's data from a shared model without retraining from scratch. Some efficient systems make deletion exact by storing compact, additive summaries of the training features and broadcasting an updated linear classifier after every accepted change. We show that these broadcasts can also reveal the hidden summaries. A malicious client can submit known changes, use the returned classifiers to identify the server state, and compare states immediately before and after an isolated deletion. This exposes the deleted sample, class, or client summary and can enable its reinsertion. We characterize exactly when the observations contain enough independent information, give a matching optimal construction for unrestricted probes, and derive a more realistic estimator based on additions formed from the attacker's own data. On MNIST and CIFAR-10, high-precision broadcasts permit exact label recovery for every tested sample deletion with both probe types. Lower-precision broadcasts sharply reduce fine-grained recovery, and insufficiently diverse responses prevent identification altogether. Unrestricted probes are readily detected by their size; most individual attacker-data additions resemble honest batches, although we do not claim that the complete sequence is inconspicuous. The results identify a concrete privacy and integrity risk, its algebraic cause, and practical limits involving broadcast precision, update verification, response rate, and concurrent activity.

0.32

IDSISA: Influence-Distilled Sharded Training for Efficient Machine Unlearning

cs.LG cs.AI ·2026-11-04 ·Deeraj SK, Sadhana Devarajan, Tanmoy Hazra, Anupam Shukla
theory similarity 0.65
Abstract · This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may…

Machine unlearning removes the influence of specific training examples to comply with privacy regulations. Full retraining gives strong guarantees but is computationally prohibitive at scale; approximate methods are efficient but lack formal guarantees. SISA (Sharded, Isolated, Sliced, and Aggregated training) achieves certified deletion through data sharding and independent retraining, but random partitioning scatters forget requests across most shards, and isolated retraining degrades utility. We introduce IDSISA (Influence-Distilled SISA), which uses influence functions to concentrate forget-prone samples into few shards and distillation from unaffected shards to recover utility lost to isolation. The deletion guarantee inherited from SISA's retraining step remains exact and requires no probabilistic qualification; the distillation-based utility recovery is a separately quantified, empirically bounded approximation ($ε_{\text{cert}} \leq 0.015$ in expectation). We further formalize certification under sequences of repeated deletion requests, where cross-shard distillation can create dependencies between shards; we prove that a closure-based cascading retraining procedure restores the deletion guarantee to the full aggregated model under any deletion sequence. Experiments on ResNet-18/CIFAR-10, ResNet-18/CIFAR-100, BiLSTM/IMDB, TCN/IMDB, and a fully connected network on Fashion-MNIST show $14-91\%$ better selective forgetting, up to $50\%$ better utility retention in relative terms (with consistent percentage-point gains on the other architectures, where a relative baseline is undefined), and unlearning-time changes ranging from $-6.7\%$ to $+6.4\%$ depending on architecture, relative to SISA.

0.22

Why Forget-Only Unlearning Needs Memorization

cs.LG cs.ITstat.ML ·2026-10-07 ·Luka Radić, Vikrant Singhal, Amartya Sanyal
similarity 0.64
Abstract

Machine unlearning asks for a deletion algorithm whose output is close to retraining from scratch without the selected forget examples. In this work, we study forget-only unlearning, where the deletion algorithm receives only the trained model and the examples to forget, with no retained data or extra training information. We ask whether forget-only unlearning is always possible. We first show that this depends on the learning method: different datasets can produce the same trained model but require very different outputs after the same examples are removed. Using this observation, we derive lower bounds on how accurately unlearning can match retraining and instantiate them for several standard learning algorithms. We then ask what must be true when forget-only unlearning succeeds. To this end, we derive lower bounds on what an algorithm must memorize about the training data to handle arbitrary deletion requests. For simple threshold learners, the required information can be as large as the entire dataset, even though ordinary training keeps only one boundary point. Overall, our results show that information discarded during ordinary learning may be needed later for deletion, so models designed for forget-only unlearning may need to retain more information than standard training does.

0.18

Source-Free Class Relearning: Diagnosing Forgetting in Class Unlearning

cs.LG cs.CV ·2026-09-02 ·Zahra Dehghani, Pablo Piantanida, Mohammadhadi Shateri
similarity 0.63
Abstract

Class unlearning aims to remove a model's ability to recognize designated forget classes while preserving performance on retain classes. However, low forget accuracy after unlearning does not necessarily mean the class structure has been erased. Approximate unlearning methods can alter classifier decision boundaries while leaving recoverable structure in the representation. Prior work has shown that forget classes can be recovered, but existing approaches require real forget or retain samples, auxiliary data, or reference checkpoints. We study class relearning in a strictly source-free setting, asking whether a forget class can be recovered through a classifier-head update using only the unlearned model. Our approach rests on a theoretical analysis establishing a sufficient alignment condition under which a single gradient step on a synthetic probe set increases the expected logit margin of the forget class. Building on this, we propose a white-box Source-Free Relearning Audit (SFRA), which generates candidate embeddings in representation space and uses model-guided confidence filtering to construct high-confidence retain probes and low-confidence boundary-adjacent probes that are relabelled as the forget class. Gaussian sampling and Softmax confidence are used by default, while ablations with alternative proposal distributions and uncertainty criteria show that recoverability is not specific to these choices. To quantify recoverability, we introduce the Relearning Score (RS), which jointly measures forget-class recovery and retain-accuracy preservation, and report class-matched $Δ$RS relative to a retrained reference. Experiments on CIFAR-10, CIFAR-100, and TinyImageNet with ResNet-18, ViT-B/16, and Swin-T show that several unlearning methods exhibit substantial source-free recoverability, and that for a subset of methods this recoverability exceeds the matched retrained reference.

0.23

Error Accumulation in Second-Order Sequential Machine Unlearning

cs.LG stat.ML ·2026-11-02 ·Chen Shang, Haolin Zou, Binghe Zhu, Arian Maleki
code/data similarity 0.63
Abstract

Machine unlearning seeks to remove the influence of deleted training data without full retraining. In the sequential setting, deletion requests arrive over time, and repeated updates can accumulate approximation errors. In this paper, we study error accumulation in high dimensions, where parameter count $p$ is proportional to sample size $n$, for second-order unlearning methods under two privacy settings: (i) \emph{partial unlearning}, which allows unpublished internal states to be retained while certified versions of the model are released; and (ii) \emph{perfect unlearning}, which retains only published models and no hidden states. We also distinguish pointwise and pathwise certification, depending on whether the certification guarantee applies to each released model or the whole released sequence. We analyze conjugate gradient (CG) as an efficient solver for the inverse Hessian-vector product (IHVP) in Newton iterations. Our theory shows that a maximum of 1/2/2/3 Newton--CG updates suffice for successful certification and vanishing population-risk degradation of the published model, for pointwise partial, pathwise partial, pointwise perfect and pathwise perfect certification, respectively. Numerical experiments demonstrate accurate sequential tracking and substantial computational gains of Newton-based methods combined with conjugate gradient, over gradient-based updates, especially for ill-conditioned problems.

0.29

Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluation

cs.CL ·2026-08-21 ·Ayush Gupta, Hima Varshini Surisetty, Sreevidya Bollineni, Varad Ingale, Tuhina Tripathi +3 more
similarity 0.62
Abstract · 19 pages, 5 figures

Machine unlearning aims to remove the influence of targeted training data from a model while preserving its remaining capabilities, but evaluating whether such information has truly become inaccessible remains challenging. Existing benchmarks primarily assess unlearning under clean, non-adversarial queries, leaving open whether information that appears forgotten can still be recovered through strategic prompting. We address this gap through a unified evaluation of prompt-based and fine-tuning-based unlearning methods on TOFU using Llama-3.2-3B-Instruct, followed by an adversarial robustness evaluation of methods that perform strongly under standard metrics. We introduce Attack Success Rate (ASR), an LLM-as-judge metric that measures the fraction of adversarial responses whose leakage score exceeds $0.2$, and evaluate recovery across eight attack suites. Our results reveal a substantial gap between clean-query forgetting and adversarial robustness. Although several fine-tuning-based methods achieve Forget Quality above $0.91$, targeted information remains recoverable with ASRs between $72.8\%$ and $84.3\%$, close to the $87.5\%$ ASR of the unprotected base model. In contrast, clean multilingual reformulations yield only $2.95\%$ measured leakage. A manual audit further finds agreement between binary ASR decisions and human factual assessments in seven of ten cases, indicating that ASR provides a useful, though imperfect, signal of behavioral recoverability. These findings show that strong standard-metric performance alone is insufficient to establish robustness after unlearning and motivate adversarial stress-testing as a complementary component of unlearning evaluation.

0.45