CS Paper Radar

Selective Knowledge Edit Reversal via Gated Singular Vector Shrinkage

Weifeng Jiang, Ruirui Chen, Qianren Mao, Junnan Liu, Qili Zhang, Kwok-Yan Lam

cs.CL ·submitted 2026-09-02 04:32 UTC · v1
top venue

Abstract

Knowledge editing provides an efficient way to update factual knowledge in large language models. However, malicious edits may introduce safety risks, making it necessary to reverse undesirable editing effects. Existing reversal methods for parameter-modifying edits mainly focus on global removal, which may also erase beneficial edits that should be preserved. In this paper, we study selective reversal of edited knowledge, where the goal is to reverse targeted edited facts while preserving the remaining edited facts. Based on the hypothesis that each edit is sparsely encoded within the dominant subspace of the edited matrix, we propose a spectral-based reversal framework that locates edit-sensitive components within the dominant singular subspace of edited weights. Experiments across multiple settings demonstrate the effectiveness of our method in reversing selected edits while preserving unrelated edited facts. These results suggest that different edits are sparsely encoded within dominant singular components and can be separable when the number of edits is moderate, making selective spectral reversal a promising direction for locating edit-specific components and repairing edited language models.

Comment
Accepted to EMNLP 2026 Findings

Closest papers

Nearest neighbours by meaning across the whole index.

The Truth Beneath the Tokens: Prompt-Space Recovery of Knowledge Edits in Language Models

cs.CL cs.AI ·2026-11-07 ·Manit Baser, Rachel Yeo Hui Min, Aditya Nawal, Dinil Mon Divakaran, Mohan Gurusamy
similarity 0.64
Abstract

Model editing provides an efficient way to update factual knowledge in transformer models. Recent approaches reverse edits by modifying, masking, or decomposing model weights, but make recovery dependent on weight-space interventions. We instead show that pre-edit knowledge can be recovered through input-space interventions alone, without modifying the edited model's weights. We introduce Memento, a prompt-space mechanism for auditing residual knowledge that learns a reusable recovery signal to steer an edited model toward its pre-edit response. Memento supports both continuous recovery prefixes and discrete textual prompts for standard text interfaces. Across three LLMs and five editing techniques, Memento recovers 80.07% of held-out edited facts, achieving a 9.58% relative improvement over baselines, suggesting that editing may suppress the facts without fully eliminating their recoverability. We further investigate whether prompt-space recovery extends beyond model editing. On pristine, unedited LLMs, Memento improves average factual recall by 53.9% relative to standard inference. These results demonstrate the potential of prompt-space recovery for auditing residual knowledge in edited models and investigating the gap between factual knowledge and its accessibility during inference. We will release our source code.

0.22

Editing Large Language Models with Geometry-Aware Regularization

cs.CL cs.AI ·2026-11-04 ·Bingqing Liu, Wei Liu, Jun Wang, Xiaobo Sun, Yuhua Li +1 more
similarity 0.64
Abstract · 10 pages for main contents

Knowledge editing has become a key technique for updating knowledge in large language models. However, existing methods neglect the geometric structure of the knowledge representation space, leading to two key limitations: they cannot distinguish which pre-trained knowledge should be preferentially preserved (core vs. non-core) nor which new edits are inherently difficult (hard vs. easy). To address this, we propose GEdit, a geometry-aware framework built on a Low-Rank Plus Diagonal (LR+D) factor model that captures the intrinsic low-rank structure of Transformer FFN hidden representations. GEdit features two complementary mechanisms. First, the low-rank regularization applies targeted protection on a low-rank core knowledge subspace. Second, the adaptive regularization dynamically adjusts the regularization strength based on each edit request's editability: relaxing constraints for hard edits to ensure success, and tightening them for easy edits to better preserve pretrained knowledge. Extensive experiments on GPT2-XL, GPT-J, and LLaMA-3-8B under large-scale editing scenarios, including both mini-batch and fully sequential editing settings with up to 10,000 continuous updates, show that GEdit consistently and significantly outperforms state-of-the-art baselines in both editing success and knowledge preservation.

0.29

When Successful Knowledge Graph Edits Displace Correct Answers: Rank-Level Locality beyond Parameter Support

cs.AI ·2026-09-10 ·Yi-Cheng Lai, Jerry Wang, Hsin-Ling Hsu, Li-Chu Chi, Ya-Wen Teng +1 more
similarity 0.63
Abstract

Editing a knowledge graph embedding (KGE) model to promote a desired answer can displace correct answers from the returned list. Locality tests based only on facts that reuse the edited parameter can miss this ranking effect. We introduce a common rank-displacement audit at three scopes: facts supported by the edited parameter, other correct answers to the target query, and correct answers across queries with the same relation. We also derive dimensional and geometric conditions for an update to improve the target while exactly preserving selected scores. On FB15k-237 with DistMult and ComplEx, direct promotion always moves the target into the top ten, but does so without damage in only 23.0--23.2\% of edits. Strict preservation causes no measured damage, yet succeeds in only 1.3--1.4\%. Support-regularized entity editing gives the highest joint success, 36.3--37.7\%, while rank-truncated preservation reaches 32.8--34.7\% and reduces the mean number of displaced answers from about 14 to 1.2. Experiments across dimensions, scorers, ranking conventions, and a learned editor show that locality depends on both the protected scope and the editing mechanism. KGE editing should therefore report correction success together with the incidence and severity of rank displacement.

0.12

ALOE: Semantically Addressed Low-Rank Operators for Knowledge Editing

cs.AI ·2026-09-24 ·Zeyan Li, Hu Xu, Jianfeng Xu
similarity 0.63
Abstract

Knowledge editing changes what a model knows by modifying parameters so that a requested fact updates while unrelated behavior is preserved. This is usually treated as a write problem, but editing also involves an address problem: deciding which hidden states should receive the new residual. An update that activates too narrowly memorizes one prompt, while one that activates too broadly disrupts neighboring knowledge. Parametric editors encode this scope implicitly, whereas memory-based editors make the selection explicit but keep it outside the edited model. We propose ALOE (Addressed Low-rank Operator for Editing), which learns semantic addresses from paraphrases and hard same-subject negatives, aligns them with autoregressive hidden states through rollout refinement and gate calibration, and embeds the resulting gated low-rank operator within one MLP layer, so that the deployed model runs in a single forward pass with no external retriever or auxiliary router. Evaluated on CounterFact, ZSRE, and KnowEdit across three 7--8B model families, ALOE achieves efficacy between 0.955 and 0.999 and locality between 0.981 and 1.000; mechanistic analyses confirm that the learned geometry separates competing edits and that calibration suppresses out-of-scope activation. The remaining errors concentrate in paraphrase coverage and write fitting.

0.21

Leveraging Association Context Retrieval in Knowledge Edit- ing to Build White-Box Attacks on LLMs

cs.LG ·2026-08-18 ·Roman Maksimov, Vladimir Aletov, Vladimir Solodkin, Dmitry Bylinkin, Daniil Medyakov +1 more
similarity 0.62
Abstract

As large language models (LLMs) are granted increasing autonomy, it is essential to investigate methods that can induce unsafe behavior. We propose a novel white-box attack inspired by locate-then-edit approaches from the field of Knowledge Editing. Our choice is motivated by the observation that models edited with such schemes tend to assign unusually high prediction probabilities to the edit target, a property that is particularly advantageous when designing attacks. We modify the editing framework by incorporating as- sociative knowledge retrieved from the model, thereby extending constraint removal to an entire thematic category rather than being limited to prompts from a predefined dataset. Experiments with various archi- tectures demonstrate improved attack effectiveness over competing methods without dealing critical damage to general model performance.

0.21

From Spectral Saliency to Finite Deletion: Evaluating Selective Language Model Edits

cs.LG ·2026-10-13 ·Ibne Farabi Shihab, Joyanta Jyoti Mondal
similarity 0.61
Abstract · 25 pages, 9 figures

Deleting singular components offers a precise way to remove a capability from a language model, yet the first-order spectral saliency used to rank components predicts only a local loss change, whereas deletion is a finite intervention. In this paper, we study how prediction error affects selective editing, where selective editing requires degradation of a target task while preserving others. We decompose the error into derivative transfer across item splits and nonlinear response along the deletion path. We establish that outer interval screening retains every feasible candidate when every candidate's intervals contain its finite validation loss changes on every task. We evaluate these effects using one answer-only negative log-likelihood objective for search, validation, and test, with disjoint calibration and evaluation masks. The primary configuration targets arithmetic in Mistral-7B-Instruct-v0.3 while preserving code and question answering; extensions cover Llama-3.1-8B-Instruct and reversed target roles. In the primary study, which deletes 1024 coordinates from a support of 7168, calibrated intervals cover 968 of 1024 evaluation masks. Outer screening reaches the direct-selected edit after 25 verification checks, whereas direct verification abstains at a 32-check cap, but it rejects 4 of 74 feasible masks. All three selected edits pass validation, yet only one meets every test constraint. Interval coverage, feasible-candidate retention, and independent test preservation are therefore distinct properties and require separate assessment.

0.21

Capacity Allocation at the Source: Sparse Target Optimization for LLM Knowledge Editing

cs.CL ·2026-11-08 ·Dongyao Chen, Jiayun Lei, Zhiyuan Ji, Zhiying Deng, Wei Liu +1 more
benchmarkcode/data similarity 0.61
Abstract · neurips2026

Knowledge editing has emerged as a promising approach for efficiently updating embedded knowledge in large language models (LLMs). It first computes an ideal target hidden state that steers the LLM toward the new output, and then edits the model parameters so that the original hidden state is mapped to this target. While most existing methods focus on the second stage of parameter editing in preserving other knowledge, we find that the upstream construction of the target hidden state also plays an important role in allocating the model's limited capacity space. We first establish a theoretical connection between interference in the parameter space and interference between target hidden states, showing that the two are positively correlated. Building on this insight, we propose a simple yet effective $\ell_1$ regularization method that prunes away unnecessary ``tentacles'' of each knowledge vector, retaining only the essential components. With fewer extraneous tentacles, each edit affects less other knowledge, enabling better coexistence. Extensive experiments across multiple LLMs and benchmark datasets verify the effectiveness of our method. Code: https://github.com/wintersdragon-c/STO.

0.30

ROSE: Risk-Aware Orthogonal Subspace Navigation for Lifelong Knowledge Editing in Multimodal Large Language Models

cs.CL cs.AIcs.CV ·2026-11-06 ·Lingyun Song, Ziyao Chen, Kang Pan, Xiaolin Han, Yifei Zhang +3 more
code/data similarity 0.61
Abstract

Multimodal Large Language Models (MLLMs) require continuous factual updates to stay current, yet lifelong knowledge editing remains a daunting challenge. The primary obstacle is the severe Locality Erosion and Catastrophic Forgetting, where sequential edits inevitably interfere with pre-trained capabilities and prior updates. In this paper, we propose \textbf{ROSE}, a \textbf{R}isk-aware \textbf{O}rthogonal \textbf{S}ubspace navigation framework for lifelong knowledge \textbf{E}diting. ROSE addresses forgetting by aiming to restrict parameter updates to the orthogonal complement of previously edited subspaces. Crucially, it introduces a risk-guided mechanism that utilizes a dynamically computed risk mask to safeguard specific factual parameter subspaces essential for model stability. During inference, ROSE employs a multi-granularity knowledge integration strategy, featuring prototype-anchored semantic gating and perturbation-driven subspace fusion to precisely activate edited knowledge for relevant queries while limiting interference on unrelated inputs. Comprehensive experiments on two prominent multimodal benchmarks show that ROSE is competitive with state-of-the-art approaches across five key metrics, while maintaining outstanding stability over $1,000$ sequential updates. Our code is available at https://github.com/sunlitsong/ROSE.

0.25