CS Paper Radar

On the Geometric Bottlenecks of Cross-Lingual Knowledge Editing

Aastha A K Verma, Anwoy Chatterjee, Sparsh Jain, Tanmoy Chakraborty

cs.CL cs.LG ·submitted 2026-11-07 10:33 UTC · v1
top venue

Abstract

Transformer-based autoregressive multilingual language models (LMs) are often assumed to store factual knowledge in an interlingua: a language-agnostic latent space. If this assumption holds, a parameter update applied in a source language during knowledge editing should inherently propagate to its semantic equivalents in target languages. However, empirical benchmarks show that cross-lingual propagation of edits often degrades substantially, a degradation that prior work treats largely as an algorithmic symptom rather than a structural bottleneck. In this work, we investigate whether representational anisotropy creates a structural bottleneck for source-only associative-memory editing, and perform a geometric dissection of the cross-lingual propagation gap. Using associative memory-based "locate-then-edit" methods, including ROME, MEMIT, and AlphaEdit, as analytical frameworks, we derive a closed-form decomposition of this gap into key-space propagation, parallel residual mismatch, and orthogonal residual mismatch. Our empirical findings show that cross-lingual propagation is primarily bottlenecked by a large orthogonal residual component, indicating that source-language edits often fail to span the target-language update directions required for successful transfer. Furthermore, we show that post-hoc correction using only source-language residual statistics encounters a large residual-prediction barrier. Finally, we perform a structural realignment via a targeted contrastive alignment objective and find that it substantially improves key-space propagation and partially improves cross-lingual edit transfer, while leaving value-space mismatch as a remaining bottleneck.

Comment
Accepted at NeurIPS 2026

Closest papers

Nearest neighbours by meaning across the whole index.

ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective

cs.CL ·2026-09-27 ·Rui Liu, Chenheng Zhang, Haoxuan Li, Zhouchen Lin
code/data similarity 0.67
Abstract · 31 pages, 8 figures

Large language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a manifold perspective, viewing it as a localized displacement of an edit sub-manifold within the global knowledge manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-manifold, and (ii) how to preserve the remaining manifold structure during the sub-manifold displacement process. Based on this perspective, we propose ManiEdit, a novel manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-manifold. Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space alignment. Experiments on two base LLMs and four unstructured editing benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit

0.35

ROSE: Risk-Aware Orthogonal Subspace Navigation for Lifelong Knowledge Editing in Multimodal Large Language Models

cs.CL cs.AIcs.CV ·2026-11-06 ·Lingyun Song, Ziyao Chen, Kang Pan, Xiaolin Han, Yifei Zhang +3 more
code/data similarity 0.66
Abstract

Multimodal Large Language Models (MLLMs) require continuous factual updates to stay current, yet lifelong knowledge editing remains a daunting challenge. The primary obstacle is the severe Locality Erosion and Catastrophic Forgetting, where sequential edits inevitably interfere with pre-trained capabilities and prior updates. In this paper, we propose \textbf{ROSE}, a \textbf{R}isk-aware \textbf{O}rthogonal \textbf{S}ubspace navigation framework for lifelong knowledge \textbf{E}diting. ROSE addresses forgetting by aiming to restrict parameter updates to the orthogonal complement of previously edited subspaces. Crucially, it introduces a risk-guided mechanism that utilizes a dynamically computed risk mask to safeguard specific factual parameter subspaces essential for model stability. During inference, ROSE employs a multi-granularity knowledge integration strategy, featuring prototype-anchored semantic gating and perturbation-driven subspace fusion to precisely activate edited knowledge for relevant queries while limiting interference on unrelated inputs. Comprehensive experiments on two prominent multimodal benchmarks show that ROSE is competitive with state-of-the-art approaches across five key metrics, while maintaining outstanding stability over $1,000$ sequential updates. Our code is available at https://github.com/sunlitsong/ROSE.

0.25

Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing

cs.CL cs.AI ·2026-10-02 ·Ding Wu, Ye Zhang, Haoyu Wang, Tianci Liu
similarity 0.66
Abstract · The first two authors contributed equally

Large language models (LLMs) increasingly serve as general-purpose interfaces to factual knowledge, but their parameters do not automatically reflect information that changes after pretraining. Knowledge editing (KE) provides a targeted alternative to costly retraining by modifying selected knowledge and preserving unrelated knowledge and general capabilities. Conventional KE uses structured factual triples, whereas unstructured KE (UKE) uses free-form passages containing multiple facts. Nonetheless, existing UKE editors exhibit a failure mode known as context reliance: edited LLMs can often reproduce the editing passage but fail to reliably recall its individual facts without the original passage context. We identify context-induced difficulty underestimation under the standard passage-level editing objective: later facts receive increasingly rich ground-truth context and consequently incur lower initial losses, making them appear easier to learn. In response, we propose FOVEATED, a plug-and-play framework that constructs focused views of each sentence by randomly shifting the Rotary Position Embedding (RoPE) positions assigned to the keys of its preceding context. The perturbation is applied during editing and removed afterward, leaving the model's native positional encoding unchanged at inference time. We instantiate FOVEATED for both direct-optimization and locate-then-edit editors. We theoretically analyze how FOVEATED counteracts context-induced difficulty underestimation and empirically demonstrate consistent improvements across five KE editors, two LLM backbones, and three benchmarks.

0.16

Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing

cs.CL cs.AIcs.LG ·2026-08-12 ·Tianci Liu, Zihan Dong, Tianchun Li, Yi-Chung Chen, Qiming Cao +6 more
similarity 0.64
Abstract

Large language models (LLMs) achieve remarkable performance across natural language tasks, yet they are trained on static corpora and their knowledge quickly becomes outdated in a fast-changing world. This motivates knowledge editing (KE), which updates specific knowledge in an LLM without changing unrelated others. Recent works move from structured knowledge triples toward unstructured KE (UKE), where the edit is a free-form passage that may state multiple facts at once. Nonetheless, existing editors inject such a passage yet fail to use it: the edited model can recall the passage, but can neither answer atomic questions about its facts nor compose them into multi-hop reasoning. We attribute this missing property, which we term composability, to editors' passive reliance on the fixed passage as the sole learning source. In response, we cast editing as a proactive self-distillation from a privileged in-context state of the same model, which requires no external supervision. We further reveal that due to the novelty of the injected knowledge, the pre-edited model's own rollouts rarely cover it, which limits the effectiveness of pure on-policy distillation. To close this gap, we propose HPSE, which builds a hybrid rollout that steps in to place missing facts onto the student's own trajectory precisely where its coverage fails, while staying on-policy elsewhere. We theoretically analyze HPSE's advantage over pure on-policy distillation, and empirically establish its plug-and-play improvements across four LLM backbones and two KE editors under various scenarios.

0.34

MuEdit: An efficient multi-task editing method towards inter-domain knowledge conflicts

cs.AI cs.CL ·2026-11-11 ·Songshi Liang, Ting-En Lin, Zixuan Huang, Yuchuan Wu, Yongbin Li +2 more
top venue similarity 0.63
Abstract · 30 pages,accepted by NeurIPS 2026

Large language models (LLMs) require frequent knowledge updates across heterogeneous domains, but existing editing methods fail when applied sequentially to multi-task settings. We trace this collapse to inter-domain null-space misalignment: heterogeneous tasks induce distinct Jacobian geometries, causing their common null space to shrink dramatically and leaving insufficient admissible directions for conflict-free updates. We propose the Conflict Index to quantify this geometric interference, and introduce Mu-Edit, which mitigates multi-task conflicts via (1) conflict-aware ordering to minimize cumulative interference, and (2) dynamic low-rank approximation to expand the null space when ordering alone is insufficient. Experiments across five functional domains and three backbones show that Mu-Edit significantly outperforms existing baselines in multi-task editing performance while preserving general capabilities, and remains effective under incremental task arrival.

0.22

Capacity Allocation at the Source: Sparse Target Optimization for LLM Knowledge Editing

cs.CL ·2026-11-08 ·Dongyao Chen, Jiayun Lei, Zhiyuan Ji, Zhiying Deng, Wei Liu +1 more
benchmarkcode/data similarity 0.63
Abstract · neurips2026

Knowledge editing has emerged as a promising approach for efficiently updating embedded knowledge in large language models (LLMs). It first computes an ideal target hidden state that steers the LLM toward the new output, and then edits the model parameters so that the original hidden state is mapped to this target. While most existing methods focus on the second stage of parameter editing in preserving other knowledge, we find that the upstream construction of the target hidden state also plays an important role in allocating the model's limited capacity space. We first establish a theoretical connection between interference in the parameter space and interference between target hidden states, showing that the two are positively correlated. Building on this insight, we propose a simple yet effective $\ell_1$ regularization method that prunes away unnecessary ``tentacles'' of each knowledge vector, retaining only the essential components. With fewer extraneous tentacles, each edit affects less other knowledge, enabling better coexistence. Extensive experiments across multiple LLMs and benchmark datasets verify the effectiveness of our method. Code: https://github.com/wintersdragon-c/STO.

0.30

Editing Large Language Models with Geometry-Aware Regularization

cs.CL cs.AI ·2026-11-04 ·Bingqing Liu, Wei Liu, Jun Wang, Xiaobo Sun, Yuhua Li +1 more
similarity 0.62
Abstract · 10 pages for main contents

Knowledge editing has become a key technique for updating knowledge in large language models. However, existing methods neglect the geometric structure of the knowledge representation space, leading to two key limitations: they cannot distinguish which pre-trained knowledge should be preferentially preserved (core vs. non-core) nor which new edits are inherently difficult (hard vs. easy). To address this, we propose GEdit, a geometry-aware framework built on a Low-Rank Plus Diagonal (LR+D) factor model that captures the intrinsic low-rank structure of Transformer FFN hidden representations. GEdit features two complementary mechanisms. First, the low-rank regularization applies targeted protection on a low-rank core knowledge subspace. Second, the adaptive regularization dynamically adjusts the regularization strength based on each edit request's editability: relaxing constraints for hard edits to ensure success, and tightening them for easy edits to better preserve pretrained knowledge. Extensive experiments on GPT2-XL, GPT-J, and LLaMA-3-8B under large-scale editing scenarios, including both mini-batch and fully sequential editing settings with up to 10,000 continuous updates, show that GEdit consistently and significantly outperforms state-of-the-art baselines in both editing success and knowledge preservation.

0.29

DynEdit: Dynamic Entropy-Guided Sequential Editing for Large Language Models

cs.CL cs.AI ·2026-10-26 ·Jinhu Fu, Yan Bai, Yihang Lou, Li Sun, Jiahong Wu +2 more
published similarity 0.62
Abstract · Accepted by main conference of NeurIPS 2026 (poster)

Knowledge editing aims to update outdated or incorrect knowledge in large language models without full retraining. Existing parameter-editing methods remain limited for long-sequence knowledge: single-point editors provide insufficient coverage for extended targets, while fixed chunk-based editors distribute editing effort uniformly across rigid predefined windows, failing to prioritize difficult, information-dense tokens. In this work, we show that token entropy is closely associated with both editing difficulty and semantic importance: high-entropy regions are harder to edit and often contain key factual content. Motivated by this observation, we propose DynEdit, a dynamic framework for long-sequence knowledge editing. DynEdit uses token-level entropy to guide non-uniform sliding, increasing the coverage of difficult high-entropy regions, and further applies entropy-guided token-level optimization to strengthen updates on information-dense tokens within each editing window. Experiments on three benchmarks and two backbone models show that DynEdit consistently outperforms competitive baselines across diverse long-form editing settings, achieving gains of up to +14.0 BLEU points on challenging paraphrase queries and near-saturated semantic scores on several diverse knowledge domains.

0.19