CS Paper Radar

Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views

Joseph Lee, Yidi Huang, Dokyoon Kim, Shu Yang, Li Shen

cs.CL cs.AI ·submitted 2026-09-03 17:57 UTC · v2
top venue

Abstract

Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We find that auxiliary views, conceptual reformulations, are key to this process. We design controlled experiments to isolate this. First, holding the token budget fixed, allocating tokens from paraphrases to auxiliary views improves learning even for factual recall, which we use near-verbatim sentences from the original document to test. Second, the effectiveness of auxiliary views is not contingent on the strength of the model that generates them. Third, we establish that paraphrasing is helpful but only at smaller batch sizes. Fourth, we identify types of knowledge that aid learning in the presence of prior knowledge gaps. Finally, we examine how these effects manifest mechanistically via layer-wise biases and compression. Together, our findings suggest that auxiliary representations of knowledge are a key factor in the success of pre-training, and explains why data diversity matters. Further, our results provide clear recipes for synthetic data augmentation along with implications on the self-improvement in LLMs.

Comment
Accepted to Findings of EMNLP 2026; Updated with refined writing, new figures, corrected tables and plots

Closest papers

Nearest neighbours by meaning across the whole index.

Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

cs.CL cs.AI ·2026-08-12 ·Arda Uzunoglu, Benjamin Van Durme, Daniel Khashabi
similarity 0.73
Abstract

Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit assumption that training on longer contexts will only help the model by exposing it to richer evidence. We challenge this view by studying how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization. We propose the Information Abundance Paradox, which hypothesizes that abundant relevant information in the training context can reduce the incentive to encode that information parametrically, thereby increasing reliance on context. In pretraining with long documents, increasing the context window improves language modeling, natural language understanding, and closed-book MCQA only up to an intermediate optimum, after which performance consistently declines. In supervised fine-tuning, more task-relevant train-time context improves performance with supporting context, but reduces robustness when context is absent or misleading at test time. Our analysis suggests that this behavior arises when longer context provides a lower complexity solution. Mechanistically, training with informative context shifts gradient pressure from feed-forward networks, often linked to parametric knowledge, toward attention modules, and causal interventions show that this shift increases reliance on context during inference. Overall, these findings support the Information Abundance Paradox and suggest that scaling toward near-infinite context is not simply a matter of supplying more data, even when high-quality long-context data is abundant.

0.39

Augmenting Pretrained Language Models with N-gram Embeddings through Incremental Training

cs.CL ·2026-11-16 ·Shaohua Hu, Xusheng Gu, Haofei Yang, Minghao Shao, Nanhui Zhao +2 more
similarity 0.73
Abstract

As a new scaling technique for large language models, the n-gram embedding improves the base ability of a model while adding almost no inference cost. Mainstream research realizes it through joint training of the n-gram embedding with the backbone, which severely restricts the settings in which an n-gram embedding can be used. We therefore propose training an n-gram embedding onto an existing model through incremental training, to extend where the technique can deliver value. Specifically, we explore several settings of the n-gram embedding under incremental training---the learning rate of the embedding, the vocabulary size, and the injection position---and train an n-gram embedding onto the existing model under the best configuration. Experiments show that the best configuration under incremental training does not fully coincide with that under pretraining; moreover, the incrementally trained knowledge is written predominantly into the n-gram embedding, while the general ability of the backbone is retained throughout this training process.

0.15

KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training

cs.CL cs.AI ·2026-08-31 ·Meghanadh Pulivarthi, Kushagra Bhushan, Vineet Kumar, Gaurav Pandey, Jaydeep Sen +3 more
top venuecode/data similarity 0.73
Abstract · 18 pages. Accepted to EMNLP 2026

LLMs acquire vast amounts of knowledge during pre-training, but often lack the specialized knowledge needed to answer questions from niche sources such as manuals or technical documents unseen during pre-training. Continued pre-training (CPT) is widely used to inject such knowledge into model parameters. However, niche documents seldom repeat facts, making it difficult for CPT to robustly acquire such knowledge. Recent works address this by generating multiple paraphrases of the new knowledge, but paraphrasing is computationally expensive and typically requires powerful LLMs. In this work, we introduce KItCAT: Knowledge Injection via Corrupted Auto-regressive Training, a lightweight training strategy that reduces the need for paraphrasing in decoder-only LLMs. KItCAT augments standard next-token prediction by stochastically corrupting the input sequence. During training, a random subset of input tokens is replaced with other vocabulary tokens while the original next-token labels are kept unchanged. This simple intervention generates diverse training inputs from each sample, enabling large-scale data augmentation at negligible cost. We show that KItCAT consistently improves over CPT across multiple datasets and model families. Code is available at https://github.com/meghanadhpulivarthi/KItCAT.

0.55

Towards In-Parameter Memory Augmentation for Large Language Models

cs.CL ·2026-10-06 ·Haoyu Huang, Zhongwei Xie, Jiaxin Bai, Yisen Gao, Hong Ting Tsang +4 more
survey similarity 0.72
Abstract

Recently Large Language Models (LLMs) and LLM-based agents increasingly need to incorporate knowledge acquired after pretraining, e.g., domain facts, user preferences, documents, and interaction experience. In-context learning (ICL) and ICL-based agent harness remain flexible, but they consume context capacity and incur repeated discretized encoding cost that grows with context length. \textbf{In-parameter memory} offers a complementary substrate: reusable memory information is represented in model parameters, adapters, or other parameter-like objects that are composed into the forward pass at inference time. This survey focuses on methods that augment LLMs with such parametric memory at deployment: a memory-bearing parameter object is plugged into the forward pass during inference, whether it is acquired before or during deployment. We organize the landscape with two orthogonal axes: \textbf{Parameter Placement}, which includes Embedding, Attention, FFN layers, or Hybrid when two or more layers are used; and \textbf{Parameter Acquisition Time}, which distinguishes methods whose memory object is acquired during deployment (online) from those acquired before it (offline). We clarify boundaries, conduct comparisons, and discuss open directions in interference, safety, co-design with ICL, and recursive self-improvement.

0.36

Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data

cs.CL ·2026-09-15 ·Takayuki Yamamoto, Daisuke Kawahara
benchmark similarity 0.72
Abstract · 23 pages, 3 figures, 13 tables

Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source corpus. The stored knowledge, however, is not always retrieved correctly. We study the eliciting side rather than the storing side: we use preference optimization, which learns from pairs of a preferred (chosen) and a dispreferred (rejected) response, so that the model elicits its stored knowledge more accurately. One proposed approach takes the model's own erroneous response as rejected and the gold answer as chosen, so as to suppress the error. When the target knowledge is partially known, however, most of these rejected responses are factually correct. Using direct preference optimization (DPO) then pushes down rejected responses that contain correct knowledge and differ from the chosen answer only in style, such as length and wording. We propose style-debiased DPO (SD-DPO), which scores whether the rejected response of each pair is factually correct, inverts the preference of such pairs, and weights them so that the learning signal due to differences in style cancels out as a whole. We first test whether, on top of EntiGraph, a representative storing-side method that runs CPT on text synthesized from the corpus, our method adds accuracy efficiently. On QuALITY, the reading-comprehension QA benchmark on which EntiGraph was evaluated, SD-DPO exceeds a baseline we CPT on EntiGraph's synthetic data from the same base model and evaluate with the same procedure. The training tokens this requires are a few dozen times fewer than the additional CPT needed for the same gain. For knowledge updating, the main goal of this work, we use AToKE, a knowledge-editing benchmark for facts that change over time. There, SD-DPO reaches an overall accuracy of 0.982 and answers with the new or the old fact according to the queried period.

0.18

Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models

cs.CL ·2026-10-18 ·Rubin Wei, Siyuan Sun, Hao Sun, Yuliang Liu, Xiang Shi +3 more
community pick similarity 0.72
Abstract

Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level interpolation provides only shallow fusion. We introduce Deep Fusion Memory (DFM), a parametric memory architecture that reduces this construction workload and incorporates learned memory into intermediate hidden states. A causal memory predictor learns to generate memory from the preceding context, using retrieved chunk embeddings from the corpus as supervision. The predicted memory is shared across tokens within each chunk and fused into their hidden states through gated cross-attention. Memory prediction and fusion are trained end to end with the backbone frozen, and inference requires no explicit retrieval. Compared with existing parametric memory approaches, which often construct token-level memory targets, DFM requires $32\times$ fewer retrieval queries and $2048\times$ fewer retrieved items to construct its training data. Across six QA tasks, DFM performs better over the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Beyond QA, domain adaptation yields average gains of 6.51 and 4.38 points over Qwen3-4B-Base and Qwen3-8B-Base, respectively.

0.50

Representational Change Under Self-Consuming Training: A Sparse Autoencoder Analysis

cs.LG cs.AI ·2026-11-16 ·Suchit Gupte, Zhongteng Cai, Xueru Zhang, Mohammad Mahdi Khalili
similarity 0.70
Abstract

Large language models are increasingly exposed to model-generated text, raising the possibility that future models will be trained recursively on the outputs of their predecessors. Prior work on such self-consuming training loops has largely focused on output quality and downstream performance, leaving their effects on internal representations poorly understood. We study how representations evolve across five generations of self-consumption and whether sparse autoencoders (SAEs) remain reliable under this shift. At each generation, we compare a lineage trained on synthetic data with a matched control trained on real data under the same training settings. We evaluate how well SAEs trained before self-consumption transfer to later generations, measuring reconstruction and sparsity across SAE widths, architectures, and model layers. We then fine-tune the original SAE on each generation and use sparse probing to test whether task-relevant information remains concentrated in a small number of latents or becomes more distributed across the representation. We find that zero-shot SAE performance degrades progressively under self-consumption, with growing reconstruction and sparsity gaps that vary substantially across layers. Fine-tuning largely restores reconstruction, but not feature-level usefulness: task-relevant information remains accessible in the full latent space while becoming less concentrated in individual features. These results show that self-consumption induces representational drift and that reconstruction fidelity alone is insufficient to establish that an adapted SAE remains equally useful for interpretation.

0.18

How Depth Growth and Looping Shape In-Context Learning in Language Models

cs.CL cs.AIcs.LG ·2026-10-22 ·Ferdinand Kapl, Kaitlin Maile, Stefan Bauer, Johannes von Oswald
similarity 0.70
Abstract

Looped and depth-grown language models reuse computations across depth and have attracted interest for their reasoning capabilities. Building on prior suggestive evidence of improved in-context-learning (ICL), we investigate how such reuse favors ICL, including how it shapes the acquisition, retention, and mechanisms of this ability. Our analysis spans pretrained language models up to 3B parameters and controlled tasks. The experiments provide evidence for a shared inductive bias toward learning from context: grown and looped models rely more on information in context and, compared with a full-depth dense reference, acquire ICL earlier in controlled tasks and retain it more strongly when learning fixed associations in the weights is also possible. Tracing attention heads through training reveals that depth growth propagates early induction heads across layers. Ablations show that grown and looped models rely more strongly on induction-ranked heads for both few-shot performance and later-token prediction. Together, these findings provide evidence for a shared inductive bias toward learning from context and suggest that (soft) parameter reuse shapes how this ability develops and which attention mechanisms support it.

0.12