CS Paper Radar

Shared relations, specific choices: conditional sensitivity and competing pathways in language models

Kaixiang Shu

cs.CL ·submitted 2026-11-01 11:25 UTC · v1

Abstract

How does a relational representation shared across queries drive a query-specific choice? Controlled entity-binding experiments with GPT-2 Small, Qwen2.5-1.5B and Llama-3.2-1B reveal a low-dimensional relational background that is predominantly shared across query roles and transfers to new names. Same-domain interactions persist when another attribute domain is queried. After removing this component, restoring independently trained relation directions brings output contrasts closer to their original values than sign-reversed or equal-norm random controls in all three models. Yet a transferable relational representation need not have a transferable functional sign. In Qwen2.5, fixing model weights, the training-mean relation vector and edit dose yields opposite selection effects across binary and lexical-mapping prompts with different answer vocabularies. A factorial contrast of downstream sensitivities across attribute-query combinations determines the shared direction's local signed selection gain. In a separate eight-condition protocol, mean local gain is positive in Qwen2.5 and negative in Llama3.2. Late-component clamps resolve opposing pathway contributions: in Llama3.2, negative transmission through selected attention-only paths is partially offset by positive transmission through paths traversing both selected attention and MLP sets. Source-coordinate analyses establish distributed factor coverage, hidden-receiver readouts reveal layout-dependent combination, and instance-geometry interventions separate response direction from amplitude. These findings show how conditional sensitivity and competing pathways give shared relational structure its effect on specific choices.

Closest papers

Nearest neighbours by meaning across the whole index.

When Direction Matters: Diagnosing Relational Instruction Following in Dense Retrieval

cs.IR ·2026-11-11 ·Zhanyu Wu, Richong Zhang
similarity 0.61
Abstract

Dense retrievers increasingly claim the ability to condition retrieval on natural-language relevance criteria beyond a short query. We identify a specific weakness in this capability: the evaluated models perform well when lexical or attribute cues distinguish the target, but become much less reliable when relevance depends on a directional relation over the same entities. We introduce SwitchIR, a controlled diagnostic that contrasts Constraint Shift and Explicit Filtering with Relational Flip, where the query and document pair remain fixed while the requested relation direction reverses. Across representative embedding and late-interaction retrievers, the primary joint metric is high on the two cue-available controls and consistently low on Relational Flip; the same ordering persists in an LLM-assisted, manually reviewed realistic variant. An exact margin decomposition shows that both directions are correct only when the direction-sensitive query--document interaction exceeds the direction-invariant pair preference. Empirically, direction-sensitive changes are common, but their aligned interaction is usually too weak to overcome that preference. Diagnostic fine-tuning shows that models can learn relational behavior that generalizes across relation families, with stronger generalization in the larger model.

0.13

Post-Training Leaves Behavioral Shadows on Unrelated Decisions

cs.CL cs.AIcs.LG ·2026-09-24 ·Ziyang Zhang, Yubin Jing, Yuanhao Zeng, Yuyao Li, Haofan Wang +1 more
code/data similarity 0.60
Abstract · 17 pages, 6 figures, 13 tables. Code: https://github.com/myboker/ATD

We find that language models can transfer capabilities through task-unrelated text. Post-training typically improves language models using task-specific data. Prior work on subliminal learning shows that information about these updates can pass through unrelated generations, but has largely focused on traits or preferences using extensive teacher outputs. We introduce Active Taskless Distillation (ATD), which achieves capability transfer using only a single word from the teacher per prompt. ATD probes the behavioral shadow of post-training by selecting prompts where the teacher and student's shared public ancestor is nearly indifferent between two ordinary words. A student initialized from this ancestor learns solely from the resulting prompt-word pairs, without target-task examples, teacher logits, or teacher parameters. In the primary coding experiment with Qwen2.5-1.5B, 5,664nses yield a 5.34 pp gain on HumanEval+ over an exact nuisance-matched control thadisrupts prompt-resperiments showtransfer in scientific knowledge, commonsense reasoning, and reading comprehensins across additional model generations, sizes, and families. Functional analyses show that the learned sid composable, andthat its strength tracks the teacher's update strength.

0.68

From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models

cs.CL cs.AI ·2026-10-13 ·Jinghan Zheng, Jiajun Shi
published similarity 0.59
Abstract · Accepted at AACL-IJCNLP 2026

Large language models rely on prompt context to perform in-context learning, follow formats, and adapt to local evidence. This ability, however, also raises a robustness concern: models may reuse irrelevant tokens from earlier context when answering a later query. Prior work identifies this behavior as contextual entrainment and finds attention heads whose ablation reduces it, arguing that these heads are largely task- or relation-specific. But are the mechanisms behind contextual entrainment truly isolated by relation, or do relation-conditioned searches recover reusable components shared across relations? Using factual-distractor relation-completion prompts across 19 lexical-semantic subtypes, we identify attention heads whose ablation reduces entrainment and test how the resulting head sets transfer recovery ability across exact relations, semantic families, and mixed relation distributions. Same-family transfer exceeds different-family transfer, family residuals weaken once globally shared heads are removed, and subtype-specific searches repeatedly rediscover many global heads. Clean-completion and context-use probes further suggest that this shared intervention can reduce harmful copying while preserving ordinary relation completion and simple in-context abilities. Our findings provide a more precise perspective on contextual entrainment and its mitigation: harmful context following is not best addressed by searching for isolated relation-specific circuits, but by using heterogeneous attack prompts to reveal reusable intervention targets whose effects are shaped by semantic relation structure.

0.39

Many Perturbations, Shared Failure Dynamics: Tracing Direction-Sensitive Perturbation Influence in Language Models

cs.CL ·2026-10-23 ·Zhibo Hu, Chen Wang, Yanfeng Shu, Liming Dong, Hye-young Paik +1 more
similarity 0.59
Abstract · 42 pages, 6 figures

Why can a task-preserving rewrite turn a correct answer into a failure while a larger internal change leaves the answer intact? We develop a direction-sensitive, readout-relative account of semantic fragility: heterogeneous language changes can share geometric and readout relationships without sharing an input-independent failure direction. Clean decision reserve, representation transformation, and response alignment provide complementary descriptions of this process. An exact longitudinal-transverse decomposition identifies directional information discarded by scalar transmission. A compact reserve-and-transmission model predicts failures across held-out families and items in Qwen3-32B and Gemma 4 31B, with useful but parameterization-dependent transfer to ReClor. Matched interventions show that orientation carries causal information beyond scalar geometry. Among naturally failed rewrites, attention-component tests on disjoint original items and subsequent answer-order controls favor the natural paired direction over tested random and same-answer-letter donor directions. Yet failures rarely persist across option orders, and current-order corrections use recomputed clean references. Shared predictive regularity thus coexists with context-dependent directional effects. Numerical path integration reduces, but does not eliminate, finite-patch approximation error within a fixed context; a larger discrepancy from the natural rewrite effect remains. A complementary fixed-CoT cue study extends this account to influence unacknowledged in recorded reasoning: readout associations transfer across domains, although the positive-mean causal criterion fails and signed-effect prediction remains post hoc. Together, these results support shared relations among reserve, transformation, and response while distinguishing local causal sensitivity from a complete explanation of natural failure.

0.32

Factual Access States in Diffusion Language Models: From Mechanism to Knowledge Editing and Retrieval

cs.CL cs.AIcs.IRcs.LG ·2026-10-20 ·Yuchen Guo, Zhuoyuan Mao, Yiwen Guo
similarity 0.59
Abstract

Factual recall is a lookup: some representation inside the network serves as the key, and what that key is made of decides which queries can retrieve the fact, what an edit must cover, and what a retriever can read. We call it the factual access state. In autoregressive (AR) models the subject position enriches attributes largely independently of the relation, and relation-specific selection happens later, at the readout position. In masked diffusion language models (DLMs) we find the state is assembled differently, and much earlier: in the first attention layer the wording of the relation is routed into the position of whichever entity is visible and the early feed-forward layers look the fact up on this fused representation. The state therefore varies with the relation's surface form and with the query direction, so the queries about one fact occupy a region of state space rather than a point, and an operation on the fact succeeds only for queries that enter it. We treat the state as an interface to a DLM's factual knowledge, and put it to two uses. Writing to the state gives a knowledge editor: from one declarative statement, Multi-View Edit fits one shared value per direction across that region and installs it in the early-MLP block, preserving source-prompt reliability while substantially improving paraphrase and reverse question answering, where AR editors gain near zero. Reading it suggests a complementary use: the state at each entity position can serve as a relation-specific retrieval vector with no training, returning both pieces of evidence for two-entity queries on which pooled, prompted and lexical retrievers almost never return both.

0.17

Mine or Yours? Self-Other Confusion in Latent LLM Communication

cs.CL cs.AI ·2026-10-08 ·Yunlong Xu
code/dataunusual similarity 0.58
Abstract · 19 pages, 7 figures, 4 tables. Code and protocols: https://github.com/BeibaiDraco/mind-connecting

If two minds were linked, would each still know which thoughts are its own? Language models are increasingly linked in this way: instead of exchanging text, one model reads another's key-value cache. Systems that join a foreign cache to the receiver's own use one of two layouts: the cache lies over the receiver's own context, at the same token positions, or in front of it. We give two copies of one model different private priorities and link them in both layouts. With the cache over its own context, the receiver gives a fact found only in its partner's cache in every answer. Yet in about half of its answers it gives the partner's priority as its own, in both Qwen3-4B and Llama-3.1-8B: as often as a receiver that ignored whose priority was whose, although the text in the cache names its owner. With the same cache moved in front, this falls to 5% in Qwen but only to 43% in Llama, although both models nearly stop naming the partner's priority as the one they are using. Where the cache sits matters, yet neither layout in use protects both models. Fewer such errors do not mean better attribution: once the cache is moved, the receiver names the partner's priority as the partner's less often, not more, and its report of an inserted memory cannot tell the partner's cache from a copy of its own. In short, shared caches deliver content but not ownership.

0.38

Graph-to-Table Interfaces Control Relational Generalisation

cs.LG ·2026-10-19 ·Alessandro Cagiano, Lorenzo Molfetta, Gianluca Moro, Matteo Zignani, Pasquale Minervini
similarity 0.58
Abstract

Graph-to-table systems combine three design choices: an encoder that maps relational evidence to feature rows, a context constructor that selects labelled examples, and a tabular decoder. We investigate whether these components can be evaluated independently through matched comparisons that change one component while holding the rest of the pipeline and the evaluation constant. Across logistic regression, CATBOOST, and TABPFN-3 on CLUTRR, 57 GraphLog worlds, and 57 knowledge graphs, we show that conclusions about one component can change when the others change. On CLUTRR, absolute positional encodings maintain all distinctions among the evaluated queries and fit the short-path setting nearly perfectly; yet, they generalise worse than relation-count features because the supported prefix acts as a shortcut that breaks down on longer paths. On GraphLog, encoder preference depends on the decoder, while the relation-count representation degrades as CatBoost tree depth increases. Across different knowledge graphs, adding the frozen ULTRA score to local features is consistently useful, whereas adding local features to the score has a smaller and less stable effect. A pointwise TABPFN-3 readout using the full encoder underperforms native ULTRA on 48 of 57 datasets. These results show that encoder assessment depends jointly on context and decoder, while panel-level conclusions can also depend on evaluation and aggregation protocols.

0.28

SemanticFold: Latent Sequence Compression SeparatesLanguage Modeling, Decodability, and Reasoning

cs.LG cs.AIcs.CL ·2026-10-07 ·Mingyan Liu, Min Huang
similarity 0.58
Abstract

We study whether latent sequence compression of prompt prefixes preserves the capabilities that large language models rely on during inference. We introduce SemanticFold, a compression scheme that folds prefix hidden states at learned boundaries, and evaluate it across five model scales: Qwen3-1.7B, Qwen3-8B, SmolLM2-1.7B, Pythia-1.4B, and Pythia-6.9B. We use a fixed-target protocol: a frozen prefix is executed natively or compressed, and both arms teacher-force identical continuation tokens. This design rules out target-selection explanations for likelihood changes. We examine five endpoint families: fixed-target negative log-likelihood, finite-label reasoning accuracy, linear probe accessibility, open-ended generation, and systems-level memory and latency. We find that compression moves these endpoints non-monotonically and that they do not share a single compression threshold. On Qwen3-1.7B at compression ratio R=1.7, compressed-minus-native mean NLL decreases by 0.135 under paired bootstrap with 10000 draws. On SmolLM2 at R=1.2, the mean change is 0.013 higher than native. On both Pythia checkpoints, NLL is effectively unchanged. An NLL decomposition separating sequence shortening from the learned residual transform shows that the favorable Qwen likelihood is attributable primarily to residual adaptation rather than to shortening alone. MLP-only, which applies the transform without shortening, achieves 0.082 lower NLL than Full SemanticFold. Linear probe accuracy and macro AUC change by less than 0.03 in absolute value across conditions, with confidence intervals crossing zero. We conclude that preservation under latent compression has no single scalar certificate: language-model fit, decodability, and reasoning behavior answer different questions and can move in different directions under the same compression operation.

0.13