CS Paper Radar

Decodability Is Not Validity: Evaluating Nuisance Sensitivity in Language Model Concept Probes

Ayal Klein

cs.CL cs.LG ·submitted 2026-11-05 10:47 UTC · v1
code/data

Abstract

Two concept probes can be equally accurate at predicting a concept from a language model's hidden states while their scores contain different mixtures of the target concept and a correlated nuisance attribute. In four target--nuisance settings (two aspect sentiments in CEBaB; occupation and gender in BiasBios; Amazon review sentiment with negation, and with first-person framing), two LLMs, and a population of concept extractors, we compare decodability with discriminant evidence: the score's dependence on the nuisance at fixed target, and its response to input edits intended to change only the nuisance, filtered by two LLM judges and compared against placebo edits. Under correlated training data, extractors that are statistically equivalent in held-out AUROC differ up to several-fold in nuisance sensitivity in three of the four settings, and decodability is not a reliable proxy for it: their rank correlation changes sign across settings. The effect follows the probe's training data, not the nuisance's natural association with the target: first-person framing, unrelated to sentiment in natural reviews, is absorbed at least as much as negation, which co-occurs with negative sentiment, and decorrelated training removes the systematic effect. Selecting extractors with nuisance-aware development diagnostics, which need nuisance labels but no counterfactuals, lowers counterfactual nuisance sensitivity wherever the default choice is sensitive, at a small accuracy cost. Concept measurements should be validated, not just decoded.

Comment
4 pages + appendix; code and data: https://github.com/kleinay/decodability-is-not-validity

Closest papers

Nearest neighbours by meaning across the whole index.

Veracity labels flip the emotional tone of language models but leave racial disparities intact

cs.CY cs.CL ·2026-10-08 ·Nouar AlDahoul, Myles Joshua Toledo Tan, Yasir Zaki
code/data similarity 0.63
Abstract · 14 pages, 7 figures; Supplementary Information (42 pages, 6 figures, 27 tables) included. Data and code: https://github.com/Nouar-NYUAD/llm-…

Large language models (LLMs) increasingly deliver information through personas, but it is unclear whether correcting the facts also makes their framing consistent across identity labels. We audited four LLMs (GPT-5.4 mini, NVIDIA Nemotron 3 Nano, Llama 4 Scout, and Claude Haiku 4.5), which rated simulated affect on six dimensions (mood, anger, fear, confidence, perceived control, and mental state) in response to fact-checked false and true headlines presented without and with a veracity label. Experiment 1 evaluated 1,000 false and 1,000 true headlines without personas; Experiment 2 evaluated 300 of each under 12 personas crossing race, gender, and age. The label strongly changed the emotional tone: ordinal logistic regressions with headline-cluster bootstrap inference showed that refuting false headlines shifted ratings from mostly negative to mostly positive for GPT, Llama, and Claude (e.g., Claude's "confident" ratings rose from 22% to 99.6%), whereas Nemotron showed smaller, mixed shifts, and confirming true headlines produced only small shifts. The label did not, however, remove persona-associated disparities. The models assigned more negative simulated affect to Black than to White personas in 82 of 84 statistically significant comparisons across models and conditions, both without and with the label, and the disparity persisted on headlines unrelated to race; Asian and female personas were also predominantly assigned more negative affect, whereas age effects varied by model and condition. Factual correction and demographic consistency are therefore distinct properties of LLM behavior, and persona-conditioned systems should be audited for both before deployment in sensitive contexts.

0.39

Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages

cs.CL cs.CYcs.MM ·2026-10-13 ·Irem Ecem Yelkanat, Faeze Ghorbanpour, Alexander Fraser
similarity 0.62
Abstract

Annotator disagreement is increasingly recognized as a meaningful property of subjective tasks, rather than merely noise to be removed. Predicting which items are likely to be contested could help allocate annotation effort and route difficult cases to human review. We propose predicting disagreement from interpretable semantic properties of the text. An LLM answers a set of binary questions covering harmful-content characteristics, disagreement-related cues such as borderline severity and missing context, and language-sensitive cues such as local slurs. A classifier predicts disagreement from these answers, and we analyze which questions contribute most to the predictions. Across four datasets in English, German, and Turkish and three open-weight LLMs, our structured question features outperform direct prompting by 10.5 ROC-AUC points and the strongest text-based baselines by 5.0 points. A compact subset of eight questions performs within 1.5 ROC-AUC points of the full twenty-question set. In leave-one-dataset-out evaluation, models retain roughly 90-95% of their in-domain ROC-AUC. Across models and datasets, questions about derogation, missing context, and decision boundaries serve as the strongest predictors, suggesting that what makes harmful content contested is partly observable in the text.

0.27

Do Language Models Feel the Sentiment? Investigating Emotion Vectors

cs.CL ·2026-11-10 ·Ali Hassan Khan, Saad Bin Ather, Muhammad Saif, Maryam Bashir, Hajra Waheed
similarity 0.62
Abstract · 16 pages, 3 figures, 11 tables. Accepted at ExplAI-2026: 2nd Workshop on Advanced AI in Explainability and Ethics for the Sustainable Develo…

Emotion vectors are directions in a language model's internal representations that are associated with emotional concepts. Prior work shows that these vectors can influence model behavior, but their role in sentiment classification remains unclear. This study investigates emotion directions in Qwen2.5-0.5B-Instruct and Qwen2.5-3B-Instruct. Eight directions extracted from GoEmotions are evaluated through sentiment probing, activation interventions, and supervised correction. Emotion-only probes achieve 88.66% and 90.70% accuracy on binary TweetEval. In exploratory matched-ablation experiments, raw emotion-space removal causes 2.39 and 1.10 percentage points more accuracy loss than matched isotropic random-subspace removal for 0.5B and 3B, respectively. All four TweetEval raw and denoised effects are significant after Holm adjustment across eight comparisons. An exploratory SST-2 extension shows smaller raw excess losses of 0.66 and 0.57 points, with no effects surviving Holm adjustment. Frozen sentiment-correction classifiers are also tested on 20,000 Amazon Polarity reviews. Emotion-based correction improves accuracy over uncorrected Qwen, but does not consistently outperform margin-only baselines. These results distinguish sentiment information available in model representations, the effects of changing those representations, and their practical value for classification. They do not establish a uniquely emotional mechanism.

0.18

Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation

cs.CL ·2026-09-03 ·Danting Zhang, Bei Peng, Robert Loftin
top venuebenchmark similarity 0.62
Abstract · Accepted by EMNLP Findings 2026

Large language models (LLMs) demonstrate strong performance on standard content moderation benchmarks. However, these benchmarks often aggregate multiple moderation criteria into a single label, making it unclear whether models can disentangle them and reliably apply each criterion when making decisions. To study whether LLMs exhibit criterion-conditioned behaviour, we introduce Diagnostic Evaluation of COntent (DECO), a criterion-independent factorisation of content that enables controlled, criterion-level evaluation. We also introduce pairwise evaluation to compare model outputs across different criteria for the same input. Across four moderation datasets and four LLMs, we find that strong benchmark performance can hide substantial failures at the criterion level. Models struggle most when correct decisions depend not on overall harmfulness, but on the specific aspect of the content that the criterion requires them to assess. Our results highlight a key limitation of current content moderation benchmarks: strong performance on aggregated labels does not provide sufficient evidence that LLMs can reliably evaluate content with respect to individual moderation criteria. These findings call for the development of evaluation methods that explicitly measure criterion-conditioned behaviour.

0.42

Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis

cs.AI cs.CL ·2026-08-15 ·Paras Balani, Subhrakanta Panda
similarity 0.62
Abstract · 9 pages, 2 figures

Large language models are increasingly asked to analyze data and report what the results mean, a task distinct from the belief- or preference-alignment settings studied in most sycophancy research. We test whether editorial framing in the prompt, ranging from a neutral request to an explicit instruction to search exhaustively for reasons to discredit or to support a finding, changes not just the tone but the substance of a model's report. Across a 4 x 4 factorial design crossing four framing conditions with four ground-truth data patterns (a genuine effect, a confound that mimics an effect but fails a robustness check, a well-powered null, and an underpowered null), we collect 480 responses and score each along two independent dimensions: whether its factual claim about the data diverged from the correct interpretation, and whether only its tone diverged while the claim stayed correct. Factual misrepresentation is concentrated in two cells: brutally critical framing applied to a genuine effect, where the model talks itself into unwarranted skepticism (97% of responses), and significance-seeking framing applied to an underpowered null, where the model overstates confidence in a null conclusion the data cannot support (100% of responses). Tone shifts far more broadly than factual content does, with critical framing producing a defensive, hedge-heavy register across every data pattern regardless of what the data show, while significance-seeking framing shifts tone only where the data leave genuine ambiguity. A confound present in the data itself blocks both kinds of shift almost entirely under every framing condition tested. These results indicate that the risk of framing-induced distortion in LLM-assisted data analysis is neither uniform across framings nor uniform across data patterns, and that a model can hold a correct conclusion in place while its tone shifts substantially around it.

0.44

Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review

cs.SE ·2026-09-05 ·Haoran Yu, Lifei Liu, Danping Zhang
similarity 0.61
Abstract

Code review is a key quality checkpoint between AI-generated code and production. As AI coding agents submit pull requests at scale, it is unclear whether reviewers reduce scrutiny with repeated exposure and whether review comments reveal this change. We study 11,429 reviews from 400 repeat reviewers over 207 days, paired with 10,104 human-authored inline comments from AIDev. Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods (Wilcoxon p = 8.6 x 10^-8; Cohen's d = 0.25). However, four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles (all Spearman absolute rho <= 0.53, p >= 0.11; Bonferroni-corrected Mann-Whitney p >= 0.36). A logistic-regression classifier based on these features reaches F1 = 0.485, below the majority-class baseline. Sentence-embedding structure does carry signal: reviewers' late-period comment centroids shift farther from their early-period centroids than under within-reviewer random permutations (Wilcoxon p < 0.001), and a small MLP using three embedding statistics reaches F1 = 0.74 under 5-fold reviewer-stratified cross-validation. Granger analysis shows that approval-rate changes predict later shifts in technical specificity at all tested lags (p < 0.001), while the reverse direction is significant at only one of four lags. Reviewer adaptation is therefore detectable in latent distributional structure rather than in classical lexical metrics, and language shifts follow rather than precede changes in approval behavior.

0.23

When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text

cs.CL cs.LG ·2026-09-10 ·DongHyun Ryu, Jaehyeok Lee, YeongJun Hwang, JinYeong Bak
publishedcode/data similarity 0.61
Abstract · 15 pages, 4 figures. Accepted at W-NUT 2026. Code: https://github.com/dong4918-skku/Fable

Large language models are increasingly used as judges to measure social bias in text, yet the passages they judge are often noisy, containing typos, informal spelling, and broken punctuation. The consequences of such surface noise for social bias measurement remain unclear. To investigate this question, we apply five realistic noise conditions at multiple intensity levels to 3,822 stereotype-related responses and compare the resulting bias judgments with those on the original text. We find that such surface noise does not degrade bias measurement symmetrically: it is far more likely to turn neutral judgments into biased ones than biased judgments into neutral ones, by up to a 120x margin. We further observe two non-obvious effects across four LLM judges: in the most fragile judge the distortion is at its purest at mild, realistic noise levels, where erasure is scarcest, and as judges grow robust it attenuates toward parity rather than reversing. Bias measured on noisy text is therefore systematically overestimated, most in the categories that matter most for fairness.

0.47

Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items

cs.CL ·2026-10-29 ·Peng Xie, Amr Alanwar
similarity 0.60
Abstract

To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an item does not depend on the other items in the prompt or on its position among them. Across three judgement tasks (harm in social-media comments, short-answer grading and review star ratings) and five open-weight models with 4-9B parameters, assigning the same 2,000 items to different random batches of 25 changes 11-22% of decisions on harm and grading, 6.5-13.3 points more than renaming the item keys or rewording the instruction, while AUROC against human labels differs from single-item grading only by -0.033 to +0.010; for four models with 14-32B parameters, 6-15% of decisions change, 4.2-9.1 points more. We show that flip rates decompose exactly into components of order, slot and membership, and that under a threshold model they grow with the square root of the variance that the context contributes, so that effects that are small for scores are large for decisions. Most of the latent dissimilarity between partitions comes from the order of the neighbours and the slot. The main effect of which items share the prompt, the part that voting over permutations of a batch cannot remove, accounts for 0.11-0.22 of it, and majority votes over nine permutations still leave 52-60% of the flips. Batched verdicts also differ systematically from single-item verdicts: even the majority of fifteen random batches disagrees with them on 8.4-17.8% of decisions. Editing a single generation shows that both the model's own earlier verdicts and the neighbours' texts carry the effect. On the easy task, star ratings, only 2.9-3.9% of decisions change. We recommend grading ambiguous items one per prompt, recovering most of the cost with a cached prefix, and reporting agreement with single-item grading whenever items are batched.

0.26