CS Paper Radar

Measuring Definition Sensitivity in LLM Annotation of Social Science Concepts

Bogdan Mamaev, Qizhou Wang, Christopher Leckie, Matteo Vergani

cs.CL cs.CY ·submitted 2026-11-09 23:04 UTC · v2
published

Abstract

In social science research, text annotation is a fundamental task that is traditionally a costly, hard-to-scale manual process. Researchers have used large language models (LLMs) to perform annotation by writing a definition of a concept into a prompt - a rule that determines whether a label should be attributed to a text depending on its content. However, concepts such as hate speech, fake news, and terrorism typically have several theoretically defensible competing definitions. It is also unknown whether the model actually applied the definition to provide the label or the label was based on the model's pretrained notion of the concept. To test this, we have the model annotate the same texts twice, each time with a different definition. We call the disagreement between the two sets of annotations definition sensitivity. As a control, we paraphrase one definition but leave its requirements unchanged - the disagreement between the annotations with the two wordings becomes the paraphrase floor. Across six concepts and three LLMs, a swap of competing definitions always changes more annotations, on average, than a paraphrase. This shows that the annotations depend on the prompted definition, so any set of annotations should be interpreted together with the definition that produced it. The LLM annotator's definition sensitivity and its paraphrase floor should both be reported in such work.

Comment
Accepted at the NeurIPS 2026 Workshop TAE: Can We Trust AI Evaluation? Robustness, Causality, and Risk in Modern AI Assessment

Closest papers

Nearest neighbours by meaning across the whole index.

Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark

cs.CL ·2026-08-24 ·Martin Wessel, Timo Spinde, Jürgen Pfeffer, Gianluca Demartini
top venuebenchmark similarity 0.70
Abstract · To appear in Findings of the Association for Computational Linguistics: EMNLP 2026

Media bias detection relies on definitions and examples that specify what counts as bias, yet these specifications often vary across datasets or remain implicit, even when given the same name. Such variation makes it unclear whether models trained for the same bias category learn the same construct or different phenomena, a problem largely overlooked in prior work. We examine how definition choice affects bias annotation in a between-subjects experiment with 354 participants and a parallel evaluation with four LLMs. Participants and models rate six news articles across four bias categories using definitions that vary in conceptual framing and elaboration. Across 8,496 human and 28,800 LLM ratings, we find that the conceptual target of a definition drives annotation divergence, while construct-preserving elaboration does not: conceptual framing significantly shifts annotations for humans and does so even more strongly for LLMs. We discuss implications for construct specification in annotation protocols and prompt-based measurement, and consider how definitional sensitivity may propagate to downstream classification beyond media bias. We also release MUDD, the Multi-Definition Bias Detection Dataset.

0.46

When the Text Is Silent: Memory Shortcuts and the Validity of LLM Annotation under Evidence Insufficiency

cs.HC cs.CL ·2026-11-01 ·Wenchang Wang, Yue Song
similarity 0.70
Abstract · 15 pages, 5 figures

Large language models (LLMs) are increasingly used to annotate text for social science research, content moderation, and training data. Their broad knowledge of real-world entities raises a concern: when labeling a text, do they rely on its evidence or substitute what they already know? We call this behavior a memory shortcut and examine when it emerges, what amplifies it, and how to mitigate it. Using 995 widely recognized public figures and four matched control names per figure, we construct counterfactual texts that separate textual evidence from prior knowledge, and evaluate three commercial LLMs on 121,594 prompts each across five stages. We find a striking asymmetry. Although all models recognize nearly every figure, none follows memory over the text when the two conflict. Shortcuts instead emerge when evidence is absent: for one model, famous names raise the rate of remembered occupations, ages, and countries by 17.0, 14.9, and 6.8 percentage points relative to synthetic names. These effects occur only for recognized figures, and 98.6% of the resulting labels are factually correct about the namesake while none is supported by the text. Evaluation threats and answer-contingent rewards amplify shortcuts by 8.7 and 12.4 points. Famous names also raise leadership ratings of sparsely documented applicants by 0.22 and 0.28 points in two models. A namesake reminder and name masking each eliminate the shortcuts, with trade-offs in gender annotation. LLM annotations can thus be factually correct yet methodologically invalid. Trustworthy LLM annotation requires audits that test evidence absence, account for incentives in annotation instructions, and evaluate the attribute-specific trade-offs of mitigation.

0.25

Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages

cs.CL cs.CYcs.MM ·2026-10-13 ·Irem Ecem Yelkanat, Faeze Ghorbanpour, Alexander Fraser
similarity 0.65
Abstract

Annotator disagreement is increasingly recognized as a meaningful property of subjective tasks, rather than merely noise to be removed. Predicting which items are likely to be contested could help allocate annotation effort and route difficult cases to human review. We propose predicting disagreement from interpretable semantic properties of the text. An LLM answers a set of binary questions covering harmful-content characteristics, disagreement-related cues such as borderline severity and missing context, and language-sensitive cues such as local slurs. A classifier predicts disagreement from these answers, and we analyze which questions contribute most to the predictions. Across four datasets in English, German, and Turkish and three open-weight LLMs, our structured question features outperform direct prompting by 10.5 ROC-AUC points and the strongest text-based baselines by 5.0 points. A compact subset of eight questions performs within 1.5 ROC-AUC points of the full twenty-question set. In leave-one-dataset-out evaluation, models retain roughly 90-95% of their in-domain ROC-AUC. Across models and datasets, questions about derogation, missing context, and decision boundaries serve as the strongest predictors, suggesting that what makes harmful content contested is partly observable in the text.

0.27

Observational Equivalence of LLM and Human Annotation

cs.CL ·2026-08-24 ·Kentaro Nakamura, Jing Ling Tan, George Yean
similarity 0.65
Abstract

In this paper, we show that humans and large language models (LLMs) are observationally equivalent in coding texts. Recent LLMs agree with attentive humans at rates comparable to those observed among humans, and cross-LLM disagreement concentrates on texts that also divide humans. We demonstrate these patterns through replications of text-classification tasks from 14 peer-reviewed political science studies, with 13 LLMs, three human coders (Ph.D. students), and 165 crowd coders, independently applying identical codebooks. We further find that clarifying an underspecified codebook increases agreement among humans and between humans and LLMs. Together, these findings point to ambiguity in texts and coding rules as a shared source of disagreement and challenge the presumption that human labels should take precedence whenever humans and LLMs differ. We therefore argue that the central challenge of text annotation is not simply choosing between humans and LLMs, but developing coding rules that minimize ambiguity and accounting for the ambiguity that remains. To this end, we propose using cross-LLM disagreement to identify difficult cases, guide human review, and refine codebooks, and we develop ambiguity-aware bounds for downstream inference without requiring ground truth for every text.

0.17

Definitions, Label Names, or Demonstrations? Auditing What Governs LLM Annotation

cs.AI ·2026-11-06 ·Viraj Mishra, Shubham Gupta, Dhruv Gupta, Yuehong Cassandra Tai, Qunhua Li
similarity 0.65
Abstract

Large language models (LLMs) are increasingly used to annotate social-science texts, but high accuracy does not establish that they apply researcher-supplied category definitions: predictions may instead reflect prior associations with category labels or labeled demonstrations. We introduce a counterfactual audit that reassigns definitions among output labels, distinguishing adherence to supplied definitions from preservation of a dataset's original mapping. Across three political-text datasets, models rarely follow reassigned definitions, and standard fine-tuning strengthens the original mapping without reliably increasing definition adherence. We then study retrieval-augmented annotation. Retrieving the nearest training examples generally outperforms retrieving examples separately within each category despite using fewer demonstrations. On the two tasks with substantial gains, direct \(k\)-nearest-neighbor classification recovers 74--93\% of the improvement, indicating that neighborhood labels provide much of the predictive signal. Relabeling fixed demonstrations redirects predictions, whereas aligning demonstrations with reassigned definitions substantially increases adherence to the new mapping. Thus, accuracy-improving interventions need not increase codebook adherence; definitions, category labels, demonstrations, retrieval, and training data should be aligned and validated jointly.

0.25

When Humans Leave Context Unsaid: Implicit Context as a Risk for Large Language Models

cs.CL ·2026-08-10 ·Bocheng Chen, Han Zi, Yuda Chen, Yawei Liu, Zimo Qi +2 more
similarity 0.64
Abstract

Human language is grounded in pragmatics, necessitating implicit (extralinguistic) context to interpret language, e.g., world knowledge and social norms. However, such context is often implicit rather than explicitly expressed in language, requiring humans to infer the underlying contextual information necessary for language understanding. Because of its implicit nature and absence from textual corpora, it is often overlooked in the subsequent alignment process. In this paper, we demonstrate that the absence of such implicit context in training data is a risk for LLMs, and we refer to this risk as the implicit risk for LLMs. To exploit this implicit risk, we propose a method that guides LLMs to infer implicit contextual cues, such as presuppositions and implicatures, from prompts that imply social harm, and then leverages these inferred implicit context to elicit additional socially harmful content. Experimental results demonstrate that attacks exploiting the implicit risk we identify consistently outperform baseline methods by substantial margins across a diverse range of open- and closed-source models, underscoring the need for new defense methodologies that account for overlooked implicit context.

0.04

Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

cs.CL stat.ME ·2026-09-22 ·Thomas Reiter, Christoph Kern, Fedor Miasnikov, Sofiia Nikolenko, Rob Chew +2 more
similarity 0.63
Abstract · Accepted to "3rd Workshop on Uncertainty-Aware NLP" @ EMNLP 2026 (archival)

Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design produced high agreement (median Fleiss' $κ= 0.91$). Agreement fell when we changed the task design for the same tweets (median Cohen's $κ= 0.76$). Task design and model choice increased the variance of estimated prevalence by factors of 76.7 for offensive language and 110.6 for hate speech compared with sampling variance alone. Variation across LLM task designs reached 560-572 basis points, compared with 270-331 basis points across five human instrument versions. Confidence scores did not solve this problem. They tracked repeated model outputs more closely than agreement with human labels, and grouping six tweets in one prompt lowered mean offensive-language confidence by 660 basis points. We call the variation caused by task design and model choice instrument uncertainty. Researchers can measure it only by comparing reasonable task designs. Repeating one setup or relying on confidence scores cannot replace that test.

0.23

Evaluating Cultural Reasoning and Robustness in LLMs for Spanish Political Discourse

cs.CL cs.CYcs.SI ·2026-11-16 ·Jorge Valverde-Rebaza, Nathan Garcia-Larios, Karime Ruvalcaba-Perez, Alejandro Rendon, Santiago Sanchez-Ballesteros +1 more
published similarity 0.63
Abstract · Accepted for publication in Latin American Conference on Informatics (CLEI 2026)

Large language models (LLMs) are increasingly used to interpret, generate, and mediate public-facing language, which makes their reliability especially consequential in politically sensitive settings. Although sentiment analysis is a common task in this area, affective meaning in social media discourse is often conveyed through culturally situated and pragmatically indirect cues that are not well captured by label-centric evaluation alone. In this paper, we evaluate the robustness of LLMs for sentiment analysis of Spanish political discourse from social media, with emphasis on culturally situated affective interpretation. We introduce a diagnostic protocol that combines three independent human annotation views into Gold and Ambiguous subsets, and extends the corpus with a Synthetic subset built from sentiment-flipped counterfactuals selected through model-conditioned perplexity and LLM-as-a-judge criteria. Using this framework, we compare proprietary and open-weight models under zero-shot and few-shot prompting, focusing on predictive robustness across human agreement conditions and counterfactual shift. Our results show that annotation ambiguity and counterfactual perturbation reveal failure modes that conventional evaluation on the original corpus tends to obscure. GPT-4.1-mini remains the most stable among the evaluated models, whereas the two smaller open-weight models degrade more markedly under synthetic perturbations, and few-shot prompting provides only limited gains. Overall, the paper contributes a reproducible protocol for robustness-oriented evaluation in political sentiment analysis, an empirical characterization of model behavior under ambiguity and sentiment-flipped counterfactuals, and a publicly available Spanish social media dataset with three-way human sentiment annotations.

0.18