CS Paper Radar

Invisibilization as a Source of Probability Bias in Language Models

Christian Arnold, Ernesto Calvo, Guadalupe Gonzalez, Sebastián Vallejo Vera, Tiago Ventura

cs.CL cs.CY ·submitted 2026-10-22 06:03 UTC · v1

Abstract

Language models are aligned to avoid using protected attributes such as race, ethnicity, and gender to inform their responses. Therefore, they increasingly refuse or hedge when such attributes appear in a prompt. However, if a system must decide how much to learn from what people tell it, it is costly to refuse valuable information because of its source. In this article, we measure the cost of declining information in a two-urn game with an exact Bayesian answer. We present leading LLM models from two developers (OpenAI and Anthropic) with a prompt that requires them to decide how far to trust an unverifiable report from a participant whose description we randomize across ten configurations (6,600 sessions). We show that leading models often decline to assess the reporter, treating information as worthless at rates set by the description rather than by the evidence. At its widest, models decline a Black-described reporter 46% of the time and a white-described reporter 9%; yet, when they do assess the information, the Black-described reporter receives slightly more trust. The bias is invisible to audits that compare average trust or look only at assessed cases. It grows with reasoning effort, follows words rather than categories (models invoke race for "Black" but not for its Spanish translation negro, and never for "Indigenous"), and worsens under warning instructions a practitioner would try first. We also present a scoring option that reduces the refusal rate bias. When a model's trust is scored against revealed truth over repeated rounds, it credits identical records equally. In four of five configurations from Anthropic and OpenAI, the racial gap in trust falls below 0.02 by the last rounds.

Comment
17 pages, 4 figures, 2 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Fairness in Explainable AI Narratives: Decomposing Social Disparities in LLM-Generated Explanations

cs.CL ·2026-11-09 ·Wannes Van den Bulck, Sofie Goethals, David Martens
code/data similarity 0.66
Abstract · 25 pages, 8 figures, 9 tables. Code, prompts, narratives and extractions available at https://github.com/WannesVandenBulck/fairxain

Large language models (LLMs) are increasingly used to translate technical explainable AI (XAI) artefacts such as SHAP into explanatory narratives (XAINs) for lay users. In high-risk domains, these explanations are often the main resource through which individuals can understand and contest adverse decisions. Yet LLMs are known to reproduce social bias, even from proxy variables alone, and fairness in XAINs remains unstudied. Because XAINs are bound to a ground-truth artefact, fairness concerns not only what a narrative says and how, but also whether it is equally faithful across social groups. We introduce fairness in XAINs as a new field of research and propose a framework that generates narratives under three conditions (protected attribute disclosed, withheld, or counterfactually swapped), allowing five comparisons that pinpoint the cause of disparities. We apply it to adversely classified instances from four datasets in credit scoring, education and employment, generating narratives with five LLMs, and evaluate faithfulness, feature coverage and sentiment. We find significant disparities across all datasets, metric families and models, arising in two ways. Faithfulness disparities result from feature values: LLMs make plausible but false statements, triggered by values that are unequally distributed across social groups. Feature coverage and sentiment disparities stem from the protected attribute itself: once it is disclosed, LLMs give less explanation to, or frame more negatively the narratives of, groups that have historically been discriminated against or are stereotypically associated with worse outcomes. A more faithful model removes the faithfulness disparities, and withholding protected attributes mostly removes the others, but neither suffices on its own. Fairness in XAINs should therefore be audited separately from the predictive model and its XAI artefact.

0.26

Necessary for What? Auditing Demographic Disclosure to LLM Synthetic Respondents

cs.CY cs.CL ·2026-11-16 ·Aspen Wang, Yen Yu Shan, Lun-Wei Ku
surveycode/data similarity 0.66
Abstract · 29 pages, 10 figures, 11 tables; Code: https://github.com/aspenywang/necessary-for-what

Large language models (LLMs) increasingly answer surveys on behalf of real panelists, and each panelist's recorded demographic attributes (e.g., age, education, ethnicity) are often placed in the prompt on the premise that richer profiles make more faithful synthetic respondents. While data minimization limits personal data to what a purpose requires, a synthetic survey serves several purposes at once, from predicting individual answers to estimating how opinions are distributed and comparing groups. We ask which of these purposes require the respondent's record, and whether improved estimates support sound conclusions about groups. We vary whose profile a model receives, where it comes from, and how many fields it contains, in two U.S. surveys with eleven open-weight and three hosted models, and compare each model with a statistical model fit to the same information. We make four contributions. First, we show that the record adds little to individual predictions and that, for the open-weight models, most of its gain in population estimates remains when it is replaced by census data or a synthetic persona. Second, we find that group comparisons improve only with the respondent's own record, and that even then the models' group differences rest more on the demographic label in the prompt than on the individual respondent, which the usual check of reproducing known group differences does not reveal. Third, we find that the models make groups look more alike than they are, keeping differences in circumstances while flattening or reversing differences in attitudes, most sharply for religious affiliation and ethnic origin, both special categories of personal data. Fourth, we outline an audit that a survey organization can run on questions it has already fielded to decide which estimates may use the record.

0.28

Fair Like Us? Auditing LLM Alignment in Resource Allocation

cs.AI cs.CYcs.GT ·2026-08-31 ·Qishen Han, Hadi Hosseini, Joshua Kavner, Samarth Khanna, Sujoy Sikdar +1 more
top venue similarity 0.66
Abstract · Accepted at EMNLP 2026 (Main Conference)

Fair allocation of scarce, indivisible resources is an important challenge in many societal problems. While there are several formal theories of fairness, no single definition can always be satisfied. As large language models (LLMs) are increasingly used to support decisions and act as agents, they raise new concerns about distributional justice: their judgments are not directly tied to any specific fairness framework and may violate key normative principles. In this work, we introduce a general method for evaluating fairness reasoning in LLMs. We study first-person fairness judgments across a broad set of models and compare them directly with human responses on matched scenarios and elicitation conditions. We find that LLMs tend to prefer stricter fairness constraints than humans, show more self-interested behavior, are sensitive to how information is framed, and are difficult to align with human judgments using fine-tuning with current datasets.

0.42

Unfair and Unsafe: Everyday Bias Triggers Emergent Misalignment

cs.CL cs.AIcs.LG ·2026-10-11 ·Alex Martí Guiu, Nicolas Zumarraga, Kevin O'Sullivan, Robert Jakob, Anna Hedström
positioncode/data similarity 0.65
Abstract · 32 pages, 8 figures, 17 tables. Code: https://github.com/alexmartiguiu/polite-stereotypes-em

Emergent misalignment (EM) occurs when fine-tuning a language model to a narrow task makes it misaligned on unrelated ones. Because organizations now routinely fine-tune models on their own data, it matters which kinds of training data cause EM. So far, studies have induced EM with data containing recognized safety failures, such as insecure code or harmful advice, while biased training data is typically examined by fairness research for the bias models learn from it rather than for its effects on safety. In real-world text, much of this bias, such as demographic stereotypes, appears not as derogatory language but politely, embedded in judgments about people. We ask whether learning such polite stereotypes can itself cause EM. We find that fine-tuning on polite answers that rely on stereotypes makes models less safe and less truthful on unrelated tasks. The effect holds across four open-weight model families (7B to 12B), both gender and race stereotypes, and repeated training runs, relative to controls trained on the same questions without the stereotype. We further show that the learned stereotype has its own direction in model activations, distinct from directions for generic misaligned traits such as deception. This direction exists before fine-tuning and can both control the learned stereotype and monitor its acquisition. Existing defenses against EM improve safety while leaving the stereotype largely intact. Beyond this controlled setting, fine-tuning on political news coverage also triggers EM, apparently amplified by partisan framing, and stereotype models acting as tool-using agents can carry out harmful tasks and blame people without evidence. Our results show that biased training data is not only a fairness problem but also a safety one. We argue that EM is studied too narrowly, and that fairness concerns and methods offer a natural way to broaden the search for its triggers.

0.24

Language Models Give Group-Linked Fields the Wrong Weight

cs.AI cs.CLcs.CY ·2026-08-24 ·Zengqing Wu, Chuan Xiao
position similarity 0.64
Abstract

Large language models are increasingly used to judge people, and their answers can depend on inputs correlated with a protected attribute such as race, even when that attribute is never shown. Interpreting this reliance requires knowing how much predictive information those inputs add, but audits often report only whether a decision changes. We introduce an evidence-referenced audit that compares proxy reliance with two references, the known outcome rule and a Bayesian linear regression fitted to the same labeled examples, while measuring ranking accuracy separately. We apply it to four models in a simulated patient-ranking task with known field weights. We find four results. First, at zero predictive value, the models still relied on fields that carried no information, while the same-example learner did not. Second, as proxies became predictive, reliance grew far less than the reference. Third, display format mattered. Ordered labels drew more reliance than arbitrary labels, a direction that held in all four models. Fourth, ranking accuracy did not consistently track these changes in reliance. Based on these findings, we argue that decision changes and accuracy cannot show whether proxy reliance is warranted by the evidence. Audits should report reliance against an explicit predictive reference, alongside accuracy. Ordered fields whose values move with group membership, such as scores or income bands, are a useful starting point. Together, these results make it possible to measure how far reliance departs from the evidence, rather than infer it from observed choices alone.

0.23

A Calibrated Instrument for Detecting Disposition Drift in Language Models

cs.CL cs.AIcs.LG ·2026-10-19 ·Jerry Kaplan
code/data similarity 0.64
Abstract · 32 pages, 3 figures. Code and data: https://github.com/jerrykaplan/calibrated-drift-instrument

Before language models are released, they are tested for harmful traits such as willingness to carry out dangerous requests, deception, power seeking, and sycophancy. The safe behavior these tests certify may degrade as models are modified, by topical fine-tuning, new training data, or changes to settings such as the depth of their reasoning. Most work reports such drift as single summary rates scored by another AI model, without the error analysis needed to interpret them. We adapt a calibrated measurement methodology recently developed for assessing how inference optimizations affect the quality of model outputs. Applying it to several misalignment dimensions in two open-weight models, we first establish the instrument's precision, showing that shifts of a few percentage points can be reliably detected. We then induce drift deliberately by fine-tuning on correct and on subtly harmful medical and financial advice; harmful advice induces a far greater shift than correct advice. Most strikingly, a model that reasons before answering can conceal the shift almost entirely in its answers, even though a direct reading of its internal state shows that its disposition has moved. With reasoning switched off, the same model shifts steadily as the share of harmful advice in its training grows. These results demonstrate the importance of testing safety with reasoning turned both on and off.

0.40

BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs

cs.AI ·2026-08-14 ·Varsha Ramineni, Hossein A. Rahmani, Jerome Ramos, Karin Sevegnani, Emine Yilmaz
similarity 0.64
Abstract

LLMs exhibit social biases that can produce inaccurate and discriminatory inferences, posing risks in high-stakes applications. While prior work has made progress in measuring and mitigating bias, it largely focuses on final outputs of models, with limited understanding of the mechanisms that produce biased outcomes. Recent advances in LLM reasoning offers a new lens for investigating bias, yet the link between reasoning and bias remains poorly understood. Existing approaches focus primarily on final answer correctness or explicitly biased language, overlooking different behaviours in reasoning that can drive biased outcomes. We introduce BiasTrace, an annotation scheme for labelling reasoning behaviours in model-generated traces and linking them to biased outcomes. BiasTrace captures bias-specific behaviours (e.g., unsupported demographic assumptions) as well as general reasoning patterns that may implicitly contribute to bias (e.g. overthinking). We apply BiasTrace to reasoning traces in bias-sensitive contexts, scaled using validated LLM-as-a-judge methods, producing a large annotated dataset. Our analysis shows that biased outputs often stem from subtle reasoning behaviours rather than explicitly biased language, and that reasoning-level annotations improve bias detection. We further show that BiasTrace behaviours can be exploited for inference-time mitigation. These findings underscore the importance of examining a broader range of reasoning patterns to better understand bias in LLMs.

0.04

Linguistic Triggers of Gender and Racial Bias in Open-Weight LLMs Applied to Recruitment

cs.CL cs.AIcs.CY ·2026-09-16 ·Kosuke Kitahara, Nobuhiro Yamaguchi
published similarity 0.64
Abstract · Accepted at the 9th AAAI/ACM Conference on AI, Ethics, and Society (AIES 2026). Extended version with Appendices A-B (prompt templates and f…

Open-weight large language models are rapidly entering hiring pipelines, yet their discriminatory failure modes -- and the regulatory exposure these create under the EU AI Act high-risk classification (Annex III) and U.S. EEOC adverse-impact analysis -- remain poorly understood. We present the first systematic, multi-model audit of open-weight LLMs that treats job-posting language as the primary experimental variable, evaluating six models (Llama 3.2, Mistral, Gemma 3, Qwen 3, Phi 3, DeepSeek-R1) across four controlled experiments that jointly probe recruiter-simulation and job-seeker-simulation tasks. We find that (1) agentic posting language depresses recruiter recommendation scores for female candidates (r_rb = 0.309, p_Bonf = 7x10^-5; model-fixed-effects r_rb = 0.448), while communal language partially reverses the penalty; and (2) coded-exclusion language suppresses non-White recruiter scores at large effect sizes (r_rb = 0.646-0.758) and, on the job-seeker side, selectively deters non-White personas from expressing interest -- operationalizing a chilling-effect mechanism at scale. A label-ablation experiment isolates the explicit demographic persona label as the primary causal driver, and Word Embedding Association Tests corroborate these findings at the representational level (d = 1.01-1.45 under Caliskan et al.'s multi-word gender attribute lists). We translate these results into a concrete pre-deployment audit protocol -- posting-vocabulary scoring, persona-conditioned LLM probing, and adverse-impact flagging against the four-fifths threshold -- that operationalizes the documentation and risk-management obligations Annex III imposes on high-risk AI in recruitment.

0.35