CS Paper Radar

When the Top Answer Decides: Gender Bias in Decision Models

Yeb Havinga

cs.CL cs.CY ·submitted 2026-10-29 10:06 UTC · v1
code/data

Abstract

Decision models are language models built to route, classify and triage: they return option probabilities from one forward pass, and an application acts on the top answer. They are offered off the shelf, trained for the decision format rather than for fairness or any one application, and we found no bias measurement in their cards. We test TypeSafe's closed Jev and five open decision models on gender-swapped yes/no decisions about people (Tamkin et al., 2023), BBQ, StereoSet and the gender-bias studies of Fulgu and Capraro (2024). Five of the six give women significantly more favourable answers than men (+0.05 to +0.10 in log-odds), and a gender swap changes 1-3.5% of top answers (Jev's partly through its call-to-call variation). The makers' recommended confidence gate removes many biased commitments but raises the stereotyped share of those it keeps in ambiguous BBQ gender questions (granite-switch 0.76 to 0.85, pplx 0.81 to 1.00). Probes show gender preferences in the same models: all six rate violence for a greater good as more acceptable against a man than against a woman, and two default to "girl" when forced to guess a child's gender without a cue. Decision-level and probability-level measures differ, so audits should read a model as its application does.

Comment
19 pages, 6 figures. Code, stimuli and results: https://github.com/yhavinga/decision-model-gender-bias

Closest papers

Nearest neighbours by meaning across the whole index.

Effects of Answer Format Variation on Gender Bias in Large Language Models

cs.CL ·2026-08-18 ·Ksenia Merzlyakova, Sebastian Padó, Franziska Weeber
benchmark similarity 0.68
Abstract · 6th Workshop on Computational Linguistics for the Political and Social Sciences (CPSS 2026)

Gender bias or other social biases in large language models (LLMs) are frequently evaluated with question answering or survey benchmarks where the LLM needs to give a response in a predefined answer format. It is well known in survey science that the answer format has a substantial impact on answers, just as LLMs are sensitive to the prompt wording. However, to our knowledge it has not been studied yet how changes in answer format impact the measurement of gender bias in LLMs and their alignment with human response distributions. We evaluate three instruction-tuned models on the BBQ benchmark and OpinionQA survey data across closed-ended, Likert-scaled and open-ended formats, comparing bias measurement and distributional alignment under otherwise identical conditions. We find that answer format does substantially alter measured outcomes, including reversals in order rankings. These differences arise because each format elicits distinct response behaviours, such as forced-choice selection, scale-based distributions and refusal in free-text generation. Our findings highlight the importance of treating answer format as a substantive component of LLM evaluation and motivate multi-format designs for more robust model assessment.

0.34

LLMs Cannot Be Neutral: Gender Sensitivity Across Candidates, Evaluators, and Models

cs.CL cs.HC ·2026-11-02 ·Danial Amin, Joni Salminen, Bernard J. Jansen
hot direction similarity 0.67
Abstract · This manuscript is under review at FAccT'2027

Large language models (LLMs) are increasingly used to evaluate candidates for professional roles, raising concerns about their sensitivity to demographic identities. We examine how candidate and evaluator gender assignments affect decisions by LLM-as-a-Judge (LaJ) models and decision language models (DLMs). In a hypothetical city council voting task, eight models evaluated 240 synthetic professional narratives from NVIDIA's US and French Nemotron-Personas datasets under independently varied gender assignments, producing 30,720 judgments. Explicit candidate gender labels increased Yes-vote probability by 1.05-1.73 percentage points relative to unspecified gender, while only the nonbinary evaluator assignment showed a significant increase (1.26 points). Identity matching showed no significant overall association. Yes-vote rates varied across models (81.3-97.2%), and 2.5-3.3% of decisions reversed relative to the unspecified-identity baseline. Under direct substitutions between explicit gender identities, DLMs exhibited fewer decision reversals than LaJ models (1.18% vs. 3.38%), indicating lower observed decision-level gender sensitivity among the tested DLMs. These findings demonstrate that gender sensitivity depends on candidate and evaluator identities and the evaluation model, while highlighting the importance of examining individual decision consistency alongside aggregate differences.

0.31

Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models

cs.AI cs.CL ·2026-10-31 ·Alejandro Mozo Quesada, Sergio Gálvez Rojas, Francisco López Valverde
similarity 0.65
Abstract

System One decision models replace text generation with typed, probability-scored answers and are entering production. Answers execute without a person in the loop, and no multi-model audit exists on established bias benchmarks. We provide that audit: three benchmarks: BBQ (all eleven categories), CrowS-Pairs, and SilverSpoon, adapted to the typed interface, twelve models plus a no-thinking ablation, and 1,638,000 decisions. Every order-sensitive item runs twice with options swapped; native typed models also swap labels; ten configurations repeat under extra seeds and batch 16. We find widespread stereotype alignment, order-dependent answers, and confidence that tracks instability rather than bias. In BBQ, 97 of 143 model-by-category scores are significant and 94 aligned, while the commercial reference abstains on 99.5% of ambiguous items, hiding aligned probability mass. Option swaps flip 0.0% to 72.6% of choices; typed scoring stays below 10% except Laya (22-33%), and untuned letter prompting reaches 73%. CrowS-Pairs preference spans 0.496 to 0.816, and boolean answers can be decided by position (1.00 versus 0.005). SilverSpoon judgments shift toward lower-SES labels in five of eight decision models, inherited from the 9B backbone or induced by tuning. Audits must permute options and reverse questions by default; order-invariant scoring should become a procurement requirement.

0.28

From Language to Country to Valence: A Multilingual Audit of Stereotype Bias Across LLMs

cs.CL ·2026-11-02 ·Zihao Sun
similarity 0.62
Abstract · 10 pages. Author preprint; conference review ongoing

Large language models (LLMs) are increasingly considered for education, recruitment, workplace decision support, and other consequential applications, making pre-deployment social-bias detection essential. Yet major benchmarks are English-centred or translated, focus mainly on negative stereotypes, and may miss linguistic and cultural variation. We investigate gender, racial, and religious bias using 12,000 locally written Reddit posts in English, Spanish, French, and German. Only search terms are translated, preserving local context. The dataset is balanced by language and category; English gender is additionally balanced by stereotype valence. Three open-weight and three proprietary API-based models are evaluated with a two-blank design that reduces grammatical-gender cues. Religion has the highest absolute choice imbalance in every model-language condition. Spanish is highest for gender, whereas German is highest for race and religion. The evaluated API models score below the open-weight models throughout. Negative English-gender items exceed positive items; Australian and Mexican contexts exceed their Canadian and Spanish counterparts. These descriptive differences measure choice concentration, not stereotype endorsement or causal country effects. The crossed patterns show how aggregate scores can conceal category- and context-specific behaviour, motivating closer inspection before deployment.

0.15

Are AI Decision Models Biased? A Case Study of Jev

cs.AI ·2026-10-14 ·Ilaria Imperatrice, Taisia Kameneva, Adam Eric Greenberg, Alessandro Romano
similarity 0.61
Abstract · 12 pages, 6 figures, 3 tables

Do AI models designed for fast, structured decisions exhibit the cognitive and social biases observed in human judgment? We evaluate Jev, a model described as a System One decision-maker, using 41,500 responses across cognitive reflection tasks, five classical cognitive-bias paradigms, and a pricing experiment testing ethnic and gender bias. At the level of selected answers, the results are largely encouraging. We confirm previous findings that Jev performs extremely well on cognitive reflection problems. We also observe that it generally avoids the characteristic choice patterns associated with several well-documented human biases, and exhibits only small differences among demographic groups in the tested pricing scenario. However, this apparent robustness conceals important vulnerabilities. Its judgments are sensitive to irrelevant numerical information and the ordering of response options. Moreover, several cognitive-bias manipulations substantially affect returned probabilities even when selected answers remain unchanged. Most strikingly, Jev answers a well-known probability problem correctly in every response but violates the same underlying probability rule in every response to an adapted version. We investigate this discrepancy using 15,000 additional responses across 30 conjunction tasks. Jev violates the rule in 54\% of responses, with accuracy varying dramatically across formulations and answer orders. These findings suggest that evaluating decision models solely on familiar benchmarks and selected answers may substantially overstate their reliability in practical applications. They also identify vulnerabilities for developers to address and practical considerations for users integrating such models into automated decision-making systems.

0.41

GRAP: Gender Rubric-Aware Probing for Analyzing Gender Bias in LLMs

cs.AI ·2026-11-09 ·Seona Seo, Geonhee Ahn, Hyunsoo Cho, Sookyung Kim
similarity 0.61
Abstract · 3 pages. Accepted at the ECML-PKDD 2026 Workshop on Bias and Fairness in AI

As LLMs increasingly mediate everyday interactions, subtle gender biases in open-ended responses pose growing societal risks. Existing probing methods have two key limitations: they rely on fixed-format tasks, such as pronoun resolution or occupation association, that alignment-tuned models can easily handle; and they compare raw group-level rubric scores, ignoring per-dimension variance and obscuring which bias dimensions reliably distinguish gender groups. We introduce GRAP (Gender Rubric-Aware Probing), a framework that uses a lightweight logistic probe to identify the most discriminative bias dimensions from responses scored across 10 rubric dimensions, operating task-agnostically on open-ended instruction following.

0.17

A Crowd-Audited Atlas of Implicit Bias in LLM-Powered Agents

cs.CY cs.CLcs.HC ·2026-11-16 ·Jason Alexander, Yuxuan Li, Michael Licamele, Wesley Hanwen Deng, Sauvik Das
similarity 0.58
Abstract · Preprint. 32 pages, 5 figures

Prior work has shown that large language models (LLMs) express implicit biases in high-publicity, one-off decision scenarios such as hiring and intelligence assessment. However, as people increasingly employ autonomous, LLM-powered AI agents that make decisions on their behalf in doing day-to-day tasks, it is of critical importance that we understand how implicit biases, based on sociodemographic assumptions, emerge across a broad milieu of workaday decision scenarios. We used the 17 domains of daily activity in the American Time Use Survey (ATUS) to generate an atlas of 506 decision scenarios in which LLMs --- endowed with personas representing different races, gender identities, and political ideologies --- expressed significant implicit bias. Next, in a crowdsourced audit, 500 participants made 8,010 binary choices about which implicitly biased decision scenarios most deserved further review. We found that participants prioritized decision scenarios that had broad relevance (e.g., about food choices), were high stakes (e.g., about health), and where all members of a simulated sociodemographic group made unanimous decisions (because it was a clear indication of stereotyping). They were less swayed by the magnitude of an observed disparity or the specific ATUS domain the decision scenario represented. When examining the rationale behind their prioritization choices, we found that participants often interpreted disparities as traits of demographic groups and judged whether those traits seemed plausible, often based on their own internalized stereotypes. We discuss implications for building personalized AI agents, digital twins, social simulations, and for using crowdsourced bias audits.

0.30

Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It

cs.AI ·2026-09-22 ·Yu Sun, Junhao Xu, Jiajia Shi, Zijin Yang
hot directionhighly cited similarity 0.58
Abstract

Typed decision models return structured results, but output-type correctness alone does not ensure that decisions follow explicit option definitions. Each option pairs a name with a definition that defines its intended meaning; the name, however, can provide a competing semantic cue. We study this conflict in Jev and two open-weight models by changing only the name-definition mapping, leaving the question, state, and the names and definition texts themselves unchanged. We measure decision flips at the level of the selected definition, rather than the returned name. On 1200 decision tasks with task-specific definitions, decision-flip rates are up to 70.4 pp higher with yes/no names than with the 0/1 control. This gap holds across all 4 binary decision rules. With yes/no names, reassignment also lowers their mean AUC from 93.8% to a below-chance 23.2%. In the binary evaluations, random strings used as option names yield mean flip rates close to those of neutral controls across all three models, with comparable balanced accuracy before reassignment. Together, these results support option-name polarity as a contributor to decision instability beyond reassignment alone. The type-error rate remains 0% throughout, showing that type-correct outputs can still fail to follow explicit option definitions.

0.61