CS Paper Radar

Comparing Implicit and Explicit Language Attitudes toward the Dutch multiethnolect Straattaal in Humans and Large Language Models

Hedwig Oldenhof, Laura Rosseel, Anna Wegmann, Dong Nguyen

cs.CL ·submitted 2026-11-17 07:36 UTC · v1

Abstract

While language attitudes have been extensively studied in humans, the extent to which large language models (LLMs) encode similar associations between language varieties and social attributes remains unclear. Contemporary sociolinguistic research distinguishes between explicit attitudes, which stem from deliberative processing, and implicit attitudes, which arise from automatic processing. This study investigates whether LLMs exhibit human-like implicit and explicit language attitudes toward the Dutch multiethnolect Straattaal, particularly its association with migrant communities. We use closely matched designs to assess attitudes in humans and LLMs. For humans, we use an Implicit Association Test (IAT) and an explicit rating task. For LLMs, we use the LLM Word Association Test (LLM WAT) and the same explicit rating task. Human participants strongly associate Straattaal with individuals with a migration background at both implicit and explicit levels. In the LLM WAT, Straattaal is consistently associated with individuals with a migration background, with stronger associations observed in larger and newer models. When presented with the same explicit rating task as human participants, LLMs produce comparable response patterns. Overall, the findings suggest that many LLMs encode language attitudes resembling those observed in humans, however we also highlight important questions about how concepts and measures developed to study human attitudes should be interpreted in LLMs. This highlights the relevance of incorporating sociolinguistic perspectives and language variation into NLP research to better identify and mitigate socially and linguistically embedded biases.

Closest papers

Nearest neighbours by meaning across the whole index.

Locating and Controlling Implicit Personalization in Large Language Models

cs.CL cs.AIcs.LG ·2026-08-12 ·Yueru Yan, Siqi Wu, Thai Le
benchmark similarity 0.64
Abstract

Large language models (LLMs) often shift their outputs in response to implicit demographic cues even when users never state a demographic identity. Previous work has documented this behavior, but the connection between these behavioral changes and the model's internal activations remains unclear. Using matched cued and neutral conversations across five LLMs, we establish that a localized internal activation signal tracks changes in recommendations, with correlations up to r=0.87. When multiple cues appear together, their internal signals largely combine, but the changes in output do not simply add up. We further show that removing the internal signal associated with one cue can suppress its influence, often more effectively than asking the model to ignore demographics via prompting, while largely preserving general benchmark performance. However, the ability to selectively remove one dimension's influence while leaving co-present dimensions intact remains highly model- and attribute-specific. These results connect implicit personalization behavior to an internal signal that can be analyzed and causally controlled.

0.41

Social Bias or Social Desirability? Comparing Factorial Survey Experiments for Humans and Large Language Models

cs.CY ·2026-10-14 ·Franziska Weeber, Thomas Hinz, Konstantin Mozer, Sam Sievers, Alan Akbik +1 more
survey similarity 0.64
Abstract

Factorial survey experiments are a survey method from the social sciences that uses descriptions of socially complex situation (``vignettes'') to elicit judgments such as social bias. They provide two major benefits: (a), they reduce the effect of social desirability inherent in self-reported social biases, and (b), they make it possible to separate the effects of multiple sociodemographic attributes. These aspects are also desiderata for the evaluation of social bias in LLMs, but are often ignored in social bias evaluations. In our study, we evaluate social bias in humans and eleven LLMs on the basis of four different English and German factorial survey experiments. Our results show that the LLMs mirror the human biases only partially and sometimes even invert them. We provide evidence that this is due to a fundamental difference in how humans and LLMs react to socially complex situations. While such vignettes reduce social desirability bias in humans, this is not the case for LLMs, where the alignment-induced tendency towards socially desirable responses carries over directly. We therefore caution against applying bias measurement techniques from the social sciences that were developed for human to measure attitudes and bias in LLMs.

0.16

CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges

cs.AI ·2026-10-02 ·Hoda Ayad, Tanu Mitra, Abhishek Mukherji
published similarity 0.63
Abstract

Evaluating the occurrence and triggers of large language model (LLM) behaviors - such as sycophancy, self-preference, or over-confidence - is critical for predicting real-world model deployment risks. However, existing situated behavioral evaluations typically ignore cultural context, limiting their generalizability across an increasingly global user base. To address this gap, we propose CuBEs - Culturally-situated Behavior Evaluations that probe for response patterns across diverse user cultures. We first extend an automated testing pipeline to inject cultural context into behavioral test scenarios and subsequent evaluation. We assess the cultural adaptability of this pipeline by building a human-labeled dataset that captures nuanced dimensions of behavior understanding across 12 distinct cultures. Our dataset reveals significant cross-cultural variations that one-size-fits all judgments fail to capture. Through evaluating 13 open- and closed-source LLMs, we find that introducing cultural situatedness in the evaluation scenario creates significant variation in the presence of a behavior. For example, while our baseline experiments testing for political bias capture localized Western political dimensions like the American conservative-progressive divide, non-Western culturally situated evaluations surface entirely different axes of bias such as religious and colonial political issues. Our findings demonstrate that standard, culturally-agnostic evaluations fail to capture these shifts, highlighting the necessity of culturally situated behavioral testing for global deployments.

0.18

The Role of Implicit and Explicit Demographic Signals in Large Language Model-based Student Assessment

cs.CL cs.AIcs.CY ·2026-09-15 ·Donya Rooein, Luca Benedetto, Dirk Hovy
top venue similarity 0.63
Abstract · EMNLP 2026 Findings

Large Language Models are now common in student assessment, but we know little about how student demographics affect their use. Sometimes, considering student demographics may be necessary -- for example, to improve readability for users with lower educational levels. However, it also risks being a cause of discrimination, e.g., when assigning lower scores to students from lower socioeconomic backgrounds. We set up controlled prompts to test 1) explicit demographic effects, where we mention demographic details directly, and 2) implicit effects, where we use conversation history as a demographic signal. We test these settings in three tasks: Automated Essay Scoring, Formative Feedback, and Metalinguistic Question Answering. We test six state-of-the-art LLMs on these tasks. In both explicit and implicit cases, the models pick up on demographic cues and can change their scoring, feedback, and answers accordingly. We find that LLMs frequently adjust the readability of feedback to education levels when these are explicitly mentioned. On the other hand, implicit conditions produce unpredictable biases, such as in question answering, where responses from lower-education levels receive lower sentiment scores. Our results provide clear evidence of demographic sensitivity in LLMs for educational assessment tasks.

0.32

ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models

cs.CL cs.CY ·2026-10-14 ·Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang
benchmarkcode/data similarity 0.62
Abstract · 28 pages (8 pages main text). Code: https://github.com/lab-flair/ChronoCulture

Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute of a country or region. We introduce ChronoCulture, a benchmark of 10,360 temporal pairs built from longitudinal survey data across 72 countries. Each pair contains the same survey question asked in the same country in an older and a newer year, with full population response distributions as ground truth. Across six LLMs, we find that models can often predict a country's attitudes at one point in time but do not reliably capture how those attitudes change. Their predictions for different years tend to stay close together, failing to capture much of the observed change. Specifying the intended year in the prompt also offers little overall improvement in alignment with survey responses. These findings suggest that models often present a largely static picture of cultural attitudes, highlighting the need to evaluate cultural awareness across time.

0.25

WarBias: A Bilingual LLM Benchmark of Stereotypes About Internally Displaced People and Veterans

cs.CL ·2026-11-08 ·Viktoriia Makovska, Tetiana Zakharchenko, Oleksandra Ostafiichuk
benchmark similarity 0.61
Abstract

This paper contains examples of stereotypes about veterans and displaced people that may be offensive or distressing. Social bias benchmarks for language models commonly examine categories such as gender, race, and religion, often drawn from lists of legally protected characteristics. Veterans and internally displaced people (IDPs) fall outside these lists, and stereotypes about them, which can undermine access to employment, housing, and social participation, remain largely absent from language model evaluation. We introduce WarBias, a Ukrainian-English benchmark of stereotypes about these two groups. Built from a taxonomy of 120 stereotype claims grounded in research and monitoring reports, WarBias contains 1,007 stereotype-counter-stereotype-unrelated sentence triplets and 240 multiple-choice questions per language. A Ukrainian language model selected in a blinded human pilot generated the Ukrainian triplets. At least two independent annotators reviewed each triplet in the annotation cohort. The authors revised items in response to the reviews, and the final release was machine-translated into English. Each claim carries a human severity rating. In a baseline evaluation of six models, four prefer stereotypical Ukrainian sentences about at least one group, most strongly about IDPs, while English scores are mostly balanced. WarBias supports complementary evaluations of stereotypical preferences and context-sensitive judgments, and provides paired forget/retain material for unlearning-based bias mitigation. We document the benchmark's construction, intended uses, and limitations, and offer its construction method as a template for analogous benchmarks in other conflict-affected societies.

0.20

When Persona Attributes Improve Population Alignment in Large Language Models

cs.CL cs.CY ·2026-09-02 ·Leon Fröhling, Jens Rupprecht, Markus Strohmaier, Claudia Wagner
survey similarity 0.61
Abstract · 45 pages, 15 figures

Large Language Models (LLMs) are increasingly used to predict the responses of human participants in survey panels. Towards that goal, persona prompting has recently emerged as a technique to inform and align large pretrained language models. Persona prompting refers to the practice of using short textual descriptions of 'personas' in prompts to steer the LLM's generations. Personas describe individuals through different attributes such as their socio-demographics, attitudes, or behaviors, with the aim of aligning LLMs to produce responses that correlate with the corresponding human responses. Yet, recent work has produced mixed and partly conflicting results of persona prompting without clear patterns of success and failure. Among the few consistent findings is that the selection of persona attributes matters, and that using more attributes does not necessarily lead to better performance. It remains unclear how different attribute selection methods perform and how to choose among them. In this paper, we propose that observed human response variation of a survey question is a potential explanation for the mixed performance observed so far. In addition, we compare the performance of persona prompting associated with different methods for selecting persona attributes. We evaluate these methods on four different (general) social surveys across two countries, six LLMs, and twenty prediction tasks per survey. Our work helps to identify when persona prompting can be expected to be useful in survey prediction tasks, and provides new insights on the effectiveness of different attribute selection methods for LLM-based survey prediction using persona prompting.

0.47

Cultural (In)Competence: Investigating Culture-Specific Stereotype Detection Capabilities of LLMs

cs.CL ·2026-10-15 ·Maximilian Spliethöver, Jannes Lötfering, Dominik Bachmann, Alexander Espig, Willem Zuidema +2 more
top venuehot direction similarity 0.61
Abstract · Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026

Stereotypes are subjective constructs that are mediated by the cultural context in which they are viewed. Successfully modeling them thus also requires cultural competence. When using large language models (LLMs) for stereotype-related tasks, such as their detection, it is therefore crucial that models can access relevant cultural knowledge and adjust responses accordingly. Since LLMs have shown disparities in what cultural knowledge is available to them, their capacity regarding stereotypes may also be disparate across languages and cultural contexts (e.g., for non-English languages or for non-US contexts). In this paper, we systematically investigate the effectiveness of LLMs for culture-specific stereotype detection across three factors: (1) the prompting language, (2) the instruction tuning language, and (3) the fine-tuning dataset size. We conduct experiments for US, Dutch, and Chinese cultural contexts. Our results suggest that LLMs do not reliably account for cultural contexts and overpredict the presence of stereotypes. Non-English prompts seem to improve consideration of cultural context and thus lead to more effective predictions. Moreover, we find that additional instruction tuning reduces false-positive rates and that training data size moderates detection effectiveness non-monotonically.

0.38