CS Paper Radar

How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans

Hasan Mahmud, Khawaja Abaid Ullah, Mohammad Javad Khojasteh, Jamison Heard, Prabu David

cs.CL cs.AIcs.CY ·submitted 2026-08-10 15:19 UTC · v1

Abstract

Large language models (LLMs) are increasingly used to perform subjective evaluations traditionally made by humans, yet their validity as social judges remains unclear. This paper examines whether LLMs can assess social attraction from theory-grounded persona profiles constructed from ten psychological and relational constructs and organized into three tiers: socially attractive, socially mixed, and socially unattractive. We examine LLM ratings in two studies and compare them with human judgments in a third study. In Study 1, 34 LLMs rated 12 profiles across three repeated runs. Although some models tended to give higher or lower ratings overall, they showed strong stability across runs, consistent three-tier ordering, and high agreement in relative profile ordering. Study 2 examined sensitivity to gender presentation using six matched name-and-pronoun profile pairs and a separate pronoun-only test with a gender-neutral name, finding no significant effects in either analysis. In Study 3, 198 human participants evaluated the six matched profiles from Study 2. Their ratings reproduced the three-tier structure and followed a profile ordering consistent with that of the LLMs. However, LLMs rated attractive profiles more positively and unattractive profiles more negatively than humans, while neither group showed a significant overall effect of gender presentation.

Comment
9 pages, 2 figures, 2 tables. Includes technical supplement

Closest papers

Nearest neighbours by meaning across the whole index.

Implicit Personality Representations in Humans and LLMs

cs.AI ·2026-09-11 ·Yilin Geng, Omri Abend, Eduard Hovy, Lea Frermann
top venue similarity 0.71
Abstract · Findings of the Association for Computational Linguistics (EMNLP 2026)

A century of psychology has found that the trait words people use to describe one another vary, but the relational structure among those traits, which ones go together and which oppose, is strikingly consistent across raters and cultures. We test whether the LLM (Qwen 2.5-7B-Instruct) reproduces this structure in its internal trait representations. From millions of crowd-sourced personality ratings of fictional characters, we build a human implicit-personality matrix over hundreds of traits; from contrastive model activations, we build a matching matrix over the same traits. The two relational structures align strongly (Mantel r = 0.77), and the agreement holds trait by trait as well as in aggregate. Two dominant axes of the model's trait representations recover the social and intellectual dimensions long known to organize human personality impressions, social warmth and intellectual competence. On held-out dialogue, projecting model activations onto these directions yields personality profiles that agree with human ratings. This work establishes a framework that enables comprehensive, human-grounded comparison between internal model trait geometry and the shared structure of human personality impressions.

0.27

A Cracked Mirror: How People Perceive Gender Performance by Large Language Models

cs.HC physics.soc-ph ·2026-11-04 ·Faezeh Fadaei, Jenny Carla Moran, Taha Yasseri
similarity 0.66
Abstract · Preprint, under review; main article and supplementary material

How do large language models (LLMs) perform gender under zero-shot prompting, and how do people interpret those performances in conversation? Drawing on gender performativity, we examine how gendered meanings emerge through generated language, conversational context and human interpretation. In a preregistered randomised online experiment, 1,060 participants each had five short conversations with an LLM. We used minimal prompts to elicit performances from "very masculine" to "very feminine" and asked participants to rate their partner's perceived gender after each conversation. Mean ratings shifted in the direction of the prompts, but ratings at the extremes were closer to the midpoint than the prompt labels implied. When prompted to perform as "neither masculine nor feminine", the models were perceived as slightly masculine on average. Ratings varied across models and conversation topics. Women and men gave similar ratings at the extremes, but women rated the models as more masculine across the intermediate prompt levels. Participants' explanations illustrate how they drew on writing style, emotional expression and conversational interests, sometimes questioning the stereotypes behind their judgements. These findings suggest that conversational AI acts as a cracked mirror of gender norms. Models reproduce recognisable cues, and people interpret and sometimes challenge the meanings those cues carry.

0.07

Veracity labels flip the emotional tone of language models but leave racial disparities intact

cs.CY cs.CL ·2026-10-08 ·Nouar AlDahoul, Myles Joshua Toledo Tan, Yasir Zaki
code/data similarity 0.65
Abstract · 14 pages, 7 figures; Supplementary Information (42 pages, 6 figures, 27 tables) included. Data and code: https://github.com/Nouar-NYUAD/llm-…

Large language models (LLMs) increasingly deliver information through personas, but it is unclear whether correcting the facts also makes their framing consistent across identity labels. We audited four LLMs (GPT-5.4 mini, NVIDIA Nemotron 3 Nano, Llama 4 Scout, and Claude Haiku 4.5), which rated simulated affect on six dimensions (mood, anger, fear, confidence, perceived control, and mental state) in response to fact-checked false and true headlines presented without and with a veracity label. Experiment 1 evaluated 1,000 false and 1,000 true headlines without personas; Experiment 2 evaluated 300 of each under 12 personas crossing race, gender, and age. The label strongly changed the emotional tone: ordinal logistic regressions with headline-cluster bootstrap inference showed that refuting false headlines shifted ratings from mostly negative to mostly positive for GPT, Llama, and Claude (e.g., Claude's "confident" ratings rose from 22% to 99.6%), whereas Nemotron showed smaller, mixed shifts, and confirming true headlines produced only small shifts. The label did not, however, remove persona-associated disparities. The models assigned more negative simulated affect to Black than to White personas in 82 of 84 statistically significant comparisons across models and conditions, both without and with the label, and the disparity persisted on headlines unrelated to race; Asian and female personas were also predominantly assigned more negative affect, whereas age effects varied by model and condition. Factual correction and demographic consistency are therefore distinct properties of LLM behavior, and persona-conditioned systems should be audited for both before deployment in sensitive contexts.

0.39

LLMs Cannot Be Neutral: Gender Sensitivity Across Candidates, Evaluators, and Models

cs.CL cs.HC ·2026-11-02 ·Danial Amin, Joni Salminen, Bernard J. Jansen
hot direction similarity 0.64
Abstract · This manuscript is under review at FAccT'2027

Large language models (LLMs) are increasingly used to evaluate candidates for professional roles, raising concerns about their sensitivity to demographic identities. We examine how candidate and evaluator gender assignments affect decisions by LLM-as-a-Judge (LaJ) models and decision language models (DLMs). In a hypothetical city council voting task, eight models evaluated 240 synthetic professional narratives from NVIDIA's US and French Nemotron-Personas datasets under independently varied gender assignments, producing 30,720 judgments. Explicit candidate gender labels increased Yes-vote probability by 1.05-1.73 percentage points relative to unspecified gender, while only the nonbinary evaluator assignment showed a significant increase (1.26 points). Identity matching showed no significant overall association. Yes-vote rates varied across models (81.3-97.2%), and 2.5-3.3% of decisions reversed relative to the unspecified-identity baseline. Under direct substitutions between explicit gender identities, DLMs exhibited fewer decision reversals than LaJ models (1.18% vs. 3.38%), indicating lower observed decision-level gender sensitivity among the tested DLMs. These findings demonstrate that gender sensitivity depends on candidate and evaluator identities and the evaluation model, while highlighting the importance of examining individual decision consistency alongside aggregate differences.

0.31

Social Bias or Social Desirability? Comparing Factorial Survey Experiments for Humans and Large Language Models

cs.CY ·2026-10-14 ·Franziska Weeber, Thomas Hinz, Konstantin Mozer, Sam Sievers, Alan Akbik +1 more
survey similarity 0.64
Abstract

Factorial survey experiments are a survey method from the social sciences that uses descriptions of socially complex situation (``vignettes'') to elicit judgments such as social bias. They provide two major benefits: (a), they reduce the effect of social desirability inherent in self-reported social biases, and (b), they make it possible to separate the effects of multiple sociodemographic attributes. These aspects are also desiderata for the evaluation of social bias in LLMs, but are often ignored in social bias evaluations. In our study, we evaluate social bias in humans and eleven LLMs on the basis of four different English and German factorial survey experiments. Our results show that the LLMs mirror the human biases only partially and sometimes even invert them. We provide evidence that this is due to a fundamental difference in how humans and LLMs react to socially complex situations. While such vignettes reduce social desirability bias in humans, this is not the case for LLMs, where the alignment-induced tendency towards socially desirable responses carries over directly. We therefore caution against applying bias measurement techniques from the social sciences that were developed for human to measure attitudes and bias in LLMs.

0.16

Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference

cs.CL cs.AI ·2026-08-27 ·Mengfan Li, Zesheng Wei, Xuanhua Shi, Yang Deng
top venuehot direction similarity 0.62
Abstract · Accepted by EMNLP 2026 main conference

As large language models are increasingly deployed to simulate diverse human characters, ensuring persona fidelity, defined as the extent to which an agent's behavior consistently reflects the psychological and stylistic characteristics of a target persona, has become a critical requirement. However, existing evaluation paradigms primarily rely on either holistic LLM-based judges, which are prone to "holistic appraisal hallucination'', or static psychometric inventories, which fail to capture the context-dependent fidelity required in dynamic dialogue. To address these limitations, we propose PRISM (Persona Reasoning with Inverse SFL-based Modeling), a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task. Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style. It estimates dimension-specific evidence over a persona-conditioned label space and aggregates these signals into an interpretable and auditable evaluation process. Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.

0.71

Beauty is in the AI of the beholder: MLLMs systematically overrate facial attractiveness

cs.CV cs.HC ·2026-09-02 ·Santiago Grandas, Juan Sebastian Cely-Acosta, Mohit Mendiratta, Shafee Hassan, Macken Murphy
similarity 0.61
Abstract

Beauty assessments from Multimodal Large Language Models (MLLMs) are increasingly popular amongst users, companies, and aestheticians. This raises the question of whether these AI models can accurately reflect human judgments of attractiveness. In a pre- registered exploratory study, we compared the attractiveness ratings of 2,513 human participants to four widely used commercial AI models: Claude, Gemini, GPT, and Grok. Results showed that MLLMs systematically rate faces more favourably and within a narrower range than humans and, at the time of study, do not reproduce human ratings in absolute terms. However, MLLMs exhibit strong correlations with human attractiveness judgments, accurately tracking the rank-ordering of faces. MLLMs may judge faces by different cues than humans; only face age was a predictor of facial attractiveness in both humans and MLLMs, with inconsistent patterns across models for ethnicity and gender. AI models strongly agree with one another, except for Grok, which also showed the lowest agreement with humans. Our findings suggest that while they may be able to approximate rank-orderings of human attractiveness, current off-the-shelf commercial MLLMs systematically overrate the beauty of human faces.

0.15

Emotional Labor Strategy Preferences in LLM Personas

cs.CL ·2026-08-31 ·Mohammad Saim, Tianyu Jiang
top venuedataset similarity 0.61
Abstract · 18 pages, 4 figures, 12 tables. Findings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026)

Emotional labor is the effortful management of emotional displays to meet social or professional expectations. Personality traits have been correlated with emotional labor strategies, yet research on this link relies almost exclusively on self-report scales administered only in occupational settings. We investigate whether large language models injected with psychometrically grounded personas reproduce these personality-driven selection patterns across everyday social scenarios. We construct the first emotional labor strategy dataset of 500 socially situated events, each offering three behavioral choices corresponding to surface acting, deep acting, and genuine expression. We source 50 fictional characters from a large-scale personality repository and profile each through two parallel tracks: observer-rated bipolar adjective composites and in-character self-report items. Five LLMs evaluate all scenarios under both persona conditions. We find that models align more towards deep acting, and that Conscientiousness and Emotional Stability consistently predict this preference. Entropy analysis confirms that persona reliably influences the output and varies across models and emotions.

0.42