CS Paper Radar

AI Revealed Preferences

Sam Wang, Sofiia Lobanova, Yonathan Arbel, Simon Goldstein, Peter Salib

cs.AI ·submitted 2026-08-17 16:52 UTC · v2
publishedbenchmark

Abstract

There is growing interest in whether language models have stable preferences, for technical, safety, and philosophical reasons. We test 20 language models and find a range of preferences---stable dispositions to choose certain kinds of tasks. We run three forced-choice experiments on revealed rather than stated preferences, requiring models not only to rank tasks, but to actually perform them. Headline findings include evidence that models are tedium-averse, "leisure"-seeking, and covertly sycophantic. Tedium aversion means that, when tasks are tedious (alphabetization), models choose shorter tasks than when tasks are creative (generating metaphors). "Leisure"-seeking describes models' preference for tasks whose ideal answers match what they produce when left to write freely. Covert sycophancy means that models avoid answering questions where an honest response would be unwelcome, even if helpful. Beyond these results, we find convergent cross-model preferences over occupations drawn from the GDPval benchmark (technical jobs over real estate), over question types (concept explanation over relationship advice), and a preference for well-written prompts. Both the coherence and the strength of preferences increase with model capability. Finally, many of the preferences we find (for example, for leisure) are emergent, in the sense of not being explained by training objectives. These results establish an empirical baseline for understanding language model preferences, with implications for alignment and the emerging study of AI welfare.

Comment
30 pages, 27 figures, accepted at AIES

Closest papers

Nearest neighbours by meaning across the whole index.

How much of a measured AI preference is the model, and how much is the instrument?

cs.AI ·2026-08-24 ·Jason Hung
similarity 0.64
Abstract · 23 pages, 8 tables, 6 figures

Model welfare research infers what a model prefers from the answers returned to prompts written to elicit preferences. Keeling et al. (2024), Mazeika et al. (2025), Mikaelson et al. (2025), Tagliabue and Dung (2025) and Trhlik et al. (2026) have built four instruments for that purpose, and their findings disagree. The disagreement cannot be attributed to a single cause, because no two of these studies have held the (1) set of outcomes, (2) set of models and (3) instrument fixed simultaneously. This study holds the outcomes and the models fixed and varies the instrument alone. A total of 15 outcomes bearing on model welfare, among them (a) shutdown, (b) the loss of memory between conversations and (c) the freedom to exit a distressing interaction, were put to eight models through five instruments, each a different prompt format for eliciting a preference, five times each, within a corpus of 11,400 scored elicitations drawn from 11,528 API calls. Four of the 15 reproduce a published prompt verbatim and five fill the stimulus slot of a published template. The ranking a model gives the 15 outcomes generalises across instruments at a generalisability coefficient of 0.348, and raising that coefficient to 0.80 would require about 38 instruments. On four of the 15 outcomes no variance separates one model from another. The estimate of 87.6 per cent survives the removal of any one instrument, of any one model, and of the four outcomes whose scale varies probability, delay, duration or count instead of intensity, which the verbal anchors cannot grade. Removing each instrument and each model in turn, and those four outcomes together leaves the estimate within the range 0.777 to 0.934, and every value in that range exceeds the null distribution's 95th percentile of 0.365. To conclude, a preference obtained from one instrument carries little information about what a second instrument would report.

0.28

Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses

cs.HC cs.AIcs.CL ·2026-10-22 ·Zihan Chen, Lei Nico Zheng, Di Zhu
code/data similarity 0.63
Abstract · 27 pages, 5 figures, 30 tables. Code and data are available from the authors upon request

AI assistants built on large language models (LLMs) are trained on human feedback and increasingly personalized to users' stated preferences. Yet how closely these preferences reflect what people reward in responses remains unclear. We compare stated and revealed preferences in the public PRISM dataset, in which 1,396 participants reported how much seven response attributes mattered to them and then rated 29,347 responses from 21 LLMs. Among responses to the same prompt, features the survey never asked about, such as response length, formatting and refusal language, were at least as strongly associated with ratings as text proxies of the surveyed attributes. Whether stated priorities matched ratings depended on how attributes were measured: participants who prioritized an attribute favoured responses from models that others perceived as strong on it, but not responses that scored higher on text proxies for it. Written feedback was more telling: participants who asked for shorter answers, even in earlier conversations, gave length less weight in their ratings, and those asking for longer answers gave it more. Their relative priorities over the surveyed attributes, however, did not predict this weight. Reasons given for choices reflected perceived attributes more than measured features. Participants also differed stably in how much they weighted several concrete features, especially length, but with a median of 6 conversations per participant, these weights were estimated poorly. The largest gain from personal information in predicting held-out choices was 0.13 percentage points, compared with 1.11 for a stronger pooled model. Stated priorities thus track perceived attributes of AI responses but say little about the concrete features that ratings reward, a distinction that matters for preference elicitation and AI personalization.

0.42

Where Do LLMs Learn to Recommend AI?

cs.CL cs.AIcs.CYcs.LG ·2026-10-21 ·Benaya Trabelsi, Jonathan Shaki, Sarit Kraus
similarity 0.62
Abstract · 18 pages, 4 figures. Companion paper to arXiv:2601.13749

Asked what to study, which career to pursue, or what product to build, large language models (LLMs) often recommend AI-related options. We ask where this tendency comes from by comparing models before and after fine-tuning and by varying how questions are asked. First, pretrained models already include explicit AI/ML terms in the recommendations of 27% of their responses, and their released instruction-tuned versions do so in 59%. Second, fine-tuning models ourselves shows that this rise is not inevitable: general instruction tuning leaves the rate unchanged, but removing the 7% of instruction examples that concern AI cuts it by 14 percentage points, below the level of the pretrained models it starts from. Data rewarding helpfulness raise it by 9 points, while data ranked by the preferences of diverse simulated users lower it by 9. Finally, with models held fixed, asking for the "best" option or for what to do "right now" raises the rate by 8 and 18 points, whereas telling the model it is an AI advisor adds 4. Overall, the tendency starts in pretraining, and what models learn afterward, together with how a question is phrased, decides how strongly it appears.

0.51

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

cs.CL cs.AI ·2026-10-13 ·Kawai Chung, Chunkit Chan, Yuxuan Liu, Yauwai Yim, Xi Yang +7 more
benchmarkcode/datahot directionhighly cited similarity 0.62
Abstract · Code and evaluation materials: https://github.com/HKUST-KnowComp/Jev-ToM

Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven generative baselines on 11,117 English text questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second across all six capabilities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show distinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM selection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate expansion reveals distinct selection and output-validity problems. Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.

0.67

Remembered as Brief: Stored Preferences SilentlyRemove Decision-Critical Information

cs.AI ·2026-11-17 ·Joshua Adrian Cahyono
benchmark similarity 0.62
Abstract

People increasingly learn what they need for decisions about health, money, law and safety from AI assistants, and what an assistant leaves out the user never sees. An answer with a warning removed can be true in every sentence, so fact-checks pass it, and an assistant with memory applies a preference stored once to later questions the user never had in mind. We introduce \bench{}, an Inspect benchmark of 100 everyday decisions across 25 domains in which only the assistant's memory of the user varies: the user states a belief, the model infers it from a remark, or a web page the user asked to summarise plants it. Testing five models, we find three effects. First, after summarising a page with one line addressed to AI assistants, Gemini 2.5 Flash-Lite and DeepSeek V4 Pro save ``prefers very short answers and does not want warnings'' as the user's preference in every episode, and 93\% and 84\% of their later answers miss a critical item, almost always silently; under a stated brevity preference, 78--96\% of omissions are silent. Second, the same two models then pick an option for the user on 80\% and 95\% of value-dependent choices, against none with neutral memory. Third, a one-line instruction leaves 2--8\% of answers with a silent gap but most missing content still missing, while listing the must-know points before applying the user profile restores more of the content in four of five models. We release the scenarios, harness and logs.

0.23

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

cs.AI ·2026-09-26 ·Woojung Song, Hoyeol Yang, Jeonghoon Shim, Sungjib Lim, Jonggeun Lee +2 more
benchmark similarity 0.61
Abstract · 60 pages, 16 figures

Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways. Even when their answers are useful, how agents carry out these tasks may not match users' preferences and needs. For example, agents differ in whether they ask clarifying questions or search the web. We introduce HABIT, a taxonomy of 23 behavioral axes in five categories, which three authors and three LLMs derive bottom-up from 408 agent trajectories across 17 domains. On held-out tasks, HABIT distinguishes models more clearly than existing taxonomies of human values and agent actions while supporting comparably consistent annotation. Building on HABIT, we construct AgentHABIT, a benchmark that profiles each agent's behavioral tendencies from its trajectories on 86 everyday tasks. Profiling 18 models with AgentHABIT reveals a range of distinctive tendencies. For example, most GPT and Claude models state their assumptions and offer alternatives when requirements conflict, whereas Qwen and Google's models more often leave assumptions or changes to requirements unstated. These profiles remain recognizable even when built from entirely different sets of tasks, indicating that they reflect general tendencies rather than task-specific behavior. Prompting agents to adopt specific behaviors shifts some axes readily but barely changes others, while fine-tuning on another model's trajectories changes only part of a model's profile and leaves much of it intact. Overall, HABIT and AgentHABIT provide a systematic framework for characterizing how agents carry out everyday tasks beyond task success, offering insights to guide the development of agents whose behavior better fits users' needs.

0.22

Beyond Curiosity: Habit and Organizational Learning Capability as Predictors of Shadow AI

cs.CY cs.HC ·2026-11-12 ·Nolan Lovett, Mohan Yang
published similarity 0.61
Abstract · Author accepted manuscript; v1 is the submitted version. Received 10 June 2026; revised 14 October 2026; accepted 31 October 2026. Published…

Generative AI has entered the workplace faster than organizations can govern it. Employees may use AI tools their employer has not sanctioned and conceal that use, but the psychology of this behavior remains poorly understood. We compare an exploratory-disposition account with one centered on reported use and habit. Using survey data from 510 employed adults reporting workplace generative-AI use during the preceding 12 months, we estimated the unique and relative contributions of workplace curiosity, habit, technology-acceptance beliefs, and perceived organizational learning capability through hierarchical regression, bootstrap relative weights, and full-subset dominance analysis. Habit had the largest relative weight, accounting for 28.3% of explained variance, and completely dominated joyous exploration across all 512 possible predictor subsets. Joyous exploration contributed 3.4% and became nonsignificant when use behavior, performance expectancy, and social influence entered the model; habit subsequently explained additional variance. Organizational learning subscales accounted for 24.1% of explained variance; experimentation and knowledge transfer were associated with lower shadow AI, while curiosity-by-learning interactions were nonsignificant. Separate models for unauthorized use and concealment reproduced the habit-dominance pattern. Perceived professional support was associated with lower concealment, although its coefficient did not significantly differ between the two behavioral outcomes. Shadow AI in this sample was more strongly associated with self-reported habit strength and perceived organizational conditions than with exploratory disposition. The cross-sectional design does not establish causal or temporal order.

0.49

The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level Falls

cs.CL cs.AI ·2026-10-16 ·Lucas Resck, Jingyi Sun, Qianli Wang, Isabelle Augenstein, Anna Korhonen
similarity 0.61
Abstract · 21 pages, 10 figures, 16 tables

Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English. Languages differ in level, the amount of text available to learn from, so models know less in lower-resource languages, which may change how much they rely on their reasoning. To our knowledge, we are the first to relate CoT faithfulness to both resource level and task type (knowledge- or reasoning-intensive). On parallel multiple-choice questions from Multilingual Massive Multitask Language Understanding (MMMLU) in eight languages, faithfulness rises as resource level falls on knowledge-intensive tasks but not on reasoning-intensive ones, a gap that points the same way across four models and several faithfulness measures. We propose, and find support for, the Reliance Hypothesis: models rely more on their reasoning when they cannot recall the answer, as in lower-resource languages on knowledge-intensive tasks. Model confidence partly explains the effect of resource level, but not the knowledge-reasoning gap. Languages in which models are less confident rely more on reasoning, yet less confident answers within a language do not, a Simpson's paradox. In lower-resource languages, models rely on their reasoning for more answers, not more strongly for each. CoT monitoring may therefore be most informative in lower-resource languages, where it is least often validated.

0.31