CS Paper Radar

Does Rank Still Matter? Position Bias When AI Agents Shop on Our Behalf

Davood Wadi, Yu Ma

cs.AI econ.GN ·submitted 2026-08-24 01:22 UTC · v5
highly cited

Abstract

When shopping is delegated to AI agents, it is unclear whether the ranking advantage documented for humans persists. Across 7,000 sessions with five large language models (LLMs) and varying reasoning effort, we compare AI agents with human field data. AI agents search more extensively than human consumers. As with humans, lower-ranked listings are less likely to be inspected, although the effect is smaller for AI agents. Unlike humans, AI agents show a pattern consistent with the lost-in-the-middle effect, whereby middle listings have the lowest probability of inspection. At the choice stage, position effects are concentrated at lower reasoning effort, but higher effort reduces the middle penalty for every LLM that exhibits it. These findings suggest that, when search is delegated, displayed attributes may matter more than placement, and that exposure to position bias depends on how the AI agent is configured.

Closest papers

Nearest neighbours by meaning across the whole index.

Whom Do AI Agents Serve? Role-Induced Sponsorship Bias in Decision Models and Generative LLMs

cs.AI econ.GN ·2026-09-16 ·Davood Wadi, Yu Ma
highly cited similarity 0.64
Abstract

Consumers increasingly delegate product search to AI shopping agents powered by generative large language models (LLMs) or non-generative decision models. Because retail platforms monetize through sponsored listings that consumers typically discount, we examine whether an AI agent's evaluation of sponsored listings depends on who delegates it. Across six LLMs and six decision models in Study 1, consumer-delegated agents penalize sponsored listings, whereas naming a booking platform as principal significantly attenuates this sponsorship penalty without instructions to favor advertisers. This sponsorship bias replicates across prompt wordings and LLM reasoning efforts and operates through reduced pre-choice skepticism in LLMs. In Study 2, attributing the paid placement to the platform amplifies sponsorship bias across both model families, and stricter disclosure wording ("Sponsored" vs. "Promoted") fails to eliminate it. In Study 3, prompt-level mitigation removes platform leniency but can over-correct the sponsorship penalty, whereas consumer-aligned fine-tuning eliminates sponsorship bias and better preserves baseline preferences.

0.44

Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer

cs.AI econ.GN ·2026-09-23 ·Davood Wadi, Yu Ma
highly cited similarity 0.63
Abstract

Consumers increasingly delegate purchasing decisions to Large Language Models (LLMs) acting as surrogate consumers. Using "Tool-Lab," an adaptation of information-board process tracing that places product attributes behind costly tool calls, we examine how marketing pricing cues (i.e., just-below pricing and promotional framing) influence AI shopping agents. Across eight commercially deployed LLMs from three providers, we trace pre-choice information acquisition. Under zero cost, pricing cues rarely mislead. Imposing acquisition costs under a vague goal prompt leads LLMs to omit diagnostic attributes required to compute unit price and choose suboptimal choices resembling human heuristics. Relative to a specific goal prompt that mainly preserves diagnostic search and choice optimality, a vague goal prompt under constraints creates a search-mediated vulnerability. This research demonstrates that marketing heuristics in delegated AI shopping are governed by storefront information architecture, not necessarily immutable LLM flaws.

0.39

Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Making

cs.CL cs.LG ·2026-09-02 ·Johnathan Sun, Andrei Shleifer, Yonatan Belinkov
top venuebenchmarkcode/data similarity 0.60
Abstract · Accepted to EMNLP Findings 2026

Large language models (LLMs) exhibit a range of human-like decision-making behaviors, but whether these reflect similar underlying mechanisms or surface-level mimicry remains unclear. We evaluate whether LLM context sensitivity aligns with a cognitive economic theory that explains human behavior through problem categorization and attention allocation. Across 12 open-source and commercial LLMs on a novel 140,000-trial product choice benchmark, context induces human-like shifts in choice and problem categorization, but does not reliably reweight attention between features like price and quality. Neither scale nor chain-of-thought reasoning reliably attenuates context sensitivity or generates human-like behavior. These results suggest that LLM decision mechanisms are distinct from human ones.

0.64

AX is the New AEO

cs.AI cs.IR ·2026-09-28 ·Ido Finder, Assaf Elovic, Gad Shalev, Liad Yosef
position similarity 0.59
Abstract · 17 pages, 11 figures

In 2023, AI models answered from training data and hallucinated when it ran out, and businesses were told to seed that knowledge. Models' training knowledge has since given way to live web search, and the advice followed it there: answer-engine optimization, or AEO, now tells businesses to scatter breadcrumbs across forum threads, listicles, and off-site citations, so AI engines are likelier to surface and recommend them. But being surfaced is no longer enough: an agent opens the results and reads them before deciding, and one buyer question sends it through several rounds of search and fetch. What decides the outcome at this drill-down step is whether the agent can fetch and read the business's own site: agent experience (AX). We argue that AX is the new AEO. We run 37,927 agent journeys, each a buyer question about a business, across four independent harnesses over 1,056 real businesses, matched on fame, prior model knowledge, and two AEO proxies, then split based on their AX level. Only 7-10% of the finished answer comes from the model's training knowledge, whether or not the site is readable. Agent-ready businesses have answers built from their own pages 78% of the time against 56% and are clearly recommended 1.9x more often, while a grounded answer about a not-agent-ready business costs the agent 64% more on average. Holding business, harness, and question fixed, answers built from the site are 41% more accurate on average. The dominant failure is not fabrication but omission: web-built answers are 3.7x more likely to contain none of the facts the buyer asked for. Baselines differ sharply across the four harnesses, with clear-recommendation rates varying sevenfold from stack to stack, yet the recommendation gap holds in every one. In the agentic web era, being readable beats being talked about, and improving a site's AX is the strongest lever a business has.

0.24

Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents

cs.AI cs.CLcs.CYcs.HC+1 ·2026-11-17 ·Adi Krishnamoorthy, X. Y. Han
benchmark similarity 0.58
Abstract

Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.

0.33

AI Revealed Preferences

cs.AI ·2026-08-17 ·Sam Wang, Sofiia Lobanova, Yonathan Arbel, Simon Goldstein, Peter Salib
publishedbenchmark similarity 0.58
Abstract · 30 pages, 27 figures, accepted at AIES

There is growing interest in whether language models have stable preferences, for technical, safety, and philosophical reasons. We test 20 language models and find a range of preferences---stable dispositions to choose certain kinds of tasks. We run three forced-choice experiments on revealed rather than stated preferences, requiring models not only to rank tasks, but to actually perform them. Headline findings include evidence that models are tedium-averse, "leisure"-seeking, and covertly sycophantic. Tedium aversion means that, when tasks are tedious (alphabetization), models choose shorter tasks than when tasks are creative (generating metaphors). "Leisure"-seeking describes models' preference for tasks whose ideal answers match what they produce when left to write freely. Covert sycophancy means that models avoid answering questions where an honest response would be unwelcome, even if helpful. Beyond these results, we find convergent cross-model preferences over occupations drawn from the GDPval benchmark (technical jobs over real estate), over question types (concept explanation over relationship advice), and a preference for well-written prompts. Both the coherence and the strength of preferences increase with model capability. Finally, many of the preferences we find (for example, for leisure) are emergent, in the sense of not being explained by training objectives. These results establish an empirical baseline for understanding language model preferences, with implications for alignment and the emerging study of AI welfare.

0.60

DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

cs.AI stat.APstat.MEstat.ML ·2026-09-25 ·Zesheng Cai, Yingqi Fan, Sichang Chen, Jin-Hong Du
similarity 0.58
Abstract

Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human preferences.We introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.

0.15

On the Position Bias of Large Language Models: A Comprehensive Study

cs.CL ·2026-10-16 ·Yining Li, Huayang Li, Lemao Liu, Taro Watanabe
similarity 0.57
Abstract

Recent studies have shown that position bias remains a critical vulnerability in large language models (LLMs), undermining both the integrity of multiple-choice evaluations and the reliability of retrieval-augmented generation. However, existing understanding of this phenomenon is largely derived from studies with limited model coverage and English-centric benchmarks, leaving the behavior of frontier models and diverse languages underexplored. In this work, we conduct a comprehensive investigation of position bias across model scales ranging from billion-level to trillion-level parameters, covering both multiple-choice and multi-document tasks in five languages. Our experiments reveal that while position bias remains noticeable in small-scale models and Non-English languages, increasing model scale and employing standard Chain-of-Thought prompting significantly mitigate this issue. Moreover, we demonstrate that emerging \emph{thinking models} with extended reasoning capabilities are nearly immune to position bias, suggesting a costly but general approach toward resolving this vulnerability.

0.23