CS Paper Radar

AutoSim: A Self-Improving User Simulator for Conversational Search Evaluation

Saber Zerhoudi, Jelena Mitrovic, Michael Granitzer

cs.IR ·submitted 2026-10-20 11:40 UTC · v1
published

Abstract

User simulators promise to evaluate conversational search systems at a scale human studies cannot reach, but only if their behavior is realistic enough to trust. We study AutoSim, a conversational-search user simulator whose relevance judgments rank a controlled pool of retrieval systems in agreement with NIST across three TREC iKAT years (Kendall's $τ$ of $0.66$, $0.91$, and $0.91$; all $p{\le}0.002$), an agreement that survives removing its persona. AutoSim self-improves against a held-out signal from real users, and we find that fine-tuning its weights makes it behave like real users, which in turn gives a more accurate estimate of how well a system serves them. Reaching that behavior takes more than prompting: when discriminating between real and simulated user sessions, no prompt we tried lowers a held-out real-versus-simulated classifier below AUC $0.85$, whereas fine-tuning the weights drives it to $0.45$, where neither the classifier nor a human reading full dialogues can tell it from a real user. That added realism is what makes the evaluation more accurate: where real users retrieve little, a prompted simulator over-estimates retrieval effectiveness by $+0.136$ nDCG and returns a relevant passage on $57\%$ of the turns where real users find none, enough to change a deployment decision. Only the weight-level simulator escapes this bias. We release the framework and code.

Comment
8 pages, 4 Figures, SIGIR-AP 2026

Closest papers

Nearest neighbours by meaning across the whole index.

MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

cs.CL cs.HC ·2026-08-25 ·Ryuichi Sumida, Koji Inoue, Tatsuya Kawahara
top venuebenchmarkcode/data similarity 0.67
Abstract · Accepted to EMNLP 2026 (Main Conference). 31 pages, 15 figures, 28 tables

Memory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a 4-month deployment (40 users, 1,872 sessions, 7 memory conditions). Existing-benchmark Direct QA varies from 19.7% to 70.1% across the 7 conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response). To examine this, we introduce MemUse, a set of real user-cued memory moments drawn from the deployment, scored by an integration-aware judgment of the natural conversational response. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation -- a 71-point gap. Within these moments, Natural Integration is associated with satisfaction, whereas Direct QA is not. We release the deployment corpus and MemUse together with all judgments and scoring prompts at https://github.com/ryuichi-sumida/memuse.

0.72

Training User Simulators with Latent Intrinsic Feedback

cs.CL ·2026-10-09 ·Tarek Naous, Craig Boutilier, Wei Xu, Ofer Meshi
similarity 0.66
Abstract

User simulators are gaining attention as environments to train LLM assistants that can better cope with the nuances of multi-turn conversations with real users. Prior work on user simulation has emphasized generating realistic user utterances, leaving unexplored their potential as a source of direct feedback on the assistant's responses. Such evaluation signals may not be explicitly communicated in language, but can instead reside in the latent reasoning of users. In this work, we introduce user simulators with intrinsic feedback, namely, models equipped with latent evaluation states: internal thoughts, reactions, and scalar ratings that evaluate the assistant from a first-person user perspective at each conversation turn. We train LLM-based simulators using interaction logs between real users and assistants and demonstrate that intrinsic feedback improves simulation realism and critiquing behaviors versus prompted simulator baselines, while also providing turn-level evaluations that existing fine-tuned simulators lack. Further, evaluations generated by our trained simulators correlate better with human judgments than those of prompted simulators of the same size. Lastly, our experiments reveal that simulators with intrinsic feedback provide effective reward signals for multi-turn RL training, matching setups that rely on external judge LLMs for rewards while cutting RL compute by more than half.

0.13

Faithful Query Resolution for Conversational Search

cs.IR ·2026-10-10 ·Saber Zerhoudi, Ivica Kostric, Jelena Mitrovic
similarity 0.66
Abstract

Conversational query rewriting makes context-dependent requests explicit for retrieval, but additional context can introduce an information need the user did not express. This raises a question about the interaction between query fidelity and retrieval breadth. This paper examines that interaction in behavior-adaptive conversational search, using iKAT SCAI 2026 as the evaluation setting. We compare five query-generation configurations within a behavior-adaptive assistant using BM25 and SPLADE retrieval. A minimal-faithful resolver preserves the user's wording and resolves only context-dependent references; weighted reciprocal rank fusion combines its rankings with those from a learned rewriter and the raw utterance. Offline evaluation shows that adding these formulations recovers recall over the faithful query alone, with a slight reduction in nDCG@10. Across our five iKAT SCAI 2026 submissions, the weighted configuration obtains the highest mean retrieval scores, reaching nDCG@5 of 0.534, a 5.3\% relative improvement over the strongest alternative. However, this retrieval advantage does not coincide with the highest conversational satisfaction scores. These findings motivate separating faithful query interpretation from retrieval expansion, while assessing their effects on conversational utility independently.

0.14

Confident Fools or Careful Searchers? A Behavioural Study of When LLMs Search Beyond Their Static Internal Memory

cs.AI ·2026-10-30 ·Nipun D. Pathirage, Kwan Srijomkwan
benchmark similarity 0.64
Abstract

Agentic LLM assistants must repeatedly decide whether to answer from memory or to call a tool such as web search. Unnecessary searches cost latency and money; missing ones yield confident but unsupported answers. Prior work mostly measures verbalised confidence or trains new adaptive-retrieval methods, and rarely audits the search decisions that deployed products actually make. We introduce a benchmark of 400 multiple-choice questions that combines widely known inside-knowledge (IK) questions, which should be answered directly, with outside-knowledge (OK) questions about facts from news published after the models' training cut-offs (to our knowledge), which should trigger search or, without tools, abstention. Each item offers an abstain option, and two action-level scores, NAS and KIS, reward searching on OK questions and restraint on IK questions. Across five commercial LLMs with and without their vendors' web search, GPT-5 mini and Grok 4 Fast search on over 90% of OK but only 1% of IK questions, whereas Mistral Small searches on just 20% of OK questions, often abstaining instead of using its tool; one product's search indicator proved unusable. Searching is partly triggered by explicit temporal cues: on uncued OK questions, the NAS of the two selective models falls from about 0.72 to 0.57--0.58. The scores thus measure the search triggering of specific products, not metacognition in a strict sense.

0.24

Why Bother Me Again? Benchmarking Personal Agent Memory for Interaction Preferences

cs.AI cs.CL ·2026-10-21 ·Jialin Li, Zhenhao Chen, Hanjun Luo, Hanan Salam
code/data similarity 0.64
Abstract · 97 pages. Project page: https://mempabench.com

Personal intelligence systems are taking on increasingly complex everyday tasks, yet repeated questions, unnecessary confirmations, and unwanted outreach can turn capable assistance into interaction burden. Reducing this burden requires agents to remember how users want to be assisted. However, existing memory benchmarks focus on factual recall and personalized answers and rarely test how agents learn and act on interaction preferences. We introduce MemPABench, an interactive environment for on-policy evaluation of memory in personal agents. MemPABench poses long-horizon tasks requiring agents to decide when to clarify, confirm, or act autonomously. Four grounded simulated users hold 14 preferences across work and personal contexts. Their feedback spans 99 sessions over four months, with five event-driven changes per user testing memory updates. We evaluate preference selection, delivered behavior, and sensitivity to context and preference change. Evaluating four memory systems across four LLM backbones, we find that current memory brings limited gains in preference selection. Retrieval-based systems (i.e. Mem0, MemOS, simple RAG) do not outperform No Memory, while File-based memory improves accuracy but applies preferences beyond relevant contexts. Our analysis shows that current memory struggles to retrieve interaction preferences and adapt to preference changes. Even with the correct preference available, overriding the agent's default interaction habits remains challenging. The environment and dataset are available at https://mempabench.com.

0.26

Measuring User Success in Randomized Search Experiments: Interpreting Good Abandonment in Generative AI Search

cs.IR ·2026-11-01 ·Minha Hwang, Shavak Agrawal, David Chen, Carto Wong, Xin Du +3 more
similarity 0.64
Abstract · 2026 CODE@MIT Conference

Randomized search experiments increasingly evaluate experiences in which large language models answer information needs directly on the results page. Click status alone cannot identify success: a click may lead to an unsatisfactory result, while no click may indicate successful answer consumption, or "good abandonment." Prior research has shown that good abandonment is difficult to detect because dwell, scrolling, and reformulation are meaningful only in context. We present Session Success Rate (SSR), a human-grounded, interpretable outcome metric designed for online controlled experiments. Trained judges label complete session experiences using privacy-protected interaction evidence and video recordings; these offline labels train models scored online from query events, viewport behavior, result-page interactions, and semantic reformulations. Out-of-sample evaluation on 6,904 labeled queries yields 0.75 balanced accuracy overall and 0.68 within abandoned queries; session-level balanced accuracy is 0.68. The sample shows why interaction status is not an adequate outcome: 53.2% of abandonments were successful and 6.5% of clicked queries were unsuccessful. In a validation portfolio of seven ranking and five non-ranking experiments, SSR agreed with Sessions per User in seven cases, was inconclusive in four, and disagreed in one under a common p < 0.05 rule. Separately, across approximately 4,000 production experiments, SSR produced about 2.5 times the binary sensitivity of Sessions per User; in the studied workflow, this supported lower traffic allocations and greater experiment throughput under a fixed traffic budget.

0.30

Towards User Simulators That Stay On Task And Still Sound Natural

cs.CL ·2026-10-22 ·Mustafa Omer Gul, Philippe Laban, Tanya Goyal, Jennifer Neville
similarity 0.64
Abstract · 30 pages, 16 figures, 6 tables. In submission

Users increasingly rely on AI systems for extended, multi-turn tasks, yet evidence indicates that LLM-based system performance degrades over prolonged interaction. To evaluate and improve AI assistants in such settings, we need user simulators that faithfully follow complex user intents while exhibiting realistic conversational behavior. Existing approaches manifest a trade-off: prompted assistants can follow complex user intents but lack behavioral fidelity, while specialized user simulators better approximate conversational behavior but struggle to remain faithful to complex intents. In this paper, we propose a reinforcement learning (RL) recipe that jointly rewards task intent adherence and behavioral naturalness. A crucial contribution of the method is operationalizing naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. We release the resulting model, UserLM-2.0, which produces valid simulated conversations (i.e., meeting both task-faithfulness and naturalness criteria) 46.2% of the time, over three times the rate of the strongest prior user simulator (14.1%).

0.08

Investigating Assistant Bias in LLM User Simulators Using a Role Vector

cs.CL cs.HC ·2026-09-01 ·Daeheon Jeong, Yoonjoo Lee, Eugene Choi, Sinie van der Ben, Juho Kim
top venue similarity 0.63
Abstract · 35 pages, EMNLP 2026 Findings

LLM-based user simulators are increasingly used to evaluate autonomous agents at scale, in place of costly human evaluations. Despite this promise, these simulators exhibit "assistant bias," a tendency to cooperate and pursue task goals. They rarely reproduce the frustration or disengagement that real users exhibit, compromising evaluation validity. Prior work outlines that this bias is baked in during model training, which role-playing prompts fail to override. We analyze this bias from model activations, extracting a user role vector by contrasting how the model represents user versus assistant perspectives on the same dialogue. We observe two findings: (i) the user direction is identifiable in activations, elicits user-like behaviors, and captures characteristics distinct from assistant traits; and (ii) although user-role activation associates with simulation realism and steering strengthens it, it can exaggerate user behaviors and override individual user profiles. Together, our findings provide a representation-level analysis of LLM user simulators, confirming that assistant bias is structurally identifiable and that user behavior can be directionally analyzed.

0.31