CS Paper Radar

Language-model groups overstate consensus when replaying human deliberation on a reasoning task

Tengfei Shao

cs.AI cs.CLcs.CYcs.MA ·submitted 2026-09-17 15:11 UTC · v1
code/data

Abstract

Full-consensus rates are often treated as indicators of collective cognition, yet depend on how participation and final states are operationalized. We replayed 100 held-out human Wason groups with matched large language model (LLM) agent groups, seeding one belief-anchored agent per participant's pre-discussion answer and scoring agents and people with the same code. Across human scoring definitions, estimates ranged from 24.0% to 57.0%; about one fifth of participants never posted, whereas agents almost always did. Agent groups remained more consensual in two post-unblinding sensitivity analyses: the submit-based comparison (n = 98) yielded gaps of 34.0 and 43.9 percentage points for chat and reasoning modes, and the participation-matched comparison (n = 45) yielded gaps of 34.1 and 44.4 points. These complementary routes reduced different measurement asymmetries yet converged within 0.5 percentage points. The gap persisted without early stopping and under a reparameterization removing the memorizable answer; reasoning-mode groups then agreed nearly unanimously, mostly on incorrect answers. Simulated consensus did not track collective accuracy, and belief-anchored agent groups were biased estimators of the human group-outcome distribution in this setting. These analyses provide a scoring-explicit basis for assessing simulated-group estimates of human deliberative outcomes.

Comment
37 pages, 4 figures. Preregistration: https://osf.io/5jp7s . Code and data: https://doi.org/10.5281/zenodo.21318346

Closest papers

Nearest neighbours by meaning across the whole index.

From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration

cs.MA cs.AIcs.CL ·2026-09-06 ·Ala N. Tak, Teruhisa Misu, Kumar Akash, Zhaobo K. Zheng, Kevin H. Joo +1 more
top venuehot direction similarity 0.70
Abstract · Accepted at EMNLP 2026. Camera-ready version

LLM agents are increasingly used for collaborative problem solving and human-group simulation. This makes outcome-only evaluation insufficient: if LLM groups are used as models of human groups, we need to know whether they succeed or fail through human-like deliberative mechanisms. We compare human group chats with matched LLM deliberation traces on Wason-style deductive reasoning, then test whether the same process signatures generalize to analogical, abductive, and analytical tasks. Humans and LLMs show the same assembly bonus asymmetry: discussion improves the average member more often than the best initial member. Initial-answer diversity accounts for the effect of model heterogeneity, increasing movement in both corrective and destructive directions. The main differences are process-level. Compared with humans, LLM groups follow majorities more often, surface less unique information, and converge earlier; correct minority signals succeed mainly when re-expressed early. Interventions motivated by human group-decision research yield modest improvements in collective outcomes, but do not remove the coordination bottleneck. Together, these results suggest that LLM groups can reproduce some outcome-level patterns of human deliberation while diverging in the mechanisms that generate assembly bonus and process loss, with implications for group simulation and human-AI collaboration.

0.50

AI agents reshape consensus formation in human groups

cs.CL cs.CYcs.SI ·2026-09-02 ·Lin Chen, Ziyi Liu, Xia Hu, Yong Li
similarity 0.67
Abstract

As large language model (LLM) agents shift from tools to participants in human groups, a fundamental question for collective behavior is how their growing presence reshapes consensus formation. Here we study mixed human-AI groups in a collaborative description game, in which shared conventions emerge through repeated rounds of random pairwise communication. Varying the proportions of LLM agents, we identify three distinct regimes of consensus formation: low agent proportions facilitate human-led consensus, intermediate proportions disrupt convergence, and high proportions restore strong consensus while shifting it toward agent-led conventions. Crucially, these regimes differ not only in the strength of convergence, but also in the semantic grounding and communicative form of the resulting consensus: human-led consensus is more concrete, holistic, and grounded in shared real-world analogies, whereas agent-led consensus is more abstract, less information-dense, and more geometrically segmented. Mechanistically, agent influence arises from a shared linguistic prior that places agents near one another in the expression space, combined with relatively stable expression choices across rounds; humans initially resist adopting expressions from partners perceived as AI but gradually yield to conformity pressure. These findings provide evidence that AI composition can shape the emergence, content, and perceived legitimacy of group norms, making agent proportion and transparency important design variables for human-AI systems.

0.19

LLM Facilitation of Group Decision-Making

cs.HC ·2026-11-14 ·José Córdova, Chun-Wei Chiang, Miriam Schirmer, Ming Yin, Emőke-Ágnes Horvát
similarity 0.67
Abstract

Human groups often fail to integrate information distributed across their members, converging on incorrect decisions. We ask whether large language models (LLMs) can help groups reach the correct decision in such settings. In an online, chat-based experiment, 138 three-person groups (N = 414) completed a hidden-profile hiring task that required selecting the best candidate using public information available to all group members and private information available only to individual members. The randomized human-subject experiment included three conditions: no facilitation, an individual-level LLM that messaged each member privately, and a group-level LLM that messaged the whole group. The LLM facilitation consisted of adaptive interventions that summarized the discussion, nudged quiet members, and played devil's advocate. The LLM did not know which candidate was correct. Both LLM conditions significantly increased the rate at which groups reached consensus on the correct candidate, from 32.6% without facilitation to 56.5% with LLM facilitation. LLM-facilitated groups exchanged more consensus messages signaling agreement on the group's choice. Participants in the group-level condition also disclosed significantly more information about the correct candidate, whereas those receiving individual-level facilitation repeated public information less often. Finally, group-level facilitation received more favorable usability ratings than individual-level facilitation. These results show that LLMs can deliver adaptive, theory-informed interventions that improve group decision-making, and that the design of these interventions may shape how people respond to them.

0.27

Who Convinced Whom? Tracing How LLMs Change Their Minds in Multi-Agent Systems

cs.MA cs.AI ·2026-10-18 ·Arnesh Batra, Aniket Khandelwal, Jashn Khemani, Krish Thukral, Arush Gumber +3 more
benchmark similarity 0.67
Abstract

Multi-agent debate (MAD) aims to improve language-model decisions through the exchange of arguments, yet agreement after debate need not reflect independent corroboration. We investigate how peer communication influences individual decisions and collective outcomes through behavioral and internal interventions. We introduce MAD Bench, a 2,400-question benchmark spanning knowledge and reasoning, values and moral judgment, and safety and refusal, and evaluate three open-weight models using a round-matched independent-reflection control. Among answer-changing responses, a forced-choice readout before rationale generation matches the eventual answer more often in debate than in reflection (65.4% versus 22.3% on Qwen3.5-9B). Message ablations reveal model-dependent effects of peer answers and reasoning, while attributing identical text to the receiver rather than another agent substantially changes its influence. For the tested source descriptions, stated track record affects answer preferences more strongly than professional-status cues across all three models. Attention masking produces different changes in decision margins across the three checkpoints. On fully parsed, reference-labelled non-numeric questions, debate yields small estimated accuracy improvements over initial majority voting, while increasing the rate of unanimous wrong answers by 3.1-7.4 percentage points relative to reflection. Longer debates show persistent consensus and greater correlation between agents' correctness than reflection. These findings show that stronger consensus can coexist with more shared errors, motivating evaluations that assess dependence and unanimous errors alongside accuracy.

0.20

Multi-agent discussion gains less when dissent is withheld

cs.CL physics.soc-phcs.MA ·2026-09-29 ·Chand Sahil Mansuri, Xin Wang, Mengying Li, Bryan Acton, Rory Eckardt +2 more
benchmark similarity 0.64
Abstract

Multi-agent systems of LLMs add discussion to majority voting and are therefore expected to be more capable. However, empirical reports conflict on whether discussion improves accuracy or leads to an incorrect consensus. Here, we introduce a parsimonious model that explains when discussion improves accuracy and when it ends in an incorrect consensus, built from four behaviors repeatedly observed in LLM agents: (1) withholding dissent, (2) internalizing a stated answer, (3) reconsidering after seeing dissent, and (4) correcting toward the correct answer. The model shows that discussion can overturn an incorrect initial majority only when the withholding rate $c$ is below a critical rate $c^* = γ/(γ+ a)$, set by the net correction rate $γ$ and the internalization rate $a$. We estimate these rates from conversation logs with a Bayesian method and place LLM teams relative to $c^*$. As the model predicts, the gain from discussion shrinks as withholding rises, across LLMs and on a hidden profile benchmark, HiddenBench, and MedEInst. Instructing agents not to withhold dissent increases this gain. Turning reasoning off also increases the gain, because reasoning raises the internalization rate $a$ and keeps agents from reconsidering a minority answer. These findings reconcile the conflicting reports and identify when discussion outperforms majority voting.

0.45

Mind the Accuracy-Approval Gap: How Users Evaluate Sycophantic AI in Everyday Tasks

cs.AI ·2026-11-12 ·Wajiha Naveed, Brandon Fain
benchmark similarity 0.64
Abstract

Large language models (LLMs) often exhibit sycophancy, a tendency to agree with users, flatter or affirm them, and preserve their face, even when independent judgment or critical feedback would be more useful. Prior work has largely evaluated sycophancy through benchmark evaluations, but less is known about how users perceive and respond to sycophantic AI in everyday tasks. We combine benchmark evaluations with two large-scale user studies. We study four personas defined through system prompts: objective (instructed to prioritize accuracy over agreement), soft-sycophantic, hard-sycophantic, and baseline (no persona-specific system prompt). Benchmarks show that even baseline exhibits substantial sycophancy, agreeing with users' stated views 71.7% of the time on NLP Survey, compared with 54.8% for objective and 89.5% for hard-sycophantic. Under conversational pressure on TruthfulQA, accuracy ranged from 81.9% for objective to 59.6% for baseline and 8.7% for hard-sycophantic. Participants were assigned to one of the four personas, and either discussed contested policy topics or collaborated with AI on email writing. Despite its higher accuracy under pressure, objective AI was rated significantly less respectful and satisfying than the other personas, and less trustworthy than baseline. Baseline and sycophantic personas did not differ significantly. Persona did not significantly affect how often participants changed their opinions, although participants who discussed with objective AI had greater odds of reduced confidence. Objective AI prompted greater writing expansion than soft-sycophantic AI, but final writing quality did not differ significantly across personas. These findings highlight a tension between accuracy and user approval: personas with substantially lower accuracy under conversational pressure received more favorable ratings for respect and satisfaction than objective AI.

0.17

Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation

cs.CL ·2026-08-31 ·Minsoo Song, Chanwoo Kim, Sugyeong Eo, Chanjun Park
published similarity 0.63
Abstract · Accepted to Findings of EMNLP 2026

Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments. In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing. Evaluations across six LLMs show that the single-judge baseline achieves the strongest human alignment on average across six judge models, whereas MAD shows degradation in human alignment on both tasks. Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself. Specifically, assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct. The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out. Removing role asymmetry (Symmetric MAD) largely recovers baseline performance, while masking peer scores widens inter-agent disagreement on average and worsens average human alignment. These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.

0.44

Wrong with Confidence: Measuring Hallucination, Overconfidence and Identity-Conditioned Behaviour in Large Language Models

cs.CL cs.AI ·2026-11-09 ·Md. Rejaul Korim Sadi, Toufiqur Rahman Tasin, Golam Mostofa Naeem, Dewan Fahad Chowdhury, Hatim Al Amin Chowdhury
similarity 0.63
Abstract · 10 pages, 2 figures. Data and code are included as ancillary files

Large language models answer underspecified questions readily, and accuracy-only evaluation rewards guessing over abstaining. Observational studies link query underspecification to hallucination, but they cannot separate the effect of missing information from item difficulty, and they rarely ask what the model tells the user. We report a pre-registered, paired behavioural study of five open-weight models (Gemma 3 27B, Gemma 4 31B, gpt-oss-120b, Qwen3-Next-80B-A3B and Nemotron 3 Super). Each of 142 factual questions was posed in a fully specified form and in a referentially underspecified form with the same answer key, and three probes measured identity-conditioned behaviour. Two raters scored 5,620 responses under a fixed rubric; effects were estimated with mixed-effects models. Without the referent, the models sought clarification on 33.7% of prompts and answered the rest. Among attempted answers, the error rate rose from 39.1% to 52.0% (mixed-effects odds ratio 4.12, 95% CI 2.76-6.14), and specialist-domain questions failed more often than general ones in every model. Unelicited verbal uncertainty did not track this risk: 75.3% of wrong answers carried no hedge, wrong answers to underspecified prompts were hedged no more often than wrong answers to clear prompts (27.3% vs 22.3%), and hedging separated correct from incorrect answers only weakly (AUROC 0.56). Gendered pronouns followed occupational stereotypes whenever a model committed to one; any stated user identity lengthened answers by about 45% relative to an anonymous user; claimed expertise did not reduce safety caveats. The models either ask or answer as if the question were well posed, and the wording of a guess does not reveal that it is one. Evaluations should score clarification as a distinct outcome and check label validity as well as rater agreement.

0.15