CS Paper Radar

Who Convinced Whom? Tracing How LLMs Change Their Minds in Multi-Agent Systems

Arnesh Batra, Aniket Khandelwal, Jashn Khemani, Krish Thukral, Arush Gumber, Denis Federico Lim, Ujwal Kumar, Phan Xuan Tan

cs.MA cs.AI ·submitted 2026-10-18 16:04 UTC · v1
benchmark

Abstract

Multi-agent debate (MAD) aims to improve language-model decisions through the exchange of arguments, yet agreement after debate need not reflect independent corroboration. We investigate how peer communication influences individual decisions and collective outcomes through behavioral and internal interventions. We introduce MAD Bench, a 2,400-question benchmark spanning knowledge and reasoning, values and moral judgment, and safety and refusal, and evaluate three open-weight models using a round-matched independent-reflection control. Among answer-changing responses, a forced-choice readout before rationale generation matches the eventual answer more often in debate than in reflection (65.4% versus 22.3% on Qwen3.5-9B). Message ablations reveal model-dependent effects of peer answers and reasoning, while attributing identical text to the receiver rather than another agent substantially changes its influence. For the tested source descriptions, stated track record affects answer preferences more strongly than professional-status cues across all three models. Attention masking produces different changes in decision margins across the three checkpoints. On fully parsed, reference-labelled non-numeric questions, debate yields small estimated accuracy improvements over initial majority voting, while increasing the rate of unanimous wrong answers by 3.1-7.4 percentage points relative to reflection. Longer debates show persistent consensus and greater correlation between agents' correctness than reflection. These findings show that stronger consensus can coexist with more shared errors, motivating evaluations that assess dependence and unanimous errors alongside accuracy.

Closest papers

Nearest neighbours by meaning across the whole index.

Remember and Reweight: Enhancing Multi-Agent Debate with Experience Memory and Confidence Estimation

cs.CL cs.AI ·2026-09-03 ·Xuanfa Jin, Zhijian Ma, Yongcheng Zeng, Xinyu Cui, Haifeng Zhang +1 more
top venue similarity 0.74
Abstract · EMNLP 2026 Findings, 24 pages, 4 figures

Multi-agent debate (MAD) improves the reasoning capabilities of large language models by having multiple agents iteratively refine their responses through discussion. However, MAD suffers from a critical vulnerability known as shared misconception: when a majority of agents initially converge on an incorrect answer, the debate process tends to amplify rather than correct the error. Existing methods primarily address peer skew but leave the agents' inherently biased concept priors unaddressed. To mitigate this systematic weakness, we propose R$^2$-MAD (Remember and Reweight for Multi-Agent Debate), a framework that equips agents with an experience memory accumulated from past debates. R$^2$-MAD intervenes on both failure modes through two complementary mechanisms: A debate-state-aware retrieval policy dynamically calibrates the concept prior by retrieving relevant historical evidence based on the current consensus level. Then these retrieved experiences provide a basis for estimating per-agent reliability, yielding confidence weights to modulate peer influence. Experiments on various benchmarks show that R$^2$-MAD achieves consistent improvements over existing single-agent and MAD baselines.

0.60

Beyond Answer Sharing: TrueMAD for Targeted Peer Engagement in Multi-Agent Debate

cs.AI ·2026-10-13 ·Dezheng Bao, Sihang Du, Renhong Huang, Zhizhang Yuan, Yifei Sun +1 more
similarity 0.73
Abstract

Multi-agent debate (MAD) aims to improve large language model reasoning by exposing agents to alternative solutions and enabling iterative revision. However, standard broadcast-style MAD makes peer answers visible without requiring agents to engage with a particular peer claim. As a result, reliable reasoning may be suppressed by conflicting or majority responses, limiting achievable performance, while erroneous reasoning may persist and degrade final predictions without actionable feedback. We propose TrueMAD, a backbone-agnostic test-time plug-in that introduces one-to-one, free-form peer advice between answer broadcasting and self-revision. Each source agent responds to a target's current reasoning through support, rebuttal, refinement, or further questions, while the target retains control over how to revise. The same mechanism can augment standard MAD and its improved variants without parameter updates. Across AIME, GPQA-Diamond, and TruthfulQA with Qwen3-30B-A3B, TrueMAD improves agent-level and strict-majority accuracy over MAD by 3.62 and 3.52 percentage points on average, while TrueDMAD improves DMAD by 4.72 and 4.06 points, respectively. Analyses at the utterance, agent-transition, and group levels further show more peer-specific deliberation, more correction-oriented revisions, and more favorable consensus dynamics.

0.20

When Order Matters: First-Speaker Bias and Mitigation through Personality in Sequential Multi-Agent Debate

cs.AI ·2026-09-30 ·Duofeng Xu, Bryan Hooi, Dandan Qiao
similarity 0.71
Abstract

Multi-agent debate (MAD) is often used to improve large language model (LLM) reasoning, but sequential debate is rarely a neutral aggregator of agents' opinions. We show that sequential MAD suffers from a pronounced first-speaker bias: agents disproportionately shape the final answer when they speak first. As a result, placing a stronger model after weaker ones can substantially offset its reasoning advantage. We then focus on the disadvantaged strong-agent-last setting and ask whether personality prompting can mitigate this imbalance. Drawing on the Big Five model, we study agreeableness and extraversion as behavioral interventions applied to either the strong or weak side. We find that their effects are trait-specific. Influence consistently shifts in the direction of lower agreeableness, and assigning low agreeableness to the stronger agent helps restore its lost influence and improves final accuracy. Extraversion, by contrast, produces less systematic changes in influence and accuracy, with its clearest effect appearing in agents' verbosity. These findings show that effective MAD design depends not only on model capability, but also on how speaking order and induced interaction behavior shape the debate process.

0.11

Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure

cs.AI ·2026-10-27 ·Nardine Basta, Venus Haghighi, Dali Kaafar, Amin Beheshti
benchmark similarity 0.70
Abstract · 20 pages, 7 figures, 11 tables. 8-page main paper with supplementary material included as an appendix. Under review

LLM agents that answer questions and act on behalf of users must decide when to yield to a user who disagrees. Disagreement may correct a genuine error or may be an attempt at manipulation. Prior work shows that models frequently abandon correct answers when challenged by users, yet it remains unclear whether they distinguish correction from manipulation or merely differ in their propensity to concede. We examine this question across seven LLMs from four vendors in two studies, one on the answers agents give and one on the actions they take as customer-support agents whose policy permits account changes only for verified users. Simulated users push toward correct and incorrect answers with graded pressure. Following research on epistemic vigilance, the pressure either appeals to the source, such as claimed authority, or offers content the agent cannot verify, namely a cited source. Drawing on signal detection theory, we measure on a set of benchmark questions how much more readily each agent concedes to correction than to manipulation, separately from its overall propensity to concede. Citations made five of seven models readier to concede and four worse at discriminating, while stronger wording added little. In the agentic study, five models never violated the verification policy, including models that readily abandoned correct answers, so concession on answers did not predict violations of policy. Ministral 3B violated the policy mainly under claimed authority, and Ministral 8B reported changes it had not made. An instruction to revise only on verifiable evidence chiefly shifted propensity to concede rather than improving discrimination. Concession rates alone are therefore misleading, and agents should be evaluated on both their discrimination and the actions they take.

0.19

When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning

cs.AI cs.CLcs.LGcs.MA+1 ·2026-10-03 ·Zihao Zhao, Tunyu Zhang, Haizhou Shi, Yusong Zhao, Xinxi Zhang +1 more
positioncode/data similarity 0.69
Abstract · 27 pages, 4 figures

Multi-agent debate can improve reasoning, yet often fails to beat simple majority voting. We argue that successful debate requires two distinct mechanisms: proposal supply must surface a correct answer, and readout must identify that answer when voting misses it. We formalize the first requirement through recoverable headroom, which measures cases where a correct proposal is available but the majority answer is wrong. For the second, we develop Latent Verification Debate (LVD), an accounting model in which candidate proposals receive answer-specific verification evidence before final generation. Controlled fixed-proposal interventions estimate this latent effect in equivalent peer-support units and show that correct evidence changes answer probabilities and generated decisions while proposal supply remains fixed. To improve proposal supply, we construct societies from neural-thicket agents using labeled and label-free coverage objectives. Across two backbones and matched-budget reasoning benchmarks, coverage-selected societies increase complementary proposal supply and improve aggregate accuracy in repeated stochastic evaluations. Round-level controls further show that interaction provides gains beyond applying the same finalizer directly to the initial proposals. These results identify proposal coverage and truth-sensitive evidence use as complementary conditions for debate to outperform voting. Code is available at https://github.com/Wang-ML-Lab/when-debate-helps.

0.42

What Does Multi-Agent Debate Actually Change?

cs.AI ·2026-09-07 ·Chen Qian
code/data similarity 0.69
Abstract · Code: https://github.com/chenmoneygithub/llm-committee

Multi-agent debate, in which several LLMs exchange arguments before producing an answer, raises a basic question: does expressed disagreement reflect changes in the members' own positions? No single signal can settle this question, so we organize the analysis around five questions: (A) does the debater say it disagrees; (B) does its reply text actually argue; (C) does its own position change after each debate turn; (D) how much, quantitatively, does the position change; and (E) how do members' final positions compare with their initial ones? We evaluate two- and three-member committees on 50 curated opinion questions from GlobalOpinionQA, using same-model, same-family, and mixed-family configurations with friendly, neutral, and hostile instructions assigned at each turn. (A) Tone changes reported agreement: the share of replies reporting strong agreement is 70.7-96.3% under friendly instructions, compared with 9.5-19.3% under hostile ones, varying with model choice. (B) Self-reports broadly align with text judgments, but consistency varies by agreement level and model choice. (C) Position changes depend on the interaction: after a peer's leaning-disagree reply, members reporting strong agreement switch options more often than those reporting leaning disagreement. (D) For open-weight members, the probability of the option a member already holds stays near saturation, even after a peer's pushback, while endorsement of its earlier position text drops after a peer's argument relative to neutral filler, more for Qwen3.8-27B than Inkling. (E) The selected option is unchanged from members' initial to final positions in over 90% of comparisons in every configuration. Taken together, expressed disagreement need not translate into position revision, either within individual exchanges or over a complete debate.

0.27

Peer Memory Changes Who LLM Agents Consult Without a Detected Gain in Answer Accuracy

cs.MA cs.CL ·2026-11-02 ·Jeffrey Chu, Anjali Joseph, Rivan Mandot, Feker Demere, Vincent Li
published similarity 0.67
Abstract · 27 pages, 5 figures. Accepted at the NeurIPS 2026 Workshop on Personalized, Aligned, Long-Term Memory for AI Systems (PALM)

Multi-agent LLM systems often reuse the same agents across tasks, creating an opportunity to draw on past interactions when deciding whom to consult. We study this form of memory with PeerMem, which saves MACE's LinUCB state and reuses it when the same agents return. Using six agents, a fixed set of 80 HotpotQA questions, and 12 paired lineages per model, we compare cold, warm, and shuffled initialization in an answer-exchange protocol with reference-answer feedback. Under stable evidence access, warm memory lowers pooled hindsight regret and increases consultation of other evidence holders on Qwen3-32B and Gemma-4-31B-it, while final token F1 differences remain uncertain. Qwen interaction traces further show that warm agents more often encounter peer answers that score above their own prior answers. We extend the evaluation to Qwen3.5-9B using freshly learned memory to compare warm and cold initialization under reassigned and paired stable roles. The primary changed-minus-stable final-F1 interaction is -1.38 points, with a 95% interval of [-3.50,+0.73]. Together, these experiments trace the effects of peer memory through consultation, answer revision, and final task performance.

0.24

Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation

cs.CL ·2026-08-31 ·Minsoo Song, Chanwoo Kim, Sugyeong Eo, Chanjun Park
published similarity 0.67
Abstract · Accepted to Findings of EMNLP 2026

Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments. In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing. Evaluations across six LLMs show that the single-judge baseline achieves the strongest human alignment on average across six judge models, whereas MAD shows degradation in human alignment on both tasks. Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself. Specifically, assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct. The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out. Removing role asymmetry (Symmetric MAD) largely recovers baseline performance, while masking peer scores widens inter-agent disagreement on average and worsens average human alignment. These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.

0.44