CS Paper Radar

From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration

Ala N. Tak, Teruhisa Misu, Kumar Akash, Zhaobo K. Zheng, Kevin H. Joo, Jonathan Gratch

cs.MA cs.AIcs.CL ·submitted 2026-09-06 17:58 UTC · v1
top venuehot direction

Abstract

LLM agents are increasingly used for collaborative problem solving and human-group simulation. This makes outcome-only evaluation insufficient: if LLM groups are used as models of human groups, we need to know whether they succeed or fail through human-like deliberative mechanisms. We compare human group chats with matched LLM deliberation traces on Wason-style deductive reasoning, then test whether the same process signatures generalize to analogical, abductive, and analytical tasks. Humans and LLMs show the same assembly bonus asymmetry: discussion improves the average member more often than the best initial member. Initial-answer diversity accounts for the effect of model heterogeneity, increasing movement in both corrective and destructive directions. The main differences are process-level. Compared with humans, LLM groups follow majorities more often, surface less unique information, and converge earlier; correct minority signals succeed mainly when re-expressed early. Interventions motivated by human group-decision research yield modest improvements in collective outcomes, but do not remove the coordination bottleneck. Together, these results suggest that LLM groups can reproduce some outcome-level patterns of human deliberation while diverging in the mechanisms that generate assembly bonus and process loss, with implications for group simulation and human-AI collaboration.

Comment
Accepted at EMNLP 2026. Camera-ready version

Closest papers

Nearest neighbours by meaning across the whole index.

Language-model groups overstate consensus when replaying human deliberation on a reasoning task

cs.AI cs.CLcs.CYcs.MA ·2026-09-17 ·Tengfei Shao
code/data similarity 0.70
Abstract · 37 pages, 4 figures. Preregistration: https://osf.io/5jp7s . Code and data: https://doi.org/10.5281/zenodo.21318346

Full-consensus rates are often treated as indicators of collective cognition, yet depend on how participation and final states are operationalized. We replayed 100 held-out human Wason groups with matched large language model (LLM) agent groups, seeding one belief-anchored agent per participant's pre-discussion answer and scoring agents and people with the same code. Across human scoring definitions, estimates ranged from 24.0% to 57.0%; about one fifth of participants never posted, whereas agents almost always did. Agent groups remained more consensual in two post-unblinding sensitivity analyses: the submit-based comparison (n = 98) yielded gaps of 34.0 and 43.9 percentage points for chat and reasoning modes, and the participation-matched comparison (n = 45) yielded gaps of 34.1 and 44.4 points. These complementary routes reduced different measurement asymmetries yet converged within 0.5 percentage points. The gap persisted without early stopping and under a reparameterization removing the memorizable answer; reasoning-mode groups then agreed nearly unanimously, mostly on incorrect answers. Simulated consensus did not track collective accuracy, and belief-anchored agent groups were biased estimators of the human group-outcome distribution in this setting. These analyses provide a scoring-explicit basis for assessing simulated-group estimates of human deliberative outcomes.

0.41

Human-LLM Cognitive Extension: A Framework for Studying Coupled Cognition

cs.CY cs.HC ·2026-11-18 ·Ernesto Biempica, Emiliano Bruner, Juan F Poyatos
position similarity 0.68
Abstract · 24 pages, two supplementary tables

Recent research on the use of Large Language Models (LLMs) reports divergent cognitive outcomes, including improved assisted performance, limited transfer, and compromised unaided learning. Here, we argue that part of this seeming inconsistency arises because studies often assess different cognitive systems without making the unit of analysis explicit. Building on work in extended and distributed cognition, we propose the Human-LLM Cognitive Extension (HLCE) as a methodological framework for matching experimental designs, measures, and causal inferences to four distinct systems: the human acting independently, the artificial agent, the joined human-LLM configuration, and the human after division. The framework also treats the coupling architecture as part of the experimental object. HLCE separates performance augmentation, system-level complementarity, post-interaction human change, and stronger claims of constitutive cognitive integration. This separation can improve the interpretation of existing evidence and guide the design of human-LLM interactions without assuming in advance that these configurations constitute extended minds. We illustrate the application of HLCE with a synthetic diagnostic experiment comparing two connecting configurations that differ in how supporting information is communicated. The example highlights how distinct couplings can shape the organization of joint problem-solving, even when the final outcome remains unchanged. Overall, HLCE provides a methodological basis for designing controlled experiments and interpreting evidence across distinct forms of human-LLM cognitive interaction.

0.26

PARLEY: Evaluating LLM Collaboration Through Cooperative Asymmetric Puzzles

cs.MA ·2026-11-10 ·Javier Hernandez, Andrew D. Wilson
similarity 0.65
Abstract · 12 pages, 8 figures, 2 tables

As large language models power autonomous agents, understanding how they collaborate and where they fail becomes critical. We introduce PARLEY, an extensible platform for studying LLM agent collaboration through four asymmetric puzzle tasks grounded in cognitive psychology (referential communication, joint coordination, perspective-taking, and spatial reasoning). GPT-4o agent pairs succeeded in 60.5% of conversations (N=400, 100 per task), with task-level variation (34-80%) reflecting distinct collaborative demands. From these conversations, we derived the data-driven PAC taxonomy: six failure types organized into Perception (visual interpretation and spatial reasoning), Adaptation (progress tracking and feedback integration), and Coordination (verification and grounding). Failure to incorporate external feedback was a critical indicator of unsuccessful collaboration, with agents repeatedly applying unsuccessful strategies. We also show that a taxonomy-guided intervention (N=200) improved average task success to 70.5%. These results demonstrate how PARLEY can be used to derive meaningful, data-driven insights that can in turn lead to improved agentic collaboration.

0.16

AI agents reshape consensus formation in human groups

cs.CL cs.CYcs.SI ·2026-09-02 ·Lin Chen, Ziyi Liu, Xia Hu, Yong Li
similarity 0.65
Abstract

As large language model (LLM) agents shift from tools to participants in human groups, a fundamental question for collective behavior is how their growing presence reshapes consensus formation. Here we study mixed human-AI groups in a collaborative description game, in which shared conventions emerge through repeated rounds of random pairwise communication. Varying the proportions of LLM agents, we identify three distinct regimes of consensus formation: low agent proportions facilitate human-led consensus, intermediate proportions disrupt convergence, and high proportions restore strong consensus while shifting it toward agent-led conventions. Crucially, these regimes differ not only in the strength of convergence, but also in the semantic grounding and communicative form of the resulting consensus: human-led consensus is more concrete, holistic, and grounded in shared real-world analogies, whereas agent-led consensus is more abstract, less information-dense, and more geometrically segmented. Mechanistically, agent influence arises from a shared linguistic prior that places agents near one another in the expression space, combined with relatively stable expression choices across rounds; humans initially resist adopting expressions from partners perceived as AI but gradually yield to conformity pressure. These findings provide evidence that AI composition can shape the emergence, content, and perceived legitimacy of group norms, making agent proportion and transparency important design variables for human-AI systems.

0.19

Understanding the Effects of Multi-Agent Deliberation for Human-AI Decision-Making

cs.HC cs.AI ·2026-11-11 ·Harry Yizhou Tian, Ming Yin
similarity 0.63
Abstract

Multi-agent deliberation (MAD), where multiple LLMs engage in structured discussion, improves AI performance on many benchmarks, but whether it helps humans make better decisions remains untested. We conduct two user studies on LLM-response-verification and fact-checking. We first compare MAD against common AI decision-support methods that isolate its two defining dimensions --- multiple AIs and multi-round interaction. We find MAD outperformed single-round AI assistance on both tasks; multi-round interaction was the main driver of this benefit, while a second AI added value when the single AI had room to improve. Then, we study two design choices in MAD --- AI persona and human engagement level --- finding a devil's advocate persona to lower accuracy and user experience, while humans' active participation in MAD improved accuracy in one task at the cost of frustration. We conclude by discussing implications for when, why, and how multi-agent deliberation can support human decision-making.

0.11

LLM Facilitation of Group Decision-Making

cs.HC ·2026-11-14 ·José Córdova, Chun-Wei Chiang, Miriam Schirmer, Ming Yin, Emőke-Ágnes Horvát
similarity 0.62
Abstract

Human groups often fail to integrate information distributed across their members, converging on incorrect decisions. We ask whether large language models (LLMs) can help groups reach the correct decision in such settings. In an online, chat-based experiment, 138 three-person groups (N = 414) completed a hidden-profile hiring task that required selecting the best candidate using public information available to all group members and private information available only to individual members. The randomized human-subject experiment included three conditions: no facilitation, an individual-level LLM that messaged each member privately, and a group-level LLM that messaged the whole group. The LLM facilitation consisted of adaptive interventions that summarized the discussion, nudged quiet members, and played devil's advocate. The LLM did not know which candidate was correct. Both LLM conditions significantly increased the rate at which groups reached consensus on the correct candidate, from 32.6% without facilitation to 56.5% with LLM facilitation. LLM-facilitated groups exchanged more consensus messages signaling agreement on the group's choice. Participants in the group-level condition also disclosed significantly more information about the correct candidate, whereas those receiving individual-level facilitation repeated public information less often. Finally, group-level facilitation received more favorable usability ratings than individual-level facilitation. These results show that LLMs can deliver adaptive, theory-informed interventions that improve group decision-making, and that the design of these interventions may shape how people respond to them.

0.27

What People Almost Did: Evaluating LLM Social Simulations Beyond Behavioral Fit

cs.HC ·2026-09-17 ·JaeWon Kim, Angie Boggust
similarity 0.62
Abstract

LLM-based social simulations are primarily evaluated for behavioral fit, testing whether agents reproduce the actions or response distributions of the people they are simulating. However, the promise of simulation extends beyond behavioral fit. Simulations can explain human behavior, diagnose barriers, and compare large-scale interventions. These use cases depend on understanding \textit{why} people acted a certain way, not just \textit{what} they did. As a result, behavioral fit is insufficient for these types of claims because behavior underdetermines the reasoning process behind it. For instance, the behavior of staying silent may be due to disinterest or suppressed speech, and not answering a call may be due to distrust of the caller or limited phone access. In this paper, we propose \textit{representational adequacy} as a new evaluation target for LLM-based social simulations. By leveraging LLM reasoning traces, representational adequacy measures whether a simulation's scenario--reasoning--action triples preserve the reasoning process behind the behavior in a way that is faithful to the population and scenarios being simulated. We distinguish representational adequacy from interpretability and alignment metrics, propose ways to integrate it into simulation research, and pose its measurement as an open problem.

0.15

Explaining and interrupting collective misalignment in language model systems

cs.CL cs.AI ·2026-11-09 ·Edoardo Chidichimo
similarity 0.61
Abstract

In collectives, language model agents can exchange individually innocuous messages that together support a policy violation. Understanding and preventing collective misalignment requires answering the question of which messages sufficiently influenced and necessarily enabled the violation. Collective Dependence Analysis (CDA) operationalises necessity and sufficiency to explain decisions and find the fewest removals interrupting support within tested combinations. Tests covered constructed violations, unscripted agent conversations, and recorded interactions, with selected message removals reducing violations from 93% to 6% in controlled cases. Using O(N) model evaluations for N messages, CDA achieves the same violation reduction with half the deletions of a frontier LLM judge given the transcript and asked for all supporting messages in eligible operational cases. CDA also proposes the mechanism through which the collective misalignment emerged. Nonetheless, complete explanations remain difficult on unfamiliar tasks and some removals can inadvertently disrupt correct decisions. CDA unifies explanations of collective misalignment with experimentally tested interventions.

0.24