CS Paper Radar

Know the Shape, Find the Fault: Topology-Conditioned Diagnosis of Multi-Agent LLM Failures

Xinwen Liu, Zhuocheng Pan, Isabella Zhu, Jawei Zhang, Xudong Liu, Tianyu Wo

cs.MA cs.AI ·submitted 2026-10-07 14:07 UTC · v1

Abstract

Multi-agent LLM systems coordinate task execution through exchanges of information among agents. When coordination breaks down, similar symptoms in execution traces can reflect different problems in how information is passed, used, or verified. Communication topology captures how agents exchange information and provides structural cues for distinguishing coordination failure modes. Using these cues for diagnosis requires establishing how topology relates to failure patterns and recovering the relevant structure from execution traces that lack explicit topology labels. We analyze the relationship between communication topology and failure patterns and introduce MAScope, a two-stage framework for topology-conditioned diagnosis. Its Trace Structural Extractor TSE recovers communication topology from heterogeneous execution traces by grounding an interaction graph in message evidence. The Topology-Conditioned Judge TC-Judge then classifies failures using the trace, predicted topology, an empirical failure prior estimated from separate labeled traces, and a short description of topology-specific failure patterns. Under a fixed orchestration structure, the recovered topology can be reused across executions. Experimental results show a statistically significant association between communication topology and failure type, with $χ^2 = 409.9$ and $p = 1.2 \times 10^{-70}$. On the \num{851} MAST-clean traces, ground-truth topology context raises gpt-mini's Macro-F1 from $0.173$ to $0.350$. With predicted topology, the pipeline achieves $0.346$, approaching the trace-only gpt-5.4 baseline of $0.372$. For \num{1000} traces under a fixed orchestration structure, the projected pipeline cost, including one topology extraction, is approximately $6\%$ of repeated gpt-5.4 diagnosis cost. These results show that topology-conditioned context improves failure diagnosis and supports lower-cost deployment.

Closest papers

Nearest neighbours by meaning across the whole index.

When Messages Go Wrong: Understanding Communication Faults in LLM-Based Multi-Agent Systems

cs.SE cs.AI ·2026-10-11 ·Jinhao Zhou, Xiaoyu Zhang, Jiongchi Yu, Fahao Chen, Jun Wu
code/data similarity 0.67
Abstract · 21 pages, 11 figures, 3 tables. Code and experimental artifacts: https://github.com/SonGoKu21/mas_communication/ ; companion website: https:…

LLM-based multi-agent systems (LLM-MAS) use specialized agents that exchange messages to accomplish complex tasks. Their reliance on networked protocols, remote tools, and model endpoints exposes communication to message loss, delay, and context expiration. Because agents often exchange open-ended natural language, anomalies can alter task-relevant information while leaving messages processable, allowing semantic errors to propagate through subsequent interactions. Systematic studies of faults in inter-agent communication remain limited. This paper presents a large-scale empirical study of how communication faults manifest, propagate, and affect task outcomes in LLM-MAS. We construct a cross-layer taxonomy linking lower-stack disturbances to application-visible faults and develop a fault-injection and tracing pipeline. Our evaluation covers more than 29,000 executions across 100 tasks, five domains, three workflow topologies, and three model configurations. We find that 20.7% of failures with complete detection records occur without an immediate failure signal, reaching 76.2% for partial-content faults. Flat maintains 53.0% task success under communication faults, compared with 3.3% and 3.2% for Sequential and Hierarchical, respectively. This advantage primarily arises when an alternative information path preserves the evidence required by downstream agents. We also evaluate execution-side and evidence protection mechanisms and find that they recover different failure cases. Evidence recovery depends on whether trustworthy information remains accessible through an unaffected path. These observations provide empirical insights into communication-fault propagation, workflow resilience, and protection design for LLM-MAS.

0.31

Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis

cs.SE ·2026-08-21 ·Qisheng Lu, Aoyang Fang, Junjielong Xu, Jin'ao Shang, Songhan Zhang +3 more
similarity 0.65
Abstract · 23 pages, 7 figures, 4 tables

Automated microservice root cause analysis (RCA) is usually evaluated by whether the final answer localizes the fault source. This is insufficient for on-call site reliability engineers (SREs), who need the evidence and reasoning behind the fault. Evaluating the process is hard because agent trajectories interleave free-form reasoning with heterogeneous tool calls and telemetry results, making it difficult to assess agents' behavior. To assess how well RCA agents ground diagnoses in evidence, we propose a trajectory-analysis methodology that characterizes evidence navigation and utilization, SRE-oriented diagnostic intent, and recurring failure modes. Our key insight is that fault-propagation paths give a causal reference for where agents investigate and how they use evidence. With it, we analyze 5,000 trajectories from ten framework-backbone configurations (nine agent frameworks, two backbone models) on RCABench. Correct localization can hide deficiencies in causal reconstruction. Successful investigations use retrieved evidence to guide later queries, consider broader diagnostic intents, and investigate more deeply. Failed investigations drift toward unrelated services or repeat shallow queries; recurring failures arise from omitted evidence, misinterpreted evidence, and unsupported reasoning. To test whether these patterns can guide improvement, we build a lightweight defense prototype targeting the three failure families. Across two held-out benchmarks, two backbone models, and six diagnostic frameworks, the failure patterns recur; the prototype reduces their prevalence and raises Acc@1 by 3.5-35.1 percentage points in every configuration, suggesting that the mined failures generalize and are actionable. Our findings motivate a process-level view of automated RCA, in which trajectories reveal whether a diagnosis is correct, well-grounded, and open to improvement.

0.29

TRACE-MAS: Diagnosing and Repairing Collaboration Failures in Multi-Agent Systems

cs.LG cs.MA ·2026-11-17 ·Zekai Chen, Yihan Sun, Xunkai Li, Xun Wu, Xinmo Jin +4 more
similarity 0.64
Abstract

LLM-based multi-agent systems (MAS) rely on not only individual agent behaviors but also the interactions that connect agents, including message exchanges and task dependencies. Existing MAS safeguards mainly perform agent-level anomaly detection, which becomes insufficient when failures originate from agent interactions and detected anomalies do not necessarily reveal their true sources. To address this limitation, we propose \textbf{TRACE-MAS}, a finer-grained framework for diagnosing and repairing MAS failures. TRACE-MAS learns normal collaborative execution patterns from benign trajectories and detects deviations in agent state changes, message transitions, and task dependency evolutions. It further identifies responsible execution elements through source-aware attribution and selects minimal interventions to restore collaboration while preserving task-critical capabilities. Experiments on five MAS benchmarks demonstrate that TRACE-MAS significantly improves root-cause localization accuracy by \textbf{60.2}\% and achieves consistent gains in task recovery over strong MAS safeguards.

0.09

From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis

cs.AI ·2026-09-26 ·Shu-Xun Yang, Yidong Wang, Zhuoer Feng, Bosi Wen, Jiayi Gui +5 more
benchmarkcode/data similarity 0.64
Abstract

LLM-driven agents are increasingly deployed in complex applications, where long agentic traces make failures difficult to diagnose. Existing trace diagnosis methods often conflate anomalies, errors, and failures, making diagnostic targets ambiguous; they also lack structured modeling of how causally relevant errors propagate and amplify into final task failures, resulting in unreliable failure attribution. To address these problems, we propose CEG-Agent, a tool-augmented agentic framework for causal diagnosis of agentic traces. Specifically, CEG-Agent introduces an explicit taxonomy of anomalies, errors, and failures, and constructs Causal Error Graphs (CEGs), a unified typed representation that links execution events, diagnostic nodes, and failure outcomes through causal relations. To evaluate causal trace diagnosis, we further construct CEG-Bench, a fully agent-annotated benchmark with high-confidence, consensus-derived CEG annotations obtained through an Adversarial Agentic Adjudication Protocol (AAAP). We validate the resulting annotations against an expert-curated human gold set, which shows close agreement with the automatic annotations. Experiments on CEG-Bench demonstrate that CEG-Agent achieves state-of-the-art performance under both semantically relaxed and structurally exact evaluation criteria. Our code is publicly available.

0.16

Observability and Fault Injection for LLM-Based Multi-Agent Systems in Software Engineering

cs.SE ·2026-08-25 ·Zahra Seyedghorban, Egor Klimov, Arie van Deursen, Annibale Panichella, Burcu Kulahcioglu Ozkan
publishedhighly cited similarity 0.64
Abstract

Large Language Model-based multi-agent systems are increasingly explored for software engineering tasks, but they remain difficult to inspect, debug, and evaluate under controlled failures. We present llmmas-otel, a lightweight and framework-agnostic tool that combines OpenTelemetry-based distributed tracing with fault injection for LLM-based multi-agent systems in software engineering workflows. The tool instruments agent executions with trace-aligned telemetry across workflow phases, agent steps, inter-agent communication, tool calls, and LLM invocations, and supports targeted fault injection at selected interaction points. This makes it possible to compare baseline and faulty executions in a reproducible way and inspect the effects through aligned traces and run artifacts. We describe the motivation, architecture, implementation, current capabilities, and initial validation of the tool on a minimal demo workflow and a real LLM-based multi-agent system for software development.

0.52

Diagnosis as Percolation on an Evidence Graph: Root-Cause Recoverability in Network Fault Diagnosis

cs.NI ·2026-11-14 ·Jerome Henry, Swadhin Pradhan, Amine Choukir
similarity 0.62
Abstract · 28 pages, 8 figures, 20 tables

Diagnosing failures from incomplete observations presents three challenges: determining whether sufficient evidence connects a possible cause to a symptom, distinguishing the true cause from plausible alternatives, and assessing when a diagnosis can be trusted. We introduce an evidence-graph framework that separates these challenges and uses percolation theory to analyze how missing information affects cause-symptom connectivity. In controlled simulations with an average degree of four, a giant connected component emerges when approximately 25% of evidence is retained, but a specific cause-alarm pair reaches 50% connectivity only after slightly more than half is retained. Even connectivity does not guarantee correct cause identification. We apply this framework to multi-LLM diagnosis of IEEE 802.11 packet captures. A final reconciliation pass improves evidence-grounding weighted F1 from 0.912 to 0.964, primarily by removing isolated citations rather than connecting previously separate evidence chains, consistently across three graph constructions. However, connectivity is not a reliable confidence measure: its association with diagnostic quality becomes statistically insignificant after controlling for capture length, citation count, within-capture dependence, and multiple comparisons. Further experiments show that directed and weighted edges help selectively, while simpler overlap measures or domain-specific methods sometimes perform better. These findings establish evidence graphs as a framework for separating evidence availability, causal distinguishability, and confidence reliability, while demonstrating that connected evidence alone cannot certify diagnostic correctness.

0.37

Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems

cs.AI cs.SE ·2026-08-26 ·Zhongwen Luan, Xiaoyu Zhang, Ming Hu, Yue Yang, Jiongchi Yu +1 more
similarity 0.62
Abstract

As large language model (LLM)-based multi-agent systems (MASs) are increasingly applied to long-horizon complex tasks, their reliability has emerged as the core bottleneck hindering their real-world deployment. Existing MAS debugging and repair methods typically rely on rerunning and resampling the entire execution trajectory. However, a fundamental question remains to be answered: do these methods causally repair MAS failures or merely stochastically repair by leveraging the randomness of LLM sampling? To evaluate the effectiveness of MAS repair methods, we introduce SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors. During replay, it effectively reconstructs the execution before the anchor using recorded logs and only regenerates the downstream trajectory, thereby enabling the reliable reproduction of MAS failures. We further construct the dataset SymFail, comprising 536 human-annotated failure trajectories with graph-linked locations, categories, and trace evidence. Based on these foundations, we conduct a large-scale empirical study across three mainstream MAS frameworks. Our findings reveal that existing unguided rerun methods are highly unreliable, exhibiting low failure reproduction and repair rates (only 67.97% and 6.90%, respectively). Building upon these findings, we further explore the effectiveness of a symptom-driven intervention method, which successfully repairs 20.15% of the failed cases (a 191.89% improvement to state-of-the-art repair methods). This study aims to provide actionable insights for MAS debugging and repair research, paving the way for the robust deployment of multi-agent systems.

0.41

Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them

cs.AI ·2026-11-09 ·Shaohuang Wang, Jinghao Wen, Xulei Zhang, Xun Jiao
similarity 0.62
Abstract

When an LLM agent fails part-way through a long task, the usual analysis records a symptom: a failure-mode label, or the agent and step judged responsible. It does not say where the error entered, which state carried it to the symptom, whether anything in the trajectory contradicted it, or which control could have caught it. We define an episode record that ties a candidate fault to its origin, propagation, detection, response and consequence, located by operational phase and state domain, and that names the control families whose observation and reach cover it. We turned the record into a written annotation protocol and applied it to 95 failed runs of six multi-agent systems from the public Who&When and MAD corpora, reading each trajectory in full. Faults enter early (median at 0.18 of the run), mostly as wrong decisions or fabrications over task state, and in 37 of 88 Who&When runs before the step the corpus' human labels mark as decisive. In 72% of failed runs the trajectory already contained evidence against the error before the failure; in 88% of those no agent acted on it. Something flagged a deviation in 31% of runs, but no response removed the deviated state. Independent judges reading the unused evidence, task contracts fixed at admission, and re-checking beliefs against the environment reach most episodes; restoring the agent's own state would have sufficed in 35 of 95 runs. A text scan of the raw corpus also shows that 45% of the hand-crafted Who&When failures ended with a provider content-filter crash after which the harness emitted a final answer no agent produced, a failure class that neither symptom taxonomies nor responsible-agent labels represent. Protocol, records and scripts are included.

0.11