CS Paper Radar

Credit Where Credit Is Due: Contrastive Attribution Topology Search for Multi-Agent Systems

Fabian Tritthart, Tom Zehle

cs.CL cs.MA ·submitted 2026-10-21 06:46 UTC · v1

Abstract

An LLM multi-agent system (MAS) is only as good as the agents it chooses to trust: its accuracy and cost hinge on its topology, meaning which agents are included and how they communicate. A single task score, however, does not reveal which agents help or hinder performance. We propose Contrastive Attribution Topology Search (CATS), which searches for compact teams within a fixed supergraph of specialized agents. An LLM attributer compares execution traces across candidate topologies and assigns credit to individual agents. CATS uses these credits to update supergraph edges while progressively pruning weak edges and agents, yielding a small team for deployment. With small open-weight models and matched candidate evaluations, CATS improves mean accuracy on BigCodeBench by 5.3 percentage points over score-only heuristic search while using 54$\%$ fewer deployment tokens. On MBPP, CATS matches the strongest baselines with fewer tokens than the other searches. In a HotpotQA prompt-poisoning stress test, CATS excludes all adversarially prompted agents in all three runs, whereas random search retains some at comparable accuracy. Per-agent credit thus supports both team optimization and the pruning of potentially harmful agents.

Closest papers

Nearest neighbours by meaning across the whole index.

Agent Benchmarks Stop Ranking Before They Run Out of Tasks

cs.AI ·2026-11-10 ·Zihan Dong, Qishi Zhan, Minxuan Hu
similarity 0.68
Abstract

Data-science and ML-engineering agents are evaluated on benchmarks such as MLE-bench, ALE-Bench, SWE-fficiency and AlgoTune, whose tasks yield a quantity (a Kaggle metric, contest score or speedup). When one stops separating the best models, the usual remedy is a harder version. Re-scoring released runs of eight benchmarks, including SWE-bench Verified and Terminal-Bench for breadth, we find much of this saturation lies in the measurement, not the tasks. Most agent benchmarks reduce a run to one pass/fail bit, which fails in two ways. It is too coarse: neighbouring top-ten agents on SWE-bench Verified differ by less than its tests' errors, and correcting them reorders neighbours no more than deleting a random sixth of each agent's passes. It is capped at the reference: ALE-Bench's top ten, separated by contest score, tie when graded against the median human, although nearly half the runs beat that human. On fixed tasks, a higher bar only moves the problem; graded credit fixes it. We derive a clipped log-ratio score with dated anchors and cost as a separate axis. In a Terminal-Bench re-run, pass/fail-tied models differ over tenfold on the quantity the tests discard.

0.11

Learning How Much to Collaborate: Difficulty-Aware Topology Selection for Multi-Agent Code Generation

cs.MA cs.SE ·2026-09-12 ·Yunsong Hong
publishedtheory similarity 0.66
Abstract · Accepted for publication in Neurocomputing. Revised manuscript incorporating peer-review revisions. 23 pages, 11 figures

Multi-agent code-generation systems often use one communication topology for every problem. Across five topologies and 614 problems from APPS, HumanEval+ and LiveCodeBench, the advantage of hierarchical collaboration over a single agent grows from 2.4 points of pass@1 on the easiest third to 21.1 points on the hardest third, at about ten times the token cost. We propose the Difficulty-Aware Topology Selector (DATS), which predicts each topology's success probability and maximises predicted success minus cost. Its graph predictor treats topologies as nodes of a connectivity order. The complete graph-head comparison reports a 1.7-point advantage over a flat multi-label head while changing several mechanisms; capacity-matched adjacency controls show smaller descriptive differences. A scalar cost penalty can be recalibrated without retraining for approximately matched-spend comparisons when targets are attained. With a shared penalty, six methods span 21.6 percentage points of spend; two leading baselines fall behind DATS after method-specific calibration near the common target. At 40% of always-hierarchical cost, DATS reaches 77.7% pass@1 versus 73.6% for always-hierarchical and 74.3% for the strongest learned competitor. The main comparison includes Holm--Bonferroni-adjusted McNemar tests; unequal-cost differences do not imply matched-budget superiority. Backbone-specific gains are 4.07--4.23 points across four backbones. Alternatives to the 39 interpretable features, including a hybrid encoder, yield a largest observed gain of 1.3 points, with no significant pairwise improvement; this establishes neither equivalence nor an upper bound. A study on 400 mathematical reasoning problems reproduces difficulty-dependent collaboration gains, widening from 2.5 to 20.9 points.

0.59

Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality Anchors

cs.SE cs.AI ·2026-11-13 ·Shervin Shahidi, James Poon, Hang Xie, Kenneth O. Xu, Robin Håkanson +3 more
benchmark similarity 0.63
Abstract · 5 pages, prepared for submission to AGENT 2027

Teams deploying coding agents need benchmarks built from their own code, but curating and validating them by hand is resource intensive, and an unvalidated benchmark can mislead silently. We present Benchmark Factory, an autonomous meta-agent loop with human checkpoints: an LLM agent writes sourcing queries over landed production code changes, runs models of known relative strength on each candidate, diagnoses why a candidate fails, and repairs it by editing its queries, prompts, and configuration until it clears quality anchors that need no per-task human labels. We used Benchmark Factory to build benchmarks for five developer cohorts at our organization. Engineering raters scored a 5% sample 2.3 for task quality and 2.6 for grader verdicts on a 1-to-3 scale. On configurations never used to build them, the benchmarks ranked Claude Opus~4.7 at high reasoning effort above low effort (60.6% vs. 54.4% of tasks passed, $p=0.002$) and GPT-5.5 above GPT-5.4 (65.8% vs. 60.5%, $p=0.004$); 9 of the 10 per-cohort comparisons pointed the same way. A deliberately trivial benchmark failed the anchors; five broken graders failed them, even under a relaxed rule fixed in advance that the real grader passes; and with model labels shuffled so that no true ordering existed, random redraws of a candidate passed in at most 1.4% of simulations. The anchors are necessary, not sufficient, conditions for a valid benchmark, designed to focus human review on what the loop cannot repair.

0.12

Not Every Agent Needs the Strongest Model: Unified Prompt-Model Optimization for Heterogeneous LLM-based Multi-Agent Systems

cs.MA ·2026-10-22 ·Mingxiu Cai, Haoyang Xian, Xinyu Ma, Yutong Wang, Zhenxi Song +2 more
code/data similarity 0.63
Abstract

Large language model (LLM)-based multi-agent systems (MAS) rely on role-specific prompts and execution models to solve complex tasks collaboratively. However, prompt optimization typically keeps model assignments fixed, leaving differences in role requirements and their associated inference costs outside the search. We present UniMASO, a Unified framework for heterogeneous Multi-Agent System Optimization through the joint adaptation of prompts and model assignments. UniMASO evaluates candidate updates through local, downstream, and global feedback, using cost-aware Pareto search to retain useful quality-cost trade-offs. Topology-aware co-evolution then coordinates these updates according to agent roles and dependencies. Cross-level trajectory memory further supplies historical successes and failures to guide subsequent optimization. Experiments on five benchmarks and five execution topologies show that UniMASO achieves competitive task performance at substantially lower deployment cost than the strongest homogeneous configurations. Under centralized execution, it improves the average score from 69.79 to 70.40 while reducing token-weighted deployment cost by 85.2% relative to DeepSeek-V4-Pro with MASPO. These results support joint prompt-model adaptation as an effective approach to efficient multi-agent collaboration. The code is available at https://github.com/Achilles008/UniMASO_main.

0.31

Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems

cs.AI cs.LGcs.MA ·2026-09-02 ·Jinxi Yu, Yubei Li, Eric Hanchen Jiang, Zhi Zhang, Dong Liu +4 more
position similarity 0.63
Abstract

Adapting the communication topology of an LLM multi-agent system to each query improves both accuracy and efficiency, yet current designers treat this as conditional graph generation: a variational, autoregressive, or diffusion decoder searches the $N \times N$ adjacency space, and a graph-network proxy trained on utility and a structural cost such as edge count ranks the sampled candidates. We argue that this formulation is misaligned with the problem. Empirically, topologies that survive a reward filter collapse to about six distinct graphs even when the codebook capacity grows from 8 to 64; edge count is negatively correlated with measured token consumption (Pearson $r \approx -0.4$), so sparsifying the graph makes inference more expensive; and a message-passing scorer over agent-profile nodes is adjacency-invariant whenever agents share a profile---the default configuration of published benchmarks---so it cannot rank candidates at all in that regime. These three facts motivate Codebook Agent: a vector-quantized autoencoder compresses successful topologies into a query-independent 16-entry codebook; a reward-weighted MLP maps the query embedding to a distribution over codes; and an MLP proxy that reads the flattened adjacency, regressed on measured utility and per-task normalized token cost, reranks the top decoded candidates in a single batched forward pass. With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks we compare (84.6 average against 83.0 for the strongest prior designer), emits a topology in 2.4 ms, and uses 21.9--33.2% fewer LLM tokens.

0.50

From Agents to Edges: Paired Optimization for Multi-Agent Collaboration

cs.CL ·2026-10-13 ·Feng Lin, Xiaotian Yu, Yifan Yang, Aijun Li
similarity 0.61
Abstract · 9 main pages, 14 appendix pages

Strong individual agents do not guarantee effective collaboration. Communication must align what one agent conveys with how another interprets and uses it. This dependency makes the communication edge a natural unit of optimization. We introduce Paired Automated Communication Optimization (PACO), which jointly optimizes the instructions at both ends of each edge and their shared communication convention in systems built on black-box language models. PACO keeps model weights, agent roles, and workflow topology fixed. A state-aware discounted bandit selects edges for refinement, while execution feedback guides paired revisions evaluated through staged end-to-end execution. The optimized instructions integrate into existing agent prompts without additional model calls at deployment. On a four-agent workflow evaluated across diverse multi-agent benchmarks, PACO consistently outperforms leading prompt-optimization and communication baselines. Configurations learned on HotpotQA retain their advantage in cross-dataset transfer without target-specific optimization. These results demonstrate that optimizing how agents communicate can substantially improve collaboration within an otherwise fixed system.

0.23

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

cs.AI ·2026-08-27 ·Mesut Toruk
benchmarkcode/data similarity 0.61
Abstract · 8 pages, 1 Figure, 6 tables

Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments-are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 13 tool-using ReAct agents across 7 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy), totalling 2,057 deterministic, committed test tasks. Every task is verifier-checkable gold answers are computed by running canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed-form lambdas including adversarial security samples paired with deliberately imperfect signature scanners so a score reflects the model's own judgment, not the copying of an oracle. We define a small set of behavioral metrics beyond accuracy-tool-call adherence, URL hallucination and source-match, and per-model token cost and report a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) whose story is in the per-family spread, not the aggregate. The benchmark runs are executed using the provided evaluation scripts. BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and latency telemetry as well as remote run termination. The benchmark, evaluation tools, and platform are publicly released.

0.32

MULTILOOP: Evolving Multi-Agent Harnesses for Complex Information-Seeking Tasks

cs.MA ·2026-11-11 ·Daniela Gottesman, Gal Yona, Ori Ram, Zorik Gekhman, Jonathan Herzig +2 more
published similarity 0.61
Abstract · Accepted to NeurIPS 2026 CL4FMAgents Workshop

The performance of LLM-based agents depends not only on the underlying model but also on the harness: the prompts, tools, and orchestration logic around it. Recent works automate harness optimization with coding agents that analyze execution traces, diagnose failures, and edit the harness code; however, those focused on single-agent systems. Multi-agent harnesses are harder to optimize as fixing them requires changing how agents interact in addition to how each behaves. We study whether automated harness optimization can diagnose and repair failures in multi-agent systems through a case study on complex information-seeking tasks from DeepSearchQA [Gupta et al., 2026]. We apply MULTILOOP, a simple optimization loop in which a coding agent iteratively analyzes, evaluates and edits the harness. With only 10 training samples, the evolved harness improves accuracy by 4.6% over the base harness and by 3.2% over the best single-agent system. However, a taxonomy of failure modes before and after optimization shows that the orchestration failures that characterize multi-agent systems largely persist. Our findings motivate harness optimization methods that encourage structural changes to how agents delegate tasks and share information.

0.20