CS Paper Radar

The Weight of Teamwork: Does Communication Structure Help LLM Scientists Search for Scientific Laws?

Jianing Shi, Zaibin Zhang, Shijun Lei, Zhenfei Yin

cs.MA cs.AIcs.LG ·submitted 2026-10-16 19:02 UTC · v1
publishedunusual

Abstract

Team performance is often reported as one delivered score, even though that score is produced in two stages: a system generates candidate answers and a selection rule returns one of them. We use symbolic regression as an observable scientific-search setting and compare fixed-budget LLM teams that differ in peer visibility and update timing, evaluating both their frozen candidate archives and the expressions selected from those archives. Distributing the call budget across more independent trajectories produced the largest measured gain, improving mean held-out fitting score Q by 0.70 at the widest tested allocation. Under a common archive-wide selector, the implemented Ring policy reduced mean Q by 0.33 relative to Independent, while Star and Complete remained unresolved. Holding the generated archives fixed, changing the delivery rule changed the measured architecture gaps, including the sign of the Complete point estimate. These results support reporting candidate generation and final selection as separate endpoints. They do not identify a communication mechanism or a universally preferred topology; rather, they show that an architecture ranking is not identified by communication structure alone when the rule that returns the final answer is left unspecified.

Comment
Accepted at the NeurIPS 2026 Workshop on Foundations of Agentic Systems Theory (FAST), non-archival. 7 pages main text, 33 pages total with appendices

Closest papers

Nearest neighbours by meaning across the whole index.

Sci$Π$vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

cs.AI ·2026-10-15 ·Youran Sun, Xingyu Ren, Chugang Yi, Kejia Zhang, Di Wu +13 more
benchmark similarity 0.56
Abstract

Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct Sci$Π$vot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while Sci$Π$vot supports systematic evaluation of scientific judgment for LLM agents.

0.56

The Between-Round Decision in Autonomous Research: An LLM Controller That Reads the Whole Trajectory

cs.AI cs.CL ·2026-11-17 ·Caoxin Wang, Shengkai Gan, Xiang Hu, Lili Pan
benchmarkcode/data similarity 0.56
Abstract · 16 pages, 2 figures

Executing an idea often changes how promising it looks, so the next step in autonomous research should rest on results, not on proposals. After each round of experiments, the system must decide whether to refine the best solution or start a new direction, which solution to build from, and what goal to set. We call this the between-round decision. Existing systems wrap it in machinery such as rules, scores, search trees, and schedules; where an LLM makes it, what the LLM needs to read is left open. We ask whether an LLM can make this decision from accumulated evidence, without a search structure around it, and how much of that evidence it needs to read. We present BRED (Between-Round Evidence-based Decision), an LLM-based controller that maintains the best verified solution so far (the Champion) and a log of past rounds, and makes one decision per round. On three widely used research-agent benchmark tasks, BRED improves on the task baselines by 4.80% in MAE on ETTh1 (to 0.4172) and by 3.44 and 3.43 points of accuracy on SST-2 (to 95.07%) and CIFAR-100 (to 84.63%), ahead of the published results. When the controller does not read the accumulated evidence, the same loop gains less: with no history it reaches 0.4299 on ETTh1 and 83.71% on CIFAR-100, whereas the full controller's MAE is 2.95% lower and its accuracy 0.92 points higher. On a new task, speech-based depression detection under a leakage-free protocol, it reaches a macro-F1 of 0.6164 on seeds held out from selection, 5.5 points above its baseline and 4.0 above the strongest published system. Code is available at https://github.com/learninginvision/BRED.

0.54

Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries

cs.AI cs.LG ·2026-10-05 ·Chonghe Jiang, Ao Qu, Siyuan Liu, Ruoyun Ma, Zijian Zhou +7 more
code/data similarity 0.55
Abstract · 35 pages, including references and appendice

Open-ended scientific discovery often requires repeatedly proposing and evaluating candidate solutions. LLM-based systems can support this process by generating and refining executable solutions from verifier feedback. Methods such as TTT-Discover use test-time training (TTT) to update the solution-generating LLM from verifier feedback, adapting its generation policy to improve subsequent proposals on the target problem. However, this becomes expensive when reliable execution requires a large model, since training must maintain gradients, optimizer states, and policy statistics while repeatedly generating long, structured outputs. It also complicates credit assignment: outcome-level verifier feedback must jointly evaluate the high-level strategy and its low-level implementation. In this work, we introduce Guidance-TTT, which separates these roles. A compact guidance model is trained at test time to propose high-level strategic changes, while a frozen execution model implements them as complete executable solutions. At each step, the system selects a promising previously discovered solution, proposes a change, executes and verifies it, and updates only the guidance model using an adaptive group-relative RL objective. This concentrates test-time learning on short strategic decisions while retaining the implementation capability of a substantially stronger model without adapting it. Without web access, Guidance-TTT produces strong solutions across four distinct domains: combinatorial optimization (Polyomino Packing), heuristic programming (AHC058), machine learning (Lasso), and GPU kernel optimization (TriMul). Across these tasks, it outperforms the best solutions reported in prior work while remaining competitive with state-of-the-art results on public online leaderboards. Code is available at https://github.com/Human-Agent-Society/reef/tree/guidance-ttt-support.

0.43

Large Teams Consistently Develop Science

cs.DL cs.SI ·2026-10-20 ·Yiling Lin, Lulin Yang, Lingfei Wu
similarity 0.55
Abstract

Scientific progress depends on both developing established knowledge and disrupting it to make way for new ideas. Using the disruption index (D), our previous work showed that large teams tend to develop science, while small teams are more likely to disrupt it. Recent work challenges this finding, arguing that D mechanically depends on reference-list length. Using an alternative measure (oneD), it reaches the opposite conclusion: large teams are more disruptive. Combining Nobel Prize case studies, mathematical analysis, and large-scale publication data, we show that oneD misclassifies developmental contributions as disruptive and introduces the very sensitivity to reference-list length that it seeks to eliminate. The original measure, D, however, shows no evidence of the claimed systematic dependence on reference-list length. More fundamentally, even after controlling for reference-list length, larger teams remain more likely to produce strongly developmental work. Our findings reaffirm that large teams consistently develop science and offer new insights into the measurement of scientific disruption.

0.21

LLM-as-a-Judge Version Shift: The Instrument Changed, Not the Standard

cs.CY cs.CL ·2026-09-07 ·Guijia Zhang, Harry Yang
similarity 0.55
Abstract · 25 pages, 5 figures, 14 tables

Automated research pipelines increasingly rely on fixed LLM judges to rank, filter, and revise proposals, yet these judges are routinely replaced by newer versions. We study whether such replacements preserve the underlying measurement standard. Using a preregistered, frozen-weight design, we evaluate three adjacent same-family transitions across Qwen, Gemma, and Mistral on frozen ML research proposals under randomized information conditions. The common assumption that newer judges are simply harsher fails: two transitions shift toward greater leniency, while the third shifts in the opposite direction on anchors but attenuates off-anchor. More importantly, version changes can alter not only score location but also criterion geometry and resolution; one transition collapses clean-flawed discrimination, placing 93% of pool scores on a single value. Supplying identical facts does not close the version gap, placebo-controlled interventions show that observed packet effects are largely format-driven rather than content-driven, and global recalibration repairs the mean while explaining little object-level dispersion. We organize these failures with a measurement model in which a judge replacement can change its origin, criterion geometry, or resolution. A small stratified anchor audit of about 200 calls identifies the failure class before deployment, and frozen thresholds correctly diagnose two post-registration API judge replacements. We release six preregistration locks and the full audit trail of 14882 judge calls.

0.23

Scaling Discovery through Test-Time Communication

cs.LG cs.AIcs.CL ·2026-09-17 ·Jongho Park, Vasilis Kontonis, Shivam Garg, Akshay Krishnamurthy, Dimitris Papailiopoulos
benchmarkhighly cited similarity 0.55
Abstract · 34 pages, 12 figures

Science advances not in isolation but through collaboration, yet existing agentic systems capture little of this. Whether communicating agents help remains an open question with mixed prior results. We show that test-time communication can substantially outperform independent parallel attempts on challenging tasks, where sharing a breakthrough can push the whole group forward. We first study the effect of scaling multi-agent test-time communication, where agents have no predefined roles and communicate via a shared directory, on ARC-AGI-3, a benchmark requiring novel problem solving. We find that a team of $k$ communicating agents, team@$k$, matches the success rate of $4k$ independent agents, and this advantage grows with $k$, suggesting gains compound with scale. The effect is not merely efficiency: a task that no single agent can solve, a team of agents can solve reliably. Furthermore, these gains transfer to research-oriented tasks, given sufficient compute. On polyomino packing, communicating agents outperform best@$k$ and exceed the prior best-known score. On MNIST classifier compression, communication surpasses the best-known human solution. A team of four agents produced a 1,957-byte classifier submission achieving 99.4% test accuracy, smaller than both the best-known human solution and the best single-agent result. These gains are not unconditional. Independent agents may outperform communication when compute is limited or when a clear measure of progress is absent. However, under sufficient compute and clear feedback, multi-agent communication consistently yields stronger results.

0.60

Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

cs.LG ·2026-10-26 ·Qiaomu Li, Ying Xie
similarity 0.54
Abstract

Retrieval-augmented generation (RAG) systems often use a dedicated reranker to choose which retrieved candidates reach the language model. Reranking improves relevance but adds latency, so teams look for faster alternatives. System-1style decision models make direct, structured predictions instead of generating long responses. Could one provide comparable reranking quality with less inference overhead? We first screen 21 open-weight models on 11 benchmarks, then compare a promising decision model with a dedicated reranker on identical candidate lists. The default decision-model setup fails on many long candidate sets, so scoring only the requests it completes gives a misleading ranking. Limiting candidate text length largely restores reliability and makes some configurations competitive. A frozen evaluation on new retrieval data supports the reliability and ranking results, but confirms the latency requirement for only one of the two selected configurations. Across workloads, which model ranks better changes, and a tuned reranker remains faster and uses far less GPU memory. A decision model can perform competitive reranking, but replacing a reranker requires weighing quality, latency and memory together, and teams should tune the reranker they already run first.

0.41

The Impact of Backbone Evolution on LLM-Based Relevance Assessments

cs.IR ·2026-10-07 ·Chuting Yu, Guido Zuccon, Teerapong Leelanupab
similarity 0.54
Abstract · 12 pages main content

LLMs are evolving rapidly, with newer models offering stronger capabilities. This suggests that in LLM-based relevance judging, more capable models will achieve higher agreement with human judgements under the same prompt. We challenge this understanding by investigating the behavior of LLM-based relevance judges under backbone evolution. Keeping the prompts fixed, we evaluate a representative single-prompt (UMBRELA) and a rubric-based prompt (EXAM) across sequential model versions of commercial (Gemini, GPT) and open-weight (Qwen, Llama) models. Overall, we find no consistent evidence that newer versions lead to better relevance judges. Crucially, similar or improved aggregate performance does not imply judgment stability: correct judgements made by an earlier version of an LLM backbone are not necessarily preserved by later versions. We investigate the potential drivers of these regressions. Our findings caution against the assumption that judging prompts designed and validated for one backbone version will perform equivalently or better when the model is updated, even within the same family.

0.23