CS Paper Radar

We Ran a Research Sprint With AI. The AI Worked. The Research Didn't

Anush Sankaran, Arijit Banerjee

cs.AI cs.LG ·submitted 2026-11-02 20:46 UTC · v1
position

Abstract

Generative AI tools for research are widely deployed, yet individual productivity gains do not guarantee better team research. We argue that how AI use is structured matters alongside access. Personal AI tools may amplify two failures under deadline pressure: a consumption bottleneck, in which generation outpaces team-visible evaluation, and AI-induced research silos, in which private human-AI loops displace shared decision-making. We motivate this position with a mechanism-generating pilot: a 100-minute research sprint with twelve researchers in three independently operating teams, within a seventeen-person session. Both patterns recurred across teams and evidence sources. Of 33 logged AI interactions, one was accepted outright and 27 lacked a recorded disposition; missing records alone do not establish absent evaluation. A parallel autonomous pipeline produced a complete draft without in-session human evaluation, illustrating production capacity rather than comparative quality. We propose Human-AI-Human collaboration: an interaction topology in which AI connects researchers' contexts and supports inspectable collective evaluation and decision-making. The pilot does not isolate AI-specific effects from time pressure or workflow practices. We close with seven modular, falsifiable design hypotheses, prioritizing shared context and team-visible evaluation checkpoints for controlled follow-up.

Closest papers

Nearest neighbours by meaning across the whole index.

Atria Dawn: The Dawn of Agentic Superintelligence

cs.AI ·2026-09-14 ·Honglin Guo, Tao Gui, Kun Cai, Haodong Chen, Yicheng Chen +138 more
code/data similarity 0.69
Abstract · 23 pages, 10 figures, https://github.com/atria-asi/Atria-Dawn-Preview

As AI agents become participants in the development of their successors, they reshape both the production of intelligence and the role of human researchers. We introduce Atria Dawn Preview, a foundation agentic language model designed for scientific research and engineering workflows, with the goal of expanding the frontier of agent productivity in the real world. This model is trained via a Verifiable Experience Pipeline that connects tool-mediated interactions to executable environments and externally verified outcomes. Across 16 benchmarks spanning real-world research, engineering, and digital work, Atria Dawn Preview is competitive with frontier agents and achieves the highest reported score on five of them. Beyond standalone performance, we examine the real research-and-development process behind this model as a case study of human--AI collaboration, analyzing 769 task records from 56 participants together with agent logs. When asked to evaluate completed tasks under comparable conditions, participants rated about one-third of completed AI-assisted tasks as infeasible without AI. More strikingly, agents frequently propose methods and implement revisions, while humans retain most final decisions and guide exploration through judgment and feedback. These observations indicate a shift from task-level execution to project-level partnership, with human effort concentrating on what is worth pursuing and how evidence should guide research. Progress toward more autonomous AI research must therefore advance both the capacity for discovery and the capacity for meaningful human oversight, preserving accountable human authority over the risks and direction of continued development.

0.31

Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

cs.AI cs.CLcs.HCcs.LG ·2026-10-15 ·Shirley Anugrah Hayati, Patricia Anugrah Setiani, Ruizi Wang, Zhouyi Li, Khanh Chi Le +2 more
benchmarkdataset similarity 0.69
Abstract

When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place greater importance on ongoing interaction with AI agents (e.g., revising plans) than on an agent's ability to complete the task in a single attempt. However, most agentic benchmarks have focused more on automation. Strong performance on such benchmarks does not imply effective real-world collaboration. Professional workflows offer a useful setting, as they often involve interdependent steps. To address this gap, we introduce Kolab, a new dataset of 115+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long-horizon problems. Kolab also contains participants' collaboration and output quality judgments. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback. Our benchmark operationalizes both: (1) forecasting how professionals use agent contributions at diverse workflows, and (2) verifying their judgments of collaboration satisfaction and output quality. Findings show that frontier models, both agentic and non-agentic, overestimate collaboration with experts and struggle to distinguish between high- and low-quality collaboration. Longer workflow context does not consistently improve these predictions. As frontier agents cannot yet understand long-horizon collaboration with experts, we position Kolab as a valuable dataset for studying human-agent collaboration grounded in real professional workflows.

0.48

AutoAnalysis: An automated, self-updating meta-analysis of human-AI collaboration

cs.HC cs.AIcs.CYcs.DL ·2026-10-30 ·Michelle Vaccaro, Abdullah Almaatouq, Thomas W. Malone
similarity 0.67
Abstract · Please visit auto-analysis.dev to explore the data and view the most recent results

As generative artificial intelligence (AI) becomes more capable and widely used, understanding when and how humans should work with these systems is becoming increasingly important. However, the empirical evidence in this space offers conflicting recommendations, which leaves individuals and organizations without clear guidance on when to combine human and AI capabilities. Existing meta-analyses can help reconcile these findings, but they require substantial manual effort and cannot readily incorporate new evidence as it emerges. To address this challenge, we introduce AutoAnalysis: an automated, self-updating meta-analysis of when humans and AI work better together, and when they work better alone. AutoAnalysis' most recent run synthesizes 1,955 effect sizes from 420 papers. Across this body of work, we found that, on average, human-AI combinations performed significantly better than humans alone, but they performed significantly worse than the better of humans or AI alone. These findings highlight the important distinction between improving human performance and achieving human-AI synergy and underscore the need to evaluate collaboration against both human and AI performance alone. As new studies emerge, AutoAnalysis will integrate their findings into the meta-analysis, providing cumulative evidence about when humans and AI perform better together and how human-AI performance varies across AI agents, tasks, and processes.

0.13

Beyond Better Ideas: How Human-AI Collaboration Shapes Scientific Ideas and Human Perceptions

cs.HC ·2026-10-22 ·Qian Chen, Saatvik Pradhan, Janice Ahn, Berk Atil, Angkai Li +5 more
similarity 0.67
Abstract · 23 pages, 10 figures, 9 tables

As AI increasingly becomes a collaborator in scientific research, its impact may extend beyond improving ideas to reshaping how they are evaluated, attributed, and perceived by humans. Yet existing research has largely examined these dimensions in isolation. We present a unified empirical study of Human-AI scientific ideation, investigating two complementary perspectives: i) how AI collaboration shapes scientific ideas and their evaluation, and ii) how firsthand collaboration reshapes human perceptions of AI. Through a shared experimental setting involving Human-AI and Human-Human research teams, human and LLM evaluators, collaboration-provenance analysis, and participant surveys, we uncover several counterintuitive findings. First, AI collaboration shows no significant advantage in human-evaluated idea quality, while iterative revision consistently improves quality. Second, AI judges can approximate human scores yet disagree on idea rankings; notably, an AI judge favors AI-assisted proposals without a corresponding advantage in human evaluations. Third, collaboration provenance remains only partially detectable, revealing substantial overlap between Human-AI and Human-Human scientific outputs. Finally, firsthand AI collaboration is associated with greater perceived usefulness but smaller efficiency gains than expected, suggesting a nuanced recalibration of human expectations. Overall, our findings challenge the notion that AI's value in scientific ideation lies primarily in producing better ideas, revealing instead its broader influence on scientific creation, evaluation, attribution, and human perceptions.

0.10

TRACE: Governing AI-Assisted Research with Principles from Computer Science

cs.CY econ.GNcs.HC ·2026-10-16 ·Amirhossein Zohrehvand, Aldona Kapacinskaite
similarity 0.67
Abstract · 30 pages, 7 tables

Artificial intelligence (AI) has made it easier than ever to produce research outputs. Adding an agentic tool to a research team's work potentially extends the team's capabilities but calls for an active approach to organizing the research process. The pressing question, therefore, is how research teams should organize their work with this technology. We contend that the question is not new: software engineering and security research have long studied how teams of people working with machines keep control of projects with many contributors. We distill principles from the practices these literatures report and transpose them to AI-assisted research. The principles form TRACE: Track records, Review against sources, Accept independently, Control transitions, and Evaluate corrections. Together, the five principles give researchers a common basis for remaining in control while integrating AI into their workflows. By bounding units of work and attending to how each is shaped, they can help researchers pace AI-assisted work to human attention rather than to the speed at which AI produces content. We also introduce TRACE-OS, an operating system for AI-assisted research projects. It puts the principles into practice through shared instructions, tools, and records that fit researchers' existing workflows. With TRACE, we aim to join a conversation about the governing principles through which the research community can benefit from AI. We contend that a simple enough and useful standard can contribute to harnessing AI capabilities, allowing teams to delegate parts of their work to AI, while also building trust within teams and the broader research community.

0.25

Too Late to Prune: Generative AI, the Screen, and Team Creative Decisions

cs.AI econ.GN ·2026-10-10 ·Difang Huang, Shumin Yu, Fadong Chen, Weihua Zhou
position similarity 0.64
Abstract

Generative AI makes options cheap and shifts the bottleneck of creative work from generating options to screening them. We argue that AI's value depends on whether its options arrive before or after the team's screen. In our model, early options compete for a place in the portfolio; late options arrive after that competition and are kept unscreened. In a preregistered experiment, we randomly assigned 148 three-person teams to five schedules of access to the same language model during a product-design task with timed search and selection stages. Moving the same 15 minutes of access from search to selection lowered expert-rated creativity by 0.62 standard deviations, although teams used the model at similar rates. Platform logs locate the difference at the screen. Early access enlarged the pool of ideas fivefold, and teams pruned it. Late-access teams reached the screen with the same small pool as teams without AI and then searched again, devoting 29% of their selection-stage queries to generating new options, compared with 11% for teams with access throughout. Late access added breadth, not quality: these teams submitted 73% more proposals than did teams without AI. Language-model judges detected the benefit of access but not the cost of late timing.

0.31

Puzzles to Paradigms: A Survey of Human--Agent Collaboration in Scientific Innovation

cs.HC cs.AI ·2026-08-31 ·Zhenni Liang, Haoxuan Zhang, Haihua Chen, Ting Xiao, Junhua Ding +1 more
top venuesurveycode/data similarity 0.64
Abstract · Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026. 25 pages, 2 figures, 5 tables

Agentic AI systems are increasingly embedded in scientific workflows, including literature synthesis, hypothesis generation, tool use, writing, and peer review. However, existing surveys and evaluations often emphasize task performance while underexamining how humans and agents coordinate under different epistemic conditions of scientific work. This survey introduces CSCW$\times$Kuhn, a coordination-aware framework that connects computer-supported cooperative work dimensions with distinct scientific contexts. Using this framework, we synthesize recent human--agent systems and benchmarks to examine how coordination demands and failure risks may vary across contexts. Our synthesis identifies five recurring failure modes: procedural drift, grounding substitution, evaluative flattening, proxy optimization, and epistemic black-boxing, and argues for evaluations that move beyond task accuracy and toward grounding, provenance, disagreement preservation, and long-horizon scientific reliability. Our corpus and coding artifact are available at https://github.com/jenlia024/CSCWxKuhn-artifact.

0.55

Human-AI Collaboration at Scale: Task Criticality, Agency, and Friction across 250,000 Conversations

cs.HC ·2026-08-17 ·Yijia Shao, Dora Zhao, Vishakh Padmakumar, Jennifer Wang, Diyi Yang
similarity 0.64
Abstract · 17 pages, 6 figures

AI is increasingly embedded in professional work, yet how people actually collaborate with it at scale remains poorly understood. Drawing on a corpus of 249,834 real-world conversations, we conduct privacy-preserving analysis by asking three questions: what work people bring to AI, what roles humans retain in completing that work, and where human-AI collaboration breaks down. We find that users bring to AI not only trivial tasks but also consequential, hard-to-reverse work, with high criticality delegation concentrating in advisory and professional domains. As stakes rise, users engage more actively with AI output, and conversation length nearly doubles from ephemeral to high-stake tasks. We then examine human agency, the degree of control and responsibility users retain in completing the task. Human-led collaboration dominates overall, but users construct their level of agency through how they prompt and iterate. Across different levels of agency lies a common tension: does working with AI build human capability, or substitute for human effort? We examine this tension through AI-supported learning and upskilling, finding that active teaching is common (67% of conversations) but its benefits are unevenly realized. We further show that friction arises in roughly half of all conversations but it is often productive, and users often employ active strategies to recover from friction when it does occur. Together, these results map how humans and AI work together at scale and when that collaboration makes people more capable, and when it doesn't.

0.50