CS Paper Radar

Building a research-software catalog with a coding agent: from hackathon prototype to public deployment

Kazuyoshi Yoshimi, Satoshi Terasaki, Gotai Yamada

cs.SE cs.AIcs.CYphysics.ed-ph ·submitted 2026-09-04 04:31 UTC · v1

Abstract

Generative AI and coding agents can accelerate research software development, but they also increase the need for efficient software discovery and maintenance. We developed a repository catalog during a three-day hackathon and subsequently examined the engineering required to make it suitable for public deployment, including adversarial review, data-quality checks, browser-level validation, and publication safeguards. We then explored whether the lessons learned from this prototype could be transferred to a much larger, human-curated portal, through a retrieval agent under development for MateriApps that combines curated portal metadata, external documentation, vector search, and local language-model generation. Implementation with coding agents was rapid, but achieving reliable operation required substantial additional engineering: the most consequential problems were not crashes but silent failures that produced plausible yet incomplete or incorrect outputs, arising from incomplete data acquisition, misleading assessments, and retrieval or preprocessing failures. These observations suggest that AI-assisted software portals require explicit validation, monitoring, and repeated review, and that curated metadata and maintained documentation remain essential. The MateriApps work is exploratory and remains under active development, so the observations reported for it are preliminary; a comparable combination of curated metadata, automatically collected documentation, and retrieval-based assistance may nevertheless be useful for extending other research-software portals.

Comment
23 pages, 6 figures, 2 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Spotlights: Discovering Improvement Opportunities in Software Repositories

cs.SE ·2026-09-17 ·Udi Barzelay, Ophir Azulai, Idan Friedman, Inbar Shapira, Foad Abo Dahood +7 more
similarity 0.73
Abstract

Coding agents and evolutionary code-search systems can improve implementations once a target and evaluation criterion have been specified. Applying these methods to an existing software repository raises an earlier question: which implementation choices are worth investigating for a high-level engineering objective? We introduce \emph{optimization-opportunity discovery}, the repository-level task of identifying candidate source regions, explaining how they relate to the objective, and proposing possible changes. The task takes as input a repository, an engineering objective, and optional runtime evidence such as offline telemetry observations or profiles. It does not require the user to specify a defect, bottleneck, or code location. We present \emph{Spotlights}, a system that performs this task through logical repository mapping, successive agent reviews, and optional research linking candidates to relevant techniques. We evaluate Spotlights across model serving, document retrieval, blockchain ordering, and document processing. Across three cases, it recovers seven of nine expert-selected targets. In the reliability study, 70\% of the top ten candidates meet the stated correctness and severity thresholds. Across five repeated retrieval runs, 73.6\% of candidate occurrences have a matching source region in all five runs. Spotlights also rediscovers the target of a withheld retrieval optimization and connects it to a relevant tiling technique. In an implementation study, a discovered change reduces end-to-end page-processing runtime by 10.6\% while preserving measured output quality. These results establish optimization-opportunity discovery as a distinct and empirically evaluable step between a broad engineering objective and subsequent implementation and validation.

0.08

AI-Research Agents in the Wild. From GitHub and arXiv to Regularities and Gaps

cs.DL ·2026-09-01 ·Aleksey Komissarov, Andrey Ustyuzhanin
similarity 0.72
Abstract · 56 pages, 1 figure

AI-research agents, or autoresearch systems, combine language models with tools, search, evaluation, and iterative modification of research artifacts. Their public software ecology is hard to compare because repositories, papers, benchmarks, libraries, and companion artifacts are often counted as one population. We connect two agent-drafted, human-adjudicated registries frozen on 10 June 2026: 139 canonical public repositories and 101 papers on AI-assisted research systems, each record carrying an evidence card anchored to its primary sources. Nine promoted design lineages contain 59 memberships among 52 repositories, seven of them assigned to two lineages. Against this evidence we examine six time-stamped candidate design regularities. A prospective audit of 25 newly ingested repositories observed zero of six specified trigger events, supplying bounded support for R1-R4; broader corpus evidence narrowed the original scope of R5, and a failed directional prediction leaves R6 exploratory. The paper-repository graph contains 212 curator-assigned relatedness edges: 64 of 101 papers carry at least one edge, the edges reach 23 of 139 repositories, and the six most-linked repositories carry 120 of the 212 edges (56.6%). A full-text check of seven papers shows the relation measures our curation, not the papers' citation behavior. An identity audit adjudicated against commit histories and public contributor records confirms at least 18 paper authors who also author a canonical registry repository (20 of 63 candidate author slots in 13 of 101 papers); a byline-matched tier reaches 58 identities but is reported separately and more weakly. The study contributes a source-grounded theory under prospective test, a curator-assigned map of paper-repository relatedness, and explicit limits on claims about public visibility and community structure.

0.18

AgentTorrent: A Dataset of Context and Harness Engineering Artifacts on GitHub

cs.SE ·2026-11-11 ·Sebastian Baltes, Seyedmoein Mohsenimofidi, Levi Böhme, Vahram Nersesyan, Muhammad Auwal Abubakar +3 more
code/data similarity 0.70
Abstract · 5 pages, 2 figures, 2 tables, under review

Over the past months, developers have moved from formulating detailed prompts to engineering the context that agentic coding tools read and to configuring the underlying harnesses. For example, developers commit context files, skills, hooks, and MCP configurations to their repositories, and these files control which instructions a harness reads and which actions it can take. Developers may also commit the specs, plans, and task files that agentic coding tools and spec-driven development frameworks write for agents to implement. The AgentTorrent dataset records these files for six proprietary and open-source agentic coding tools in 10,944 engineered software projects on GitHub. It contains 62,374 configuration artifacts, the files that these artifacts import, 27,593 agentic planning documents, and more than 1.2 million commits that credit an AI tool. All heuristics used for data collection are grounded in tool documentation or derived from GitHub repositories, and we evaluate their detection accuracy with each collection. We use open-weight models to classify whether GitHub repositories host engineered software projects. Unlike most datasets, AgentTorrent links configuration artifacts, imported files, agentic planning documents, and commits that credit an AI tool within the same repositories. Configuration artifacts, planning documents, and credited commits occur together in 388 repositories. For example, our dataset allows researchers to study whether a tool is credited in commits that add instructions for that same tool and how instruction files import and contradict each other. Our data collection pipeline supports regular updates and automatically checks vendor documentation and reports unknown artifacts and agentic coding tools. The dataset and the pipeline are open-source and are archived on Zenodo.

0.36

When Relevant Code Is Wrong: Runtime-Aware Context for AI Coding Agents

cs.SE ·2026-10-28 ·Wesley K. G. Assunção, Ramesh Periyathambi
similarity 0.70
Abstract

AI coding agents increasingly rely on retrieved code and development history to guide software changes. However, highly relevant artifacts may describe implementations that are inactive or incompatible with the target deployment. Neither semantic similarity nor recency establishes whether retrieved information remains applicable under feature flags, architectural migrations, and deployment-specific configurations. We challenge relevance-oriented retrieval by introducing operational context validity, a task-, deployment-, and time-dependent criterion for determining whether software evidence can legitimately guide implementation decisions. We instantiate this new retrieval principle in CASE, an agent-agnostic framework that connects repository artifacts with runtime configurations and architectural evolution, classifies evidence as valid, invalid, or unknown, and generates context packages distinguishing actionable guidance from historical rationale and unresolved constraints. We investigate its feasibility through a reproducible preliminary evaluation involving six development tasks, 15 artifacts, and three deployments. Conventional retrieval exposes 16 incompatible artifact occurrences as actionable, whereas CASE excludes them while retaining all applicable artifacts under the prototype's reference rules. Recency filtering reduces valid recall to 0.43, while removing feature-flag evidence from CASE reduces it to 0.13. Although these results demonstrate feasibility rather than improvements in coding-agent correctness, they reveal limitations of relevance-based retrieval and motivate a new research direction: enabling coding agents to reason not only about which information is relevant, but also whether it remains operationally valid.

0.09

What Open-Source Projects Tell Their Coding Agents: A Census of AGENTS.md and CLAUDE.md

cs.SE cs.AI ·2026-10-22 ·Pawel Zarzycki
similarity 0.69
Abstract · Corrected abstract metadata formatting and restored missing numerical values; paper unchanged

Coding agents can be guided with instruction files in a repository. We examine this emerging practice through a census of 22,854 popular, active GitHub repositories in thirteen languages, treating each repository's root 'AGENTS$.$md' and 'CLAUDE$.$md' as one instruction set. Agent instructions appear in 26.3% of repositories and 59.6% of the most-starred; where both files exist, 'CLAUDE$.$md' usually points to 'AGENTS$.$md'. We analyse 341,852 statements using model-assisted annotation and human review. The annotations reveal a practical mix of project knowledge and rules about code, building and workflow. Collaboration rules name concrete requirements, from test plans to commit formats, yet 74.0% of directives give no reason. Projects disagree on AI attribution: commit-message rules more often prohibit it than require it, while pull-request rules more often require disclosure. A third of instruction sets have a generator-associated signature, a tool-inserted block, shared text or an AI-marked creation commit; four in five are edited after creation. Prespecified tests associate repository popularity with rules on pull requests and AI contributions. Exploratory analyses link activity to process-oriented content and show that LLM and agent projects are over-represented among adopters. The models agree much more closely with each other than with the human reviewer's topic assignments, a caution for automated annotation. We release the annotations, taxonomies and pointers to the analysed file versions.

0.42

A Study of Cursorrules Files in GitHub Open Source Projects

cs.SE ·2026-08-11 ·Shuang Sun, Jafar Akhoundali, Arina Kudriavtseva, Sengim Karayalcin, Olga Gadyatskaya
publishedhighly cited similarity 0.69
Abstract · Published in ICSOFT 2026. This is the author copy version

Prompts are the primary mechanism for communicating with AI agents, and they directly influence the quality and reliability of AI-generated code. As AI-assisted programming becomes widely adopted, modern tools increasingly combine dynamic conversational prompts with static configuration-like prompt files. Despite the growing focus on prompt engineering, prior research has primarily focused on conversational prompts, while prompt files remain understudied. To address this gap, we conduct an empirical study of configuration prompt files in Cursor, a widely used AI-assisted code editor. We collect and analyze over 12,110 .cursorrules files from 11,427 GitHub repositories to characterize their distribution, evolution, and maintenance. Complementing this, we perform qualitative analysis on a random sample of 65 prompt files and develop a 65-code codebook capturing how developers express programming intent, project context, engineering practices, and security considerations. Our results show that .cursorrules files emerged rapidly from mid-2024. Their adoption is concentrated in small-scale, low-activity, single-maintainer repositories, suggesting toy projects rather than professional development. The content of prompt files is dominated by guidance on code quality and engineering practices, project structure and configuration, and maintainability, while security-related content appears less frequently. Our analysis shows that there is a continuity of themes and topics between the now-legacy .cursorrules files and the current standard .mdc files.

0.57

Methodological Harness in Agentic Software Engineering: An Empirical Study on Mining Software Repositories

cs.SE ·2026-09-25 ·Jessica Díaz, Jorge Pérez, Sergio Gil-Borrás
similarity 0.68
Abstract

Context. Agentic software engineering requires mechanisms to coordinate and govern agents' work. The framework motivating this study proposes a methodological harness with eight mechanisms: context engineering, persistent shared knowledge, executable specifications, N-version mindset and parallel agents, normative specifications, structured consultation, evidence-based acceptance, and graduated autonomy. These are realized through artifacts such as rule or context files, specifications, and architectural decision records, but the framework had not been empirically validated at repository scale. Objective. We analyze to what extent and how this harness is observable in repositories with agentic activity. RQ1 characterizes adoption through artifact prevalence, breadth, co-occurrence, and temporal evolution; RQ2 examines rule files - a key observable artifact and persistent source of agent instructions - to assess how their content reflects the proposed mechanisms. Method. From the AIDev dataset of 116,211 GitHub repositories, we analyzed 5,435 using a design stratified by visibility, measured through stars as an indicator of popularity and/or reputation. For RQ1, we detected and quantified artifacts and introduction dates for the seven observable mechanisms. For RQ2, we qualitatively coded rule files from 150 repositories. Results. Population prevalence of at least one mechanism is 21.7%, versus 65.3% among the most visible repositories. Multi-mechanism configurations are rare. Where rule files exist, they almost always guide the agent and state norms, while persistent shared knowledge, executable specifications, structured consultation, and graduated autonomy appear only in a minority. Conclusions. The harness is empirically observable, but mainly through isolated mechanisms rather than the integrated system proposed by the framework.

0.43

OpenSaC: A Python Library for Search as Code in Deep Research Agents

cs.AI cs.IR ·2026-10-19 ·Qi Liu, Jiaxin Mao, Fengbin Zhu, Tat-Seng Chua, Ji-Rong Wen
code/data similarity 0.68
Abstract · Source code: https://github.com/liuqi6777/OpenSaC

Deep research agents investigate complex questions through iterative search, source examination, and evidence synthesis. A conventional workflow alternates reasoning with individual search and reading actions. Search as Code introduces a new paradigm: agents write programs that compose search, source reading, reranking, model calls, and local processing, returning selected observations for further reasoning. To support this workflow in existing coding agents, we present OpenSaC, an open-source Python library with a companion Search as Code skill. Configurable capability interfaces let agents combine these operations while retaining evidence for inspection and reuse. Compared with standard Codex using its built-in web search, the OpenSaC configuration uses less execution time and fewer model tokens on BrowseComp and WideSearch, with higher accuracy on BrowseComp and slightly lower row F1 on WideSearch. We aim to lower the barrier to adopting Search as Code and provide an open foundation for community development and transparent research on deep research agents.

0.36