CS Paper Radar

Papers Without Code: Availability of GitHub Repositories Linked in *CL Publications

Selina Meyer, Michael Roth

cs.CL ·submitted 2026-09-28 08:02 UTC · v1
published

Abstract

Source code and data published at computational linguistics (*CL) venues are increasingly being shared via GitHub. While this generally is a favourable development for the accessibility and potential reusability of research artifacts in natural language processing (NLP), the long-term availability of such repositories has not been evaluated. In this squib, we discuss the availability of repositories linked in papers published in the Computational Linguistics (CL) journal as well as at ACL and its co-located events over the past ten years. Contrary to our expectations, we find that GitHub repositories linked in more recent ACL publications are unavailable at similar rates as in older publications, in parts due to an increase in empty and placeholder repositories. Similar trends hold for other *CL venues, but not for platforms other than GitHub.

Comment
Accepted for publication in Computational Linguistics. Author's final version (pre-MIT Press publication)

Closest papers

Nearest neighbours by meaning across the whole index.

A Two-Dimensional Study of the Model Context Protocol: Publication and Adoption

cs.SE ·2026-09-13 ·Natarajan Chidambaram, Mauro Dalle Lucca Tosi, Jordi Cabot
surveycode/data similarity 0.63
Abstract

The Model Context Protocol (MCP), released by Anthropic in November 2024, standardizes how large language model applications connect to external tools and data sources. Despite MCP's rapid growth, no study has jointly characterized its emergence in the research literature, its adoption on GitHub and the relationship between them. We address this gap with a longitudinal, two-dimensional study of 802 MCP-related publications and 33,319 GitHub repositories. We characterize their growth and identify their application domains via BERTopic and usage contexts via a human-validated GPT-5.6-Luna classifier. We find that publications and repositories began rapid growth within a month of each other in 2025, and peaked together in March 2026. Notably, a second growth wave alone accounts for 49.5% of the GitHub repositories. Most publications (57.8%) and nearly all repositories (93.7%) use MCP as an enabling technology rather than for analyzing, evaluating, extending, or securing it directly. Among repositories, the latter category concentrates almost entirely in one general-purpose topic, while applied usage spreads into product-specific niches. Finally, we provide an overview of the MCP landscape and recommendations to researchers and practitioners in this field.

0.40

From Code Archival to Knowledge Graph: Bridging Software Heritage, COAR Notify and Wikidata

cs.DL cs.AIcs.SE ·2026-09-18 ·Camillo Carlo Pellizzari di San Girolamo, Francesco Tosoni
publishedcode/data similarity 0.60
Abstract · 15 pages, 4 figures. Accepted at the 7th Wikidata Workshop (Wikidata 2026), co-located with ISWC 2026. Open-source pipeline and code availab…

Software is a first-class scientific object, yet validated links between source code and the scholarly record remain largely absent from the Linked Open Data (LOD) cloud, isolating archived artefacts from semantic discovery. This paper presents an end-to-end reconciliation pipeline that harvests, validates, and models publication-to-repository pairs from sources where the link between a paper and its source code is explicit and editorially verified: the software-centric journals JOSS, SoftwareX, and IPOL, together with the reproducibility reports of the SIGMOD Availability and Reproducibility Initiative (ARI). This yields a curated corpus of 4,397 $\langle$DOI, repository-URL$\rangle$ pairs. We design two distinct application profiles grounded in Wikidata classes (one for scholarly articles, one for software instances) aligned with the schema.org and CodeMeta vocabularies. This architectural separation enables rule-based reconciliation at two granularities: lightweight, inline publication references or standalone, first-class Wikidata software nodes equipped with SWHIDs, Software Heritage's content-addressed identifiers. A read-only lookup against Wikidata shows that only 82 of the harvested repositories were already modelled there; human-reviewed batches have since created 4{,}182 new software items cross-linked to their articles. We further show that payloads of the emerging COAR Notify protocol, an external effort we do not develop, map natively onto our input format, so the same backend could later serve a live enrichment stream. Our core contribution is a pair of application profiles that turn Wikidata into a connector between the scholarly record and archived source code; we openly release all code, application profiles, and harvested datasets.

0.53

AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research

cs.AI ·2026-09-15 ·Bernie Boscoe, Srinath Saikrishnan, Vikram Seenivasan, Jack Stark, Andrew Lizarraga +4 more
benchmarkcode/data similarity 0.59
Abstract · 14 pages, 1 figure

Scientific research increasingly relies on large, heterogeneous data sources, motivating interest in retrieval-augmented generation (RAG) systems that provide natural language access to scientific knowledge and research workflows. Researchers are exploring the viability of these systems as natural language interfaces for document search and for generating analysis code and pipeline components. At the same time, concerns about data privacy and control over research infrastructure have motivated interest in open-weight models and open-source deployments hosted within research institutions. In astronomy, this development follows a long history of computational infrastructure development, from archival databases and Structured Query Language (SQL)-based systems to large language model (LLM)-assisted research tools. This paper presents a domain-expert evaluation of faithfulness for AquiLLM, an open-weight, offline RAG-LLM platform designed to support scientific research groups in the use and preservation of tacit and formal knowledge. We define faithfulness as the extent to which generated responses remain grounded in retrieved scientific context without unsupported claims or omissions. We report results from an astronomy case study evaluating AquiLLM across retrieval and scientific analysis tasks. AquiLLM performs most reliably on explicit retrieval-oriented questions grounded in the RAG collection, while faithfulness degrades for queries requiring synthesis or ambiguity resolution. These results highlight both the promise and limitations of open-weight RAG-LLM systems for scientific research and demonstrate the importance of domain-expert evaluation beyond standard benchmark leaderboards.

0.52

Beyond Correctness: Attribution of Natural-Language Documentation for Formal Libraries

cs.SE cs.AI ·2026-10-22 ·Sirui Jiang, Dongming Xiang, Shichao Wu, Yuxuan Zhou, Yihong Wu +3 more
theory similarity 0.59
Abstract · 20 pages, 9 figures, 6 tables

Formal languages are widely used in software engineering to specify and verify program properties. Because their mathematical notation and logic are hard to read, developers often rely on natural-language documentation to understand the functions in formal libraries. Yet a documentation statement can be provably true and still hold after the function's behavior changes. We call a statement's dependence on the function's specific behavior attribution, and we give a method to check it. We design and implement DocAuth, a documentation-generation method that uses attribution to decide what to publish and attaches to each published sentence the formal property recovered from it, together with checkable evidence. On 150 functions from the Community Books library of the ACL2 theorem prover, three large language models produce proved but non-attributable properties for about a third of the functions. DocAuth matches or exceeds the attribution rate of a mutation-based filter while delivering 37% more attributable sentences. Compared with a pipeline that checks truth alone, it raises the share of published sentences that a second model confirms as attributable from 82.9% to 94.7%. These results show that provable truth does not guarantee that documentation reflects what a function actually does. Attribution complements truth as a criterion for generating and evaluating documentation for formal libraries.

0.33

INDRA: A New AI Tool for Exploring Tobacco, Fossil Fuel, and Chemical Industry Archives

cs.DL cs.CLcs.CY ·2026-09-10 ·Daniel Akselrad, Robert N. Proctor
similarity 0.59
Abstract · 35 pages, 6 figures, Appendices available at https://indra.stanford.edu/methods/appendices

Five decades of litigation have disgorged hundreds of millions of pages of formerly secret business records from the tobacco industry, along with documents from the makers of drugs, chemicals, food, firearms, and fossil fuels. Yet these archives have been effectively inaccessible to general-purpose large language models (LLMs) because they have never been compiled into an LLM-readable corpus. Chatbots may be familiar with some of the materials contained in such archives but, with no direct access to the documents, they are vulnerable to hallucination and other defects. Here we introduce INDRA, a research platform designed to remedy such failures by embedding the conventions of archival historiography into a system-level protocol governing every output. The platform federates UCSF's Industry Documents Library, Columbia and CUNY's ToxicDocs, Stanford's SRITA, and other heretofore siloed collections, and provides three interlinked safeguards: (1) a closed evidentiary sandbox confines the model to a user-selected corpus, blocking retrieval from external sources that could introduce bias; (2) real-time provenance tagging marks the boundary between archival evidence and parametric inference; and (3) a system-level protocol enforced by deterministic scripts guides the structure of every output. Together these safeguards prevent the model from conflating "the documents say X" with "I think X" or "I learned X from prior training." The result is an LLM-powered research partner enabling massive multi-archival investigations, a tool whose outputs are designed to be checked rather than trusted, and whose architecture makes the conditions of knowledge production visible and auditable. Three case studies demonstrate the method's analytical value and limitations, including what we call the Heraclitus effect, the steppingstone dilemma, and the gullibility (or mafia) problem.

0.20

Converging Naming Styles, Persistent Network Locality: GitHub in the LLM Era

cs.SI ·2026-09-17 ·Yuto Tamura, Sho Tsugawa
similarity 0.58
Abstract

Social conventions often emerge through repeated interactions within social networks, allowing shared practices to coexist with variation across groups. Large language models (LLMs) introduce a potentially different coordination structure: a small number of widely used models can expose socially distant users to similar patterns and suggestions. Whether broad convergence under such shared technological influences eliminates network-local variation remains unclear. We examine this question in software development, where identifier naming styles provide observable conventions and LLM-based tools have rapidly diffused. Using public GitHub repositories created between 2015 and September 2025 across six programming languages, we characterize naming styles with 27 features and examine their association with detected LLM-related commits, their diversity across repository creation cohorts, and their relationship to owner proximity in a large-scale collaboration network. Repositories with detected LLM-related commits tend to use longer identifiers and, in several languages, make greater use of naming patterns already prevalent within the language. We also observe lower naming-style diversity in recent creation cohorts, with marked declines appearing around 2023-2024 in several languages, although their timing and trajectories differ. At the same time, network locality persists: in five of the six languages, repositories whose owners are closer in the collaboration network remain more similar in naming style even among recent, more homogeneous cohorts. These findings show that aggregate convergence and network-local variation can coexist, highlighting the need to examine not only how much cultural variation remains, but also how that variation continues to be structured by human social relationships in the era of widely shared AI systems.

0.23

Ritgard: T(r)opical Islands of Socio-Technical Artifacts on GitHub

cs.SE ·2026-09-04 ·Adam Štěpánek, Marco Raglianti, Jan Byška, Barbora Kozlíková, Michele Lanza
top venue similarity 0.58
Abstract · 5 pages, 3 figures, tool demo paper accepted at VISSOFT 2026

A software project is more than just code. Non-code artifacts often document the human processes and decisions behind source code. The rationale behind a library change, an architectural decision, a problem encountered by a user are all examples of information typically present in socio-technical artifacts (STAs), created and persisted in channels separate from the repository itself (yet sometimes very close---e.g., GitHub Issues with GitHub repositories). These STAs are a trove of information about the project's architecture and its evolution, containing details and insights that code alone cannot provide. Unfortunately, this information is not easily extracted and explored as STAs are frequently fragmented over different communication channels, and are written in natural language. We present Ritgard, a tool that mines GitHub repositories for their STAs, namely Issues, Pull Requests, and Discussions, and visualizes them as 3D islands covered with trees. Each tree represents a single artifact and each island is a topic extracted from the artifacts through a combination of text embedding and text summarization. The terrain of the islands rises out of the ocean as the topic becomes active and sinks back in when it becomes stale, thus depicting the evolution of features and concerns throughout the project's lifetime. We describe the tool's usage and implementation, showing the numerous technical challenges behind Ritgard's visualization.

0.47

AI-Research Agents in the Wild. From GitHub and arXiv to Regularities and Gaps

cs.DL ·2026-09-01 ·Aleksey Komissarov, Andrey Ustyuzhanin
similarity 0.58
Abstract · 56 pages, 1 figure

AI-research agents, or autoresearch systems, combine language models with tools, search, evaluation, and iterative modification of research artifacts. Their public software ecology is hard to compare because repositories, papers, benchmarks, libraries, and companion artifacts are often counted as one population. We connect two agent-drafted, human-adjudicated registries frozen on 10 June 2026: 139 canonical public repositories and 101 papers on AI-assisted research systems, each record carrying an evidence card anchored to its primary sources. Nine promoted design lineages contain 59 memberships among 52 repositories, seven of them assigned to two lineages. Against this evidence we examine six time-stamped candidate design regularities. A prospective audit of 25 newly ingested repositories observed zero of six specified trigger events, supplying bounded support for R1-R4; broader corpus evidence narrowed the original scope of R5, and a failed directional prediction leaves R6 exploratory. The paper-repository graph contains 212 curator-assigned relatedness edges: 64 of 101 papers carry at least one edge, the edges reach 23 of 139 repositories, and the six most-linked repositories carry 120 of the 212 edges (56.6%). A full-text check of seven papers shows the relation measures our curation, not the papers' citation behavior. An identity audit adjudicated against commit histories and public contributor records confirms at least 18 paper authors who also author a canonical registry repository (20 of 63 candidate author slots in 13 of 101 papers); a byline-matched tier reaches 58 identities but is reported separately and more weakly. The study contributes a source-grounded theory under prospective test, a curator-assigned map of paper-repository relatedness, and explicit limits on claims about public visibility and community structure.

0.18