CS Paper Radar

LLM Signals in Funded Grants: Evidence from International Funding Agencies

M. Z. Naser

cs.DL ·submitted 2026-09-03 11:39 UTC · v1

Abstract

The release of ChatGPT in November 2022 introduced a writing tool of unprecedented fluency into the daily routines of researchers across the sciences. Prior work has measured what follows in journal abstracts and in peer reviews by documenting an upward shift in the frequency of words and short phrases that large language models (LLMs) tend to overproduce. However, a much less-examined question concerns whether the same shift extends to grant proposals, the documents through which researchers compete for public funding, and, in particular, to the subset of proposals that succeed. Such a distinction is important to note because funded grants represent decisions to allocate public research resources, and because the proposing population spans every scientific discipline in which a country supports research. This brief report draws on an analysis of 221,425 funded grant abstracts from three major science-funding agencies in two countries: the U.S. National Science Foundation (NSF), the U.S. National Institutes of Health (NIH), and UK Research and Innovation (UKRI). The window covers late 2017 through mid-2026, providing roughly 5 years of pre-ChatGPT baseline and 3.5 years of post-release observation in each corpus.

Closest papers

Nearest neighbours by meaning across the whole index.

Scoring Grant Applications with Large Language Models

cs.DL ·2026-09-21 ·Mike Thelwall, David Pride, Francesco Osborne, Angelo Salatino, Petr Knoth +2 more
similarity 0.66
Abstract

Purpose: Assessing grant applications is time-consuming and difficult, adding to the overall burden of academic peer review. Whilst funders are exploring whether AI can help, there is no published research into the accuracy of Large Language Models (LLMs) for scoring contemporary grants. Design/methodology/approach: This study investigates whether six open-weight LLMs (Gemma 3 1B/4B/12B/27B, DeepSeek R1 32B, Qwen 3 32B) can give useful scores for 2267 recent UK Economic and Social Research Council (ESRC), and Engineering and Physical Sciences Research Council (EPSRC) grant applications, comparing them with scores from the original reviewers and funding panel members. Findings: Although the LLM scores are individually inaccurate, when averaged and converted to ranks they correlate positively with expert average scores. The best performing LLM, Gemma 3 27B (10 iterations with varied prompts), had moderate rank correlations with average reviewer scores (mean rho=0.26). Gemma 3 27B's average correlation with individual reviewers was 0.19, which is lower than the inter-reviewer mean correlation of 0.24, suggesting that it scores are slightly weaker than individual reviewer scores. Gemma 3 27B had weak rank correlations with average panel member scores (mean rho=0.17), with lower average correlations with individual panellists (mean rho=0.14), which is substantially lower than the inter-panellist correlation (mean rho=0.38). Whilst the correlations seem too weak to replace expert review at the final panel stage, LLM scores might help with the initial reviewing state, such as by helping identify the weakest proposals for fast-track desk rejections, to replace one human reviewer, or for triangulation to check for bias.

0.11

Large Language Models Have Not Yet Transformed Science

cs.CL cs.AIcs.CYcs.DL ·2026-11-17 ·Ron Bekkerman, Nichole Li, Bahareh Kargar, Mark Somers
similarity 0.64
Abstract

The emergence of large language models (LLMs) in late 2022 has raised questions about the future of science, but assessing their impact requires understanding how scientific practices were already evolving. We analyze 2.5 million full-text papers across five disciplines from 2015 to 2025, tracing trends over a broad spectrum of lexical, structural, epistemological, and social dimensions. Aside from frequency shifts in some words, most pre-2023 trends continued with limited detectable breaks. Among the notable exceptions, growth in computer science coauthor teams accelerated significantly in Asia, while apparent declines in biomedical topic diversity largely reflected standardization in article formats. Our study establishes an empirical foundation for monitoring the health of global science as LLM adoption deepens.

0.22

LLMs and the Organization of Scientific Work

cs.CY ·2026-11-02 ·Stefano Bianchini, Georgiana-Ioana Coroama
similarity 0.63
Abstract

The advent of large language models (LLMs) represents a technological revolution for science. Their diffusion may reshape not only how scientists conduct research, but also how they organize their work within teams. Here, we develop a conceptual framework in which LLMs expand individual capabilities across a range of research activities, reducing reliance on complementary human inputs within tasks and facilitating a more specialized division of labor among team members. We further propose that teams' history of collaboration may constrain their ability to reorganize in response to the new technological capabilities. Using CRediT contribution statements from approximately 120,000 PLOS ONE articles published between 2018 and 2025, we investigate changes in the division of labor within scientific teams following the release of ChatGPT in late 2022. We document three main findings. First, participation in scientific tasks falls below its previous trajectory, particularly in conceptualization, execution, and writing. Second, researchers become more specialized, exhibiting narrower task portfolios and less overlap in their contributions with co-authors. Third, these changes are concentrated among newly formed teams, whose members have no prior collaboration ties, whereas established teams, whose members have all collaborated previously, exhibit considerably greater organizational inertia. Our findings suggest that, in the wake of LLM diffusion, scientific work is being reorganized around greater individual autonomy and an increasingly specialized division of labor within teams.

0.22

Accelerated Drift of Scientific Prose in the LLM Era: Evidence from an Interrupted Time-Series Analysis

cs.CL cs.AI ·2026-10-19 ·Hanna Hubarava, Benjamin V. Ineichen
top venue similarity 0.62
Abstract · Accepted at EMNLP 2026. 22 pages, including references and appendices

Every scientific text is embedded in the writing norms of its time: it reflects earlier publications while also shaping the texts that follow. The growing use of LLM-based writing tools introduces a new factor in this evolving system, but the linguistic changes coinciding with their emergence remain insufficiently understood. We investigate whether the trajectories of scientific-preprint writing changed in rate or direction after conversational LLMs became publicly accessible, using the release of ChatGPT in November 2022 as the observational breakpoint. We compare pre- and post-2022 trajectories across vocabulary, punctuation, readability, and syntax in abstracts from arXiv's artificial intelligence, statistics, and quantitative biology categories and from medRxiv. Using interrupted time-series regression, we find substantial post-2022 changes across multiple feature families and research domains, including increases in average word length, reduced formula-based readability, and declines in maximum dependency-tree depth and sentence-level depth variation. Individual features follow different dynamics: some lexical markers accelerate persistently, while others rise and then recede. Our findings show that isolated marker frequencies capture only part of the shift: changes in the rate, direction, and structure of linguistic trajectories provide complementary evidence of broader post-2022 change in scientific-preprint writing. More broadly, these findings motivate monitoring scientific writing as a multidimensional, evolving system rather than through a fixed set of LLM-associated markers.

0.47

Patterns of Research Funding Across Research Subjects: The Case of NSERC

cs.DL ·2026-08-11 ·Rabeeh Parhizkari, Gita Ghiasi
similarity 0.61
Abstract

Funding agencies aim to maximize scientific output from the resources they allocate, a goal that may inadvertently reinforce cumulative advantage within funding systems. Focusing on research areas within Natural Sciences and Engineering (NSE), this study examines the distribution of Canadian research funding over time, with particular attention to the extent to which certain subjects gain prominence as emerging or high-growth fields, while others experience relative stagnation or decline. The results indicate that many research subject groups receiving decreased funding in recent years are concentrated in the pure sciences and mathematics. In contrast, social sciences and medical sciences, which initially had relatively low average funding per grant application, have demonstrated increasing funding trends over time. These findings offer valuable insights for funding agencies seeking to design policies that promote more effective and equitable allocation of resources.

0.04

Most biomedical publications show signs of LLM-assisted writing

cs.CL cs.AIcs.CYcs.DL+1 ·2026-08-11 ·Lena Holzwarth, Rita González-Márquez, Dmitry Kobak
similarity 0.60
Abstract

Over the past several years, LLM-powered chatbots and agents have become widely used as a tool for academic writing. LLM-assisted writing can be valuable by removing language barriers but at the same time causes concerns about misconduct and fraud. To inform policy decisions, it is necessary to monitor the prevalence of LLM-altered texts in scholarly publications. Despite some recent progress in this direction, no existing method can produce reliable estimates. Here we suggest and validate a new unbiased approach to estimate LLM usage in a corpus of texts based on changing word frequencies. We apply our method to the full texts of open-access biomedical papers from Pubmed Central, and show that by the end of 2025, 89% of papers show excess of LLM-associated vocabulary. We also find that LLMs are twice as likely to be used when writing a paragraph in the Discussion section (68%) compared to a paragraph in the Methods section (32%), but even inside the Methods section, the overall prevalence of LLM usage is over 50%. We believe that our estimates are crucial to shape future guidelines and policies.

0.41

Early Reflections on How Large Language Models are Reshaping Software Engineering Research

cs.SE ·2026-11-11 ·David Williams, Michail Kyrkos, Max Hort, Maria Kechagia, Justyna Petke +1 more
similarity 0.59
Abstract · 5 pages, 2 figures, 1 table

Large language models (LLMs) are radically reshaping software engineering (SE) research. Yet, their steep financial and computational costs raise questions about who participates in that change, and who does not. By analysing all 1,918 papers published in the ICSE 2019-2026 Research and SEIP tracks, we uncover a shifting and highly concentrated research landscape. China and the US dominate LLM-based SE research, appearing in 74% of LLM-based publications between 2023-2026, yet they diverge sharply in their use of commercial versus open-weight models. Beneath this national dominance lies extreme institutional concentration: just 5 institutions in China and 7 in the US appear in more than half of their respective nations' LLM-based output. In the US, only half of the unique institutions active in 2023-2026 participated in LLM-based research, whereas Singapore displays broader usage, with 11 of 15 active institutions participating in LLM-based work. Furthermore, across 2023-2026 Research-track papers, industry involvement is more prevalent in LLM-based work (23.4% vs 16.9%), and academia-only papers make up a smaller share (63.4% vs 70.9%). Crucially, we expose a gap in model usage across industry and academia. SEIP track papers lean strongly towards commercial LLMs, with 77.4% featuring them and 40.5% using them exclusively. In contrast, only 21.2% of Research-track work relies strictly on commercial models, making greater use of open-weight models that typically support more sustainable and reproducible experiments. However, this divide warrants reflection on the extent to which SE research aligns with industry practice.

0.30

IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

cs.CL ·2026-11-02 ·Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar +1 more
benchmarkcommunity pick similarity 0.59
Abstract

Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.

0.28