CS Paper Radar

Efficient Context-Limited Telescope Bibliography Classification for the WASP-2025 Shared Task Using SciBERT

Madhusudhana Naidu

cs.LG astro-ph.IM ·submitted 2026-08-29 17:21 UTC · v1
publishedbenchmarkcode/data

Abstract

The creation of telescope bibliographies is a crucial part of assessing the scientific impact of observatories and ensuring reproducibility in astronomy. This task involves identifying, categorizing, and linking scientific publications that reference or use specific telescopes. However, this process remains largely manual and resource intensive. In this work, we present an efficient SciBERT-based approach for automatic classification of scientific papers into four categories - science, instrumentation, mention, and not telescope. Despite strict context-length constraints (maximum 512 tokens) and limited compute resources, our approach achieved a macro F1 score of 0.89, ranking at the top of the WASP-2025 leaderboard. We analyze the effect of truncation and show that even with half the samples exceeding the token limit, SciBERT's domain alignment enables robust classification. We discuss trade-offs between truncation, chunking, and long-context models, providing insights into the efficiency frontier for scientific text curation.

Comment
3 pages, 2 tables. 1st place system description for the TRACS shared task at WASP 2025 (Third Workshop for Artificial Intelligence for Scientific Publications), co-located with IJCNLP-AACL 2025. Published version: https://aclanthology.org/2025.wasp-main.21/ . Code: https://github.com/E0NIA/TRACS-WASP-2025-1st-Place
Journal ref
Proceedings of the Third Workshop for Artificial Intelligence for Scientific Publications (WASP 2025), pages 192-194, Mumbai, India and virtual, December 2025. Association for Computational Linguistics
DOI
10.18653/v1/2025.wasp-main.21

Closest papers

Nearest neighbours by meaning across the whole index.

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

cs.AI ·2026-09-30 ·Yerim Oh, Young-Jun Lee, Jaewoo Ahn, Gunhee Kim, Dongyeop Kang
benchmarkcode/data similarity 0.67
Abstract · 28 pages, 6 figures, 13 tables. Project page: https://yerimoh.github.io/scientific-slop-demo/

AI-generated content, often called AI slop, is increasingly common everywhere, particularly in academia. Slop in AI-generated scientific papers, however, has more complex patterns that cannot be easily detected by existing token-based AI detectors. Each part of such a paper looks plausible while the scientific reasoning that connects the parts breaks down, which can mislead how readers assess the work. We benchmark these failures as scientific slop through six measures across Structure, Argument, and Artifacts. We construct SciSlopBench with 390 AI-generated papers, mostly in computer science but spanning the life, social, and natural sciences, each paired with a human-written paper matched by research problem and contribution type. Our measures identify the AI paper in each pair with 85.9% accuracy, compared with 68.7% for Binoculars. Higher scientific slop accompanies lower ICLR ratings and distinguishes rejected from accepted papers above chance in every year from 2017 to 2025. Reducing these patterns, however, is not as simple as directly optimizing the measures. We therefore propose SciSlopHarness, a harness-level framework that guides a fixed LLM to revise slop only where the experiment records support the change. While standard revisions leave residual slop and direct slop-aware prompting triggers reward hacking, SciSlopHarness reduces the remaining AI-human gap by 63% over the strongest revision baseline without requiring human reference targets. Overall, we demonstrate that AI-generated scientific papers leave fundamental traces in their global reasoning, and that responsible mitigation demands strict evidentiary grounding rather than mere prose refinement.

0.52

MATCH: Multi-stage Article Tagging to Concept Hierarchies

cs.DL ·2026-10-19 ·Daniele Raimondi, Feichi Lu, Oliver Grun, Mariia Eremina, Andrea Perlato
similarity 0.65
Abstract

The increasing specialization of scientific research calls for fine-grained representations of publication content that can be applied at scale. SCALE extends the OpenAlex taxonomy with approximately 114,000 Scientific Concepts derived from author keywords, but assigning these Concepts to individual publications remains challenging. We introduce MATCH (Multi-stage Article Tagging to Concept Hierarchies), a framework that assigns SCALE Concepts using article titles, abstracts, and author keywords. MATCH combines multi-view embedding retrieval, deterministic taxonomy-aware filtering, and language-model reranking to select relevant Concepts while preserving their hierarchical paths. Its modular design supports both API-based and self-hosted inference. In a blind evaluation involving 22 domain experts across 11 fields and 110 articles, the self-hosted and API-based configurations achieved Precision of 81.5% and 83.3%, respectively, and Precision@5 of approximately 89% for both, with no statistically significant difference detected. These findings support the practical viability of the evaluated self-hosted configuration for large-scale Concept assignment without per-token API charges. Deployed on over two million MDPI publications, MATCH assigns a median of five Concepts per article, covers approximately 94% of the SCALE Concepts, and supports daily indexing of newly available publications. By operationalizing the SCALE taxonomy at publication level, MATCH provides a foundation for fine-grained scientometric analysis and monitoring of scientific developments.

0.21

RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

cs.CL cs.IR ·2026-08-27 ·Maayan Sharon, Tom Hope
benchmark similarity 0.65
Abstract

Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction---zooming out to a more general view or zooming in to a concrete realization. We introduce RATIO (Retrieval Across Typed Ideation Operations), a large-scale benchmark in which relevance is defined by three operations which we name ideation moves: Address retrieves potential approaches for stated problems, Broaden retrieves more general formulations, and Specify retrieves concrete instantiations. RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision---previously used only for classification---to corpus-scale retrieval, combined with extensive LLM and human vetting. Experiments show that operation-specific fine-tuning substantially boosts retrievers and LLM reranking helps further, but much room for improvement remains. Researchers blind to condition found the fine-tuned retriever's candidates more useful as inspirations. RATIO provides a scalable training and evaluation framework for retrieval components that support literature-grounded ideation, opening up new research avenues in scientific inspiration retrieval. We release full data, code, models, and annotations.

0.16

Taramandal-GPT: Enhancing Astrodynamics Problem-Solving with Knowledge Retrieval and Structured Thinking

cs.CL cs.AI ·2026-09-21 ·Akhil Sharma, Jatin Gupta, Ali Imam Abidi
publishedbenchmark similarity 0.65
Abstract · Proceedings of All India Hindi Technical Conference, 05-06 February 2026

Large language models (LLMs) have shown remarkable progress in natural language understanding, yet their effectiveness in specialized fields like astronomy and astrodynamics remains limited due to challenges in multi-step reasoning, symbolic manipulation, and domain-specific terminology. To address this, we present Taramandal-GPT (Constellation-GPT), a domain-adapted framework built on the Qwen3-8b backbone, enhanced with a Retrieval-Augmented Generation (RAG) pipeline and a fallback mechanism for improved contextual precision. We evaluate it on the Astrodynamics Problems Benchmark (APBench), a dataset of 299 questions covering foundational to advanced levels of space science. Using a dual evaluation method - numeric margin-based scoring and semantic similarity assessment - Taramandal-GPT achieves competitive performance against state-of-the-art open- and closed-source models, with notable strength in thinking-intensive tasks. These results highlight the value of specialized LLMs for domains demanding accuracy and interpretability, positioning Taramandal-GPT as a step toward reliable Artificial Intelligence (AI) assistants for astrophysics, spacecraft engineering, and space exploration.

0.17

Local Citation Recommendation Without Graph Signals

cs.IR cs.CLcs.DL ·2026-10-11 ·Benjamin Basseri, Pavlos Protopapas, Kemeng Zhang, Alberto Accomazzi, Stephanie Jarmak +1 more
publisheddatasetcode/datahot direction similarity 0.64
Abstract · Accepted at WASP 2026, co-located with AACL-IJCNLP 2026. 16 pages, 3 figures, 2 tables. Dataset: https://huggingface.co/datasets/adsabs/LCRC…

Resolving correct citations can be a time-consuming task for research writers. We address local citation recommendation (LCR), where the goal is to retrieve appropriate references for a given sentence. Web tools generally depend on the existing citation graph, which may bias results toward citation count or more recent publications. LCR research using full paper texts has been limited by the lack of publicly available data. We present a new dataset, the Local Citation Recommendation Corpus (LCRC), containing 14,735 scientific statements derived from astrophysics review papers, each with one or more citation targets whose full text is accessible via API. Using this dataset we systematically compare retrieval methods and develop a citation recommendation system that avoids citation graph bias while achieving 8.0% improvement in Recall@25 and 6.9% improvement in NDCG@25 over dense retrieval baselines without model fine-tuning. To support reproducibility, we make our code and dataset publicly available.

0.73

NLPCC 2026 Task 10: Citation-Level Faithfulness Verification with DeBERTa Ensembles and Class-Wise Calibration

cs.CL ·2026-09-19 ·Yanling Li, Zirui Li, Mingyu Wan
publishedbenchmark similarity 0.64
Abstract · Accepted to the NLPCC 2026 Shared Task 10 system paper track. Ranked 2nd in Track 2 on the final leaderboard

This paper presents our system for Track 2 of the NLPCC 2026 Shared Task 10 on citation-level faithfulness in AI-assisted scientific reporting. Given an atomic scientific claim and the structured full text of its cited paper, the task requires both a four-way relation label and up to three evidence paragraph identifiers. The label head ensembles a paragraph-aware cross-encoder with a document-level DeBERTa-large classifier, followed by class-wise decision calibration. Probability-level fusion is motivated by an out-of-fold tendency to over-predict Topical Match. The evidence head combines paragraph scores from top-20 and top-30 joint models with BM25 scores. The system runs fully offline without external retrieval or LLM prompting. On the final leaderboard, our system achieved 82.9898 overall (89.5491 Macro-F1 and 76.4305 Joint@3), ranking second in Track 2. Ablations and error analysis show that model complementarity and calibration drive the label gains. Gold-evidence inference changes label Macro-F1 negligibly, whereas evidence ranking remains important for Joint@3.

0.28

Retrieve-then-Disambiguate: Scalable Reviewer Discovery

cs.IR cs.DL ·2026-10-29 ·Vicente Amado Olivo, Joe Petrecca, Wolfgang E. Kerzendorf
code/data similarity 0.63
Abstract

The growing number of scientific submissions is straining the peer review process, while a large pool of relevant researchers remains unidentified and is therefore not considered as potential reviewers. We formulate the discovery of these researchers as a retrieval problem: given a submission, we retrieve and rank researchers based on domain similarity. We present retrieve-then-disambiguate, an open-source algorithm that identifies domain relevant researchers for a query submission by retrieving similar publications from the scientific literature and resolving ambiguous author names into unique researcher identities. We evaluate the algorithm on 500 astronomy and physics publications. We use ORCID-linked first authors, coauthors, and cited researchers as proxy labels to evaluate the recovery of researchers with known connections to each query publication. When retrieving the top 500 most similar papers, the algorithm recovers roughly 68\% of first authors, 60\% of coauthors, and 45\% of cited researchers. Our code is available at https://github.com/deepthought-initiative/Retrieve-then-Disambiguate.

0.39

Astro2Method: Benchmarking LLMs on Machine Learning Method Selection in Astronomy

cs.AI astro-ph.IM ·2026-10-21 ·Xingzhong Fan, Hangfeng He, Fatemeh Nargesian
benchmark similarity 0.63
Abstract · 15 pages, 4 figures, 4 tables

Machine learning (ML) is increasingly used in astronomy, but choosing a suitable method for a scientific problem often requires expertise in both astronomy and ML. Large language models (LLMs) could help astronomers make such choices, yet this ability remains underexplored in existing benchmarks. We introduce Astro2Method, a benchmark of 500 expert-reviewed binary questions from 346 astronomy papers (2024/01-2026/06), based on clear performance differences and complete (Easy) or partial (Hard) explanations from the authors. Across five LLMs, direct prompting achieves 87.50-92.55% accuracy on Easy questions but only 58.87-64.52% on Hard. Using historical papers, neither retrieval-augmented generation nor most agent methods with additional reasoning or retrieval (including ReAct, Self-Ask, and CRITIC) bring consistent improvement. Agents required to search the web improve accuracy mainly when they find the papers the questions are based on, with no consistent improvement otherwise. Providing the source paper's pipeline description with method names masked improves Hard accuracy, matching or exceeding web search. Together, these results suggest that relevant experimental information helps distinguish ML methods, but inferring it from related studies and domain knowledge remains challenging for LLMs in astronomy.

0.28