CS Paper Radar

PyReprism: A Shared Lexical Foundation for Polyglot Code Analysis

Daniel Ogenrwot, Jorge Gonzalo Delgado Cervantes, Richard Sserunjogi, John Businge

cs.SE ·submitted 2026-10-29 02:43 UTC · v1
code/data

Abstract

Empirical studies of polyglot repositories apply the same analyses to every language of a corpus, such as separating code from comments, computing metrics, normalizing identifiers, and screening for duplicated code. Assembling them from specialized tools introduces differences in language coverage, measurement conventions, and workflows. We present PYREPRISM, an open-source Python library and command-line tool that separates language-specific lexical processing from reusable analyses: each language declares its lexical surface, which yields a lossless stream of typed tokens, and every analysis is implemented once on that stream. Across 187 built-in languages, language-specific declarations have a median length of 16 lines, and no language overrides a shared analysis. Against a reference parser, comment and string classification reach F1 scores of 0.994 and 0.958 on 1,412 files in 12 languages that were never used to develop the tokenizer. In a study of a polyglot repository, one tool replaces three; with matched inputs and analysis operations, it takes 2.9s instead of 8.7s on one core. We also characterize where lexical analysis falls short, in complexity estimation and clone screening. Tool: https://github.com/unlv-evol/PyReprism Video: https://youtu.be/IXc-NP5t4GU

Comment
5 pages, 2 figures, Submitted to SANER 2027 Tool Demo Track

Closest papers

Nearest neighbours by meaning across the whole index.

SIEVE: A Parameterized GitHub Corpus Builder for Software Engineering Research

cs.SE ·2026-10-15 ·Musfiqur Rahman, Mahi Begoug, Emad Shihab
code/data similarity 0.69
Abstract

Software engineering research increasingly relies on large, richly annotated corpora of real-world code. Still, researchers typically assemble these with ad-hoc scripts or static, pre-built datasets that offer no control over corpus composition. This leaves a gap for a reusable tool that goes below the repository level and builds annotated, function- and class-level code corpora under researcher-specified criteria. We present SIEVE (Software Ingestion & Extraction for Verifiable Evaluation), an open-source, parameterized GitHub corpus builder that, unlike repository-level mining tools, works at code granularity: it locates individual functions, methods, and classes in repositories and extracts them as records, giving direct control over corpus construction based on several criteria, including widely used programming languages (Python, Java, JavaScript, C++), repository quality, and creation date. To support software engineering research and LLM evaluation on real-world code, SIEVE extracts CodeSearchNet- and OpenClassGen-compatible records, each annotated with 22 structural metrics and AST-derived features, and optionally with a CodeBERT-based P(AI-generated) score (F1=0.9478, AUROC=0.9902), alongside the third-party dependencies declared in each repository's package manifest. These annotations enable researchers to conduct structural analyses, investigate code dependencies, assess AI-generated code, and construct customized benchmarks for evaluating LLMs, including under temporal constraints. A manual validation of 400 records in four languages and two granularities shows 100% accuracy in code metrics and import detection, 99.0% in source completeness, and 99.5% in metadata accuracy. SIEVE is publicly available with a live demo at https://mrahman2025-sieve.hf.space.

0.30

PolyCodeEval: Benchmarking Multilingual Code Generation from Functions to Repositories

cs.SE ·2026-10-08 ·Bowen Yang, Jiajun Jiang, Luxue Yu, Yihao Wang, Fengjie Li +1 more
benchmarkcode/data similarity 0.69
Abstract · 22 pages,4 figures

As large language models increasingly move toward repository-level software engineering, existing code-generation benchmarks remain fragmented across language coverage, task granularity, and evaluation protocols, impeding systematic comparison. To address this gap, we present PolyCodeEval, a unified multilingual and multi-granularity benchmark for code generation. It comprises 2,590 code generation tasks spanning functions to repositories, derived from 58 real, executable open-source repositories in five programming languages. All tasks are evaluated under a unified execution-based protocol with integration procedures tailored to their generation targets. Building on this benchmark, we evaluate frontier large language models, state-of-the-art specialized methods, and general coding agents. Our results show that existing approaches still struggle to correctly generate complete code fragments across granularities and languages. Specifically, the studied methods generate at most 71.7%, 76.7%, and 31.0% correct functions, files, and repositories, respectively, with performance varying widely across languages. Paired experiments further show that implementation context from related functions in the same file improves the executable correctness of function generation. Method rankings also vary across task granularities and programming languages, highlighting the importance of multilingual, multi-granularity evaluation for comprehensively assessing code generation capabilities.

0.28

Evaluating Language Models on Cross-Language Code Functional Equivalence

cs.SE cs.AIcs.CL ·2026-08-25 ·Hui Sun, Anderson Uchôa, Rohit Gheyi, Wesley K. G. Assunção
top venuedataset similarity 0.66
Abstract · 20 pages. To appear in the 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026), LIPIcs vol. 394, Arti…

Background: Large Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. However, existing evaluations primarily focus on single-language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. Aims: We investigate whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity. Method: We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python. Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. Results: We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Conclusions: Current LLMs do not reliably capture functional equivalence within or across languages.

0.59

Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities

cs.CL cs.CY ·2026-08-10 ·Avijit Roy, Proma Roy, Hrishitva Patel
publishedbenchmarkhighly cited similarity 0.66
Abstract · Accepted at IJCAI 2026 Workshop (https://lm4uc.github.io/)

Large language models are increasingly deployed as general-purpose educational and technical assistance systems, but their underlying infrastructure does not treat languages equally. One underexamined source of disparity is tokenization: semantically equivalent content can require substantially different token counts across languages, affecting API cost, latency, and usable context length before a model is invoked. We introduce the Tokenization Equity Audit (TEA), a reproducible benchmark for measuring tokenization premiums in technical tutoring content. TEA evaluates three widely used tokenizers, GPT-4o's o200k base, Qwen2.5-7B, and Mistral-7B, on a 120-item Python debugging corpus translated from English into Bengali, Hindi, Arabic, Tamil, and Yoruba. Bengali and Hindi serve as the primary validated cases, while the remaining languages provide exploratory cross-script and cross-family comparisons. Across this corpus, Bengali requires (1.56\times) as many GPT-4o tokens as English, reducing a nominal 128k-token context window to an effective 82k-token English-equivalent capacity for the same semantic content. With the Qwen2.5 and Mistral tokenizers, Bengali requires up to (4.5\times) the English token count. Yoruba, despite using the Latin script, exhibits the highest GPT-4o tokenization premium at (2.37\times), indicating that tokenization inequity cannot be explained by script family alone. These results demonstrate that tokenization can create measurable economic and functional barriers, highlighting the need to treat tokenization as an equity-relevant infrastructure layer for underserved language communities, particularly where educational systems depend on low-cost or offline-capable AI tools.

0.72

Detecting Argument-Swap Bugs Using Context-Enhanced Code Representations

cs.SE ·2026-09-15 ·Subrata Das, Ali Aman, Muhammad Asaduzzaman, Kawser Wazed Nafi, Salimur Choudhury
publishedbenchmarkdataset similarity 0.65
Abstract · Accepted in the 26th IEEE International Conference on Source Code Analysis and Manipulation (SCAM 2026)

Names of source code elements convey rich semantic information and have been widely used in software engineering tasks such as bug detection, code completion, type prediction, and code classification. Prior studies exploit lexical similarity between method arguments and formal parameter names to detect bugs caused by incorrectly ordered arguments, typically relying on establishing mappings between method calls and their corresponding definitions. However, such mappings are often difficult to obtain in dynamically typed languages like Python. In this paper, we present BugProbe, a learning-based approach for detecting incorrectly ordered arguments in Python method calls that does not require call-to-definition mappings. Our approach leverages multiple sources of contextual information, including local context and argument usage context, and combines name-based similarity with machine learning to construct expressive representations of method arguments. We collect a new dataset of 132,739 Python source files from the top-1,000 starred GitHub repositories, yielding 3,371,244 synthetic training examples, and contribute a curated benchmark of 55 real-world argument-swap bugs manually verified from commit histories. We evaluate our approach on this dataset and show that it achieves high accuracy and consistently outperforms a state-of-the-art baseline across standard evaluation metrics. These results demonstrate that effective detection of argument-ordering bugs is possible without relying on explicit call-to-definition resolution, making the approach well suited for dynamically typed language settings.

0.17

Program Bindings Inside Code Models

cs.SE ·2026-10-29 ·Prateek Kumar Rajput, Yewei Song, Jacques Klein, Tegawendé F. Bissyandé
similarity 0.65
Abstract · 22 pages, 2 Figures, 4 Tables

Code language models complete, review, and repair programs without exposing the program structure encoded in their hidden states. In this paper, we test whether the hidden states of a code model can be described and edited using the information a compiler front end computes. For each token of a Python program, standard analyzers provide its position in the abstract syntax tree, its scope, its definition-use links, and its syntactic category. We fit a decomposition that expresses every hidden state of a frozen model as a sum of one term per token, each combining the token with this structural label, and we replace the model's own states with the decomposition at every layer while it predicts the output of a small program. If the model still answers correctly, the decomposition preserves the information the model uses. We apply this test to 14 open models, seven of them trained for code, and compare against a baseline decomposition of the same size built from token positions. The static-analysis decomposition preserves the model's answers better than the positional one in 10 of the 14 models, including six of the seven code models, and deleting the definition-use links from it removes most of that advantage on the two models tested. Editing the decomposition, for instance rebinding the argument of a call, makes StarCoder2-15B produce the answer of the edited program in 90 of 100 cases, compared with 19 for the positional baseline. The Qwen2.5 family is the exception on several measures. For most of the models tested, standard static analysis therefore provides both a description of what the hidden states encode about a program and a way to change it without retraining.

0.24

Understanding Python Linters through a Comprehensive and Systematic Empirical Study

cs.SE ·2026-10-26 ·Qingyun Yang, Zejun Zhang, Dapeng Yan, Tian Zhang, Zhenchang Xing +2 more
benchmark similarity 0.65
Abstract

Code linters, tools that automatically detect code issues, help developers accelerate development and improve code quality by identifying errors early. Unlike previous studies that focused mainly on single linters or informal comparisons, this paper presents the first large-scale empirical study of five widely used Python linters: Ruff, Pylint, Flake8, Mypy and Black, through four key user concerns: linting rule checks, speed, configuration methods, and linter preferences. We analyze 1,551 linting rules, categorize them into five types, and construct rule mappings, finding that more than 40% are shared between multiple linters, with Ruff and Pylint together covering over 80% of all rules. Pylint has the most rules in the first two categories, and Ruff leads in the last three. Next, we measure their running times on 1,000 GitHub projects under three setups (single-rule, default, all-rules), finding a consistent speed ranking from fastest to slowest: Ruff, Mypy, Flake8, Black, and Pylint. Ruff is extremely fast, with runtimes typically within one second. Then, we summarize four configuration methods and four types of configuration options from 443 available options across the five linters. While all linters support the four configuration methods, they differ in project configuration formats and flexibility in their configuration options. Finally, an analysis of 3,094 GitHub projects using the five linters reveals that over 50% use multiple linters, share strong consensus on option selections and rule suppressions, but exhibit project-specific variations in enabled-rule configurations. Our findings provide actionable insights for linter users and establish a benchmark dataset and taxonomy to support future research, such as cross-linter rule mapping, performance modeling, and automated configuration.

0.13

MetaphIR: Corpus-Scale Applicability Analysis for LLVM Peephole Optimizations

cs.PL cs.SE ·2026-11-17 ·Hongxu Xu, Xintong Zhou, Chunhao Liao, Chengnian Sun
similarity 0.65
Abstract

Peephole optimizations replace short source instruction sequences with more efficient equivalents, typically guarded by preconditions such as requiring a constant operand to be a power of two. Since every new rewrite adds maintenance burden and compile-time cost, compiler developers need to know how often real-world programs both match the source pattern and satisfy its preconditions before committing to an implementation. Existing workflows fall short: structural search over an intermediate representation (IR) corpus of 4.8 million functions ignores preconditions and therefore overcounts, whereas implementing the rewrite and recompiling the corpus yields exact counts only after paying the engineering cost, with each run of the corpus's on-demand recompilation pipeline taking a median of 47 minutes. This delay is especially limiting when generalizing a rewrite, as developers often iterate over multiple variants to trade off coverage against complexity. We introduce MetaphIR, a system pairing a domain-specific language for expressing LLVM peephole optimizations with an indexed search engine over large IR corpora. Using a syntax modeled on LLVM's C++ matchers, MetaphIR patterns augment LLVM IR with holes and Boolean preconditions over matched values. To answer queries at interactive speed, MetaphIR builds compact per-function summaries ahead of time and uses abstract interpretation over those summaries to soundly filter out functions that cannot satisfy a query's constraints. Across twenty recent peephole optimizations from LLVM's InstCombine pass, MetaphIR recovers the exact set of functions where each rewrite applies in isolation, achieving 100% precision and recall. Queries complete in a median of 1.27 seconds---a 22x speedup over a full corpus scan---enabling developers to quickly gauge an optimization's reach and generality prior to implementation.

0.14