CS Paper Radar

Yesterday's Code, Today's Dependencies: How Dependency Selection Shapes PHP Migration Benchmarks

Afridi Mahmud, Md. As-Aid Rahman, Rifat Shahriyar

cs.SE ·submitted 2026-10-30 05:12 UTC · v1
benchmark

Abstract

Migration benchmarks need projects whose tests pass before migration begins. Yet an old project can behave differently without any change to its code. Reinstalling it today may introduce newer libraries, changing both its initial test results and the failures observed during migration. We investigate this effect in PHP 7.4-to-8.3 migration. Using unchanged project source and saved package records, we compare two dependency policies. Modern selection allows all versions in the snapshot, while release-date selection allows only versions published by the project's release date. With Composer 2 fixed, release-date selection improves the PHP 7.4 pass rate by 7.1 pp across 3,660 releases. A separate comparison against committed locks shows that it also recovers recorded dependency versions more closely than modern selection. The choice matters after migration as well. Among projects that pass on PHP 7.4 under both policies, PHP 8.3 test outcomes differ in 15.0% of eligible pairs. We use a shared test runner and bypass Composer's platform check to isolate test behavior. The resulting candidate migration task sets overlap by only 58.5%. Most differences arise in Laravel projects, where older dependencies trigger deprecations that newer versions avoid. Targeted interventions show that removing these initial failures can restore passing tests or expose further incompatibilities. Thus, dependency selection changes not just whether an old project passes, but also which migration failures a benchmark can observe. Preserved locks provide the strongest available version reference; without them, release-date selection offers a historically grounded alternative. Migration benchmarks should make this choice explicit and report its effect on their results.

Comment
12 pages, 4 figures, 8 tables. Preprint

Closest papers

Nearest neighbours by meaning across the whole index.

When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP

cs.SE ·2026-10-29 ·Afridi Mahmud, Abid Hasan Khondaker, Sadif Ahmed, Md Nafiu Rahman, Rifat Shahriyar
similarity 0.62
Abstract · 6 pages, 5 figures, 5 tables. Preprint

Pasting a legacy file into an LLM and asking it to "modernize this" is an easy shortcut. But is the result better than leaving the code alone? In this paper, we study this single-file setting with five LLMs, each asked to modernize 100 WordPress 4.0 PHP files to PHP 8.3 while preserving behavior, with no project context or execution feedback. We evaluate every output with 630 hidden behavioral checks and 511 Rector-detected modernization findings. We compare against two baselines: the unchanged originals and the rule-based tool Rector. Often, the answer is no. Untouched, 83 files already pass every check; the best LLM leaves only 59. Each model breaks 32-51 working files while repairing at most 10, whereas Rector breaks 7 and repairs 6. Most of this damage is unnecessary. Only 37 findings (7 percent) require a change under PHP 8.3, yet in compatible files with no required change, the LLMs break 36-60 percent (Rector: 8 percent). The model with the highest modernization score is no more compatible than its closest competitors. Most regressed checks come from files that no longer load at all, often because of type declarations PHP itself rejects. These are errors that need no project context to avoid. Evaluations of LLM-based modernization should therefore measure against a no-change baseline and report modernization and behavioral preservation separately.

0.33

VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green

cs.SE cs.CR ·2026-10-12 ·Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal, Luke Tu
benchmarkcode/data similarity 0.56
Abstract · 22 pages, 5 figures. Code and data: https://github.com/snyk-labs/snyk-vulnbump

Software composition analysis (SCA) tools report far more vulnerable dependencies than teams fix: in Snyk Open Source production data, 94% of vulnerability findings have a known fixed version, yet only about 10% are resolved. Agentic coding tools could close this gap, but no benchmark evaluates them on vulnerability-driven upgrades in real, runnable projects across ecosystems, verified independently of the agent. We present VulnBump: 307 upgrade tasks covering 2,066 fixable vulnerabilities in 43 repositories across six package managers, matched to Snyk Open Source production vulnerability distributions. An upgrade passes only if, in a fresh container, dependencies resolve, a rescan shows the targeted vulnerabilities fixed, the project builds, and every previously passing test still passes. A non-agentic baseline that only applies Snyk's recommended version passes 46.9% of upgrades; with the same model (Claude Sonnet 5.5), Snyk's Evo Remediation Agent passes 90.2%, Claude Code + Snyk scan 77.5% and Claude Code unaided 59.6%. The agents fail in opposite ways: Claude Code mostly leaves the vulnerability in place, whereas most of Evo Remediation Agent's failures break the build or tests. An SCA scanner is a key tool for agents: Snyk's guidance raises Claude Code's PASS rate by 17.9 percentage points; without it, Claude Code often settles on a still-vulnerable version. Yet knowing the target is not enough: given the same Snyk findings, Evo Remediation Agent's purpose-built harness passes 12.7 points more than Claude Code + Snyk scan. Major framework migrations remain hard for all harnesses. Because the agents keep projects working at similar rates, evaluating remediation quality in dependency upgrades requires verifying the fix, the build and the tests together.

0.31

CrossPloit: A Benchmark for Cross-Version PoC Migration in Python Web Applications

cs.CR cs.SE ·2026-11-12 ·Hung Nguyen-Huu, Nguyen Anh Tuan, Ting Zhang, Yue Liu, Chengran Yang +3 more
benchmark similarity 0.55
Abstract

Affected-version ranges in vulnerability databases decide which deployments scanners, patching pipelines, and incident responders treat as vulnerable, yet these ranges are often incomplete, so vulnerable releases are silently treated as safe. Auditing them requires executable evidence: given a proof-of-concept (PoC) exploit for a listed version, show that the same vulnerability can be triggered on an earlier, unlisted version. This cross-version PoC migration combines source-code analysis, environment repair, and execution-based verification. We introduce CrossPloit, a benchmark of 127 migration instances across 45 CVEs in the Python web ecosystem. Each instance includes a version-pinned container, a reference PoC, and a vulnerability-specific oracle; two validators check that the observed effect arises from the disclosed vulnerability. Benchmark construction confirms 149 vulnerable prior versions absent from the affected-version lists of four databases, 127 of which require migration. Across 18 agent configurations (nine models and two harnesses), the best configuration resolves 58.3% of instances, compared with 8.7% for an adapted trace-guided baseline. Attack-vector shifts are particularly difficult: no configuration resolves more than 37.9% of these instances. Higher resolve rates are associated with a smaller share of missed-API failures and a larger share of incomplete executions among the remaining failures. In one configuration, withholding the reference PoC and its environment specification reduces resolve rate from 50.4% to 15.0%, highlighting the value of the supplied reference package. CrossPloit provides an execution-based measurement of under-reported exposure and of how far its enumeration can be automated.

0.31

Evolution of Deprecated APIs and Their Replacements in Python Libraries

cs.SE ·2026-09-26 ·Gangqiang He, Guanping Xiao
similarity 0.54
Abstract · 18 pages, 11 figures

Context: API deprecation is common in library evolution, but migration requires finding replacements and adapting calls. Prior studies examine deprecation, replacement identification, and client migration separately, leaving deprecated API-replacement API relationships across releases and lifecycle states insufficiently understood. Objective: We examine replacement locality, parameter-interface differences, changes in replacement rankings across versions, and post-deprecation lifecycles from source-definition and original-invocation perspectives. Method: We construct 830 maintainer-specified mappings from 33 Python libraries, covering classes, functions, and methods. We compare definitions, track similarity-based rankings under two migration scenarios, and identify lifecycle events through source inspection and release-by-release execution. Results: Same-module replacements account for 39.1%, 55.6%, and 79.9% of class, function, and method mappings, respectively. Most mappings have no parameter changes, but function mappings show more frequent and diverse changes. Rankings vary across releases and code representations, and the applicable methods agree on the overall trend in only about 60% of cases. Other-candidate effects are dominant or mixed in 49.0% of key versions. After deprecation, 56.4% of mappings retain the source definition and 68.6% retain an executable original invocation. Among 753 mappings with at least one observed ending event, 25.2% are source-first or invocation-first rather than same-release. Of these cases, 77.4% are source-first. Conclusions: Deprecated API-replacement API relationships depend on API granularity, release context, code representation, candidate competition, and lifecycle state. These results support evolution-aware replacement API recommendation and automated API migration.

0.24

PatchRisk: Forecasting Future Vulnerability Exposure in Open-Source Dependency Networks

cs.CR cs.SE ·2026-09-12 ·Ashfaq Ali Shafin, Khandaker Mamun Ahmed
benchmarkcode/data similarity 0.52
Abstract · 8 pages

Open-source software ecosystems are web-scale dependency networks. A downstream package can become exposed to security risk not because its own source code changes, but because one of its transitive dependencies later receives a vulnerability advisory. Existing vulnerability-detection work often focuses on whether code is currently vulnerable or whether a known vulnerable dependency is already present. We study a different problem: future transitive vulnerability exposure. Given a package-version dependency graph observed at release time, the task is to predict whether any non-root dependency will receive a vulnerability advisory within a future horizon. This problem is important for Web intelligence and software supply-chain security because it supports proactive dependency triage before future exposure is visible. It is also easy to evaluate incorrectly: current vulnerable dependencies can leak the label, and different versions of the same root package can create package-level memorization across train and test splits. We therefore construct PatchRisk, a leakage-aware benchmark from Open Source Vulnerability advisories and deps.dev dependency graphs for npm and PyPI. The benchmark uses filtration-aware labels, package-disjoint evaluation, temporal testing, and nested 1K, 3K, 5K, and 10K sampling scales. The largest cleaned setting contains 9,007 root package-version graphs spanning 4,157 root packages. We evaluate three feature families: TimeOnly, GraphStruct, and HistoryGraph. On the 10K temporal-test benchmark, HistoryGraph improves AUPRC over the strongest TimeOnly baseline from 0.351 to 0.640 for 90-day forecasting and from 0.473 to 0.813 for 365-day forecasting. The improvement remains stable across smaller scales and package-group shuffle robustness tests.

0.32

SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

cs.CL cs.AIcs.SE ·2026-08-24 ·Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao +5 more
benchmarkhighly cited similarity 0.51
Abstract

Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs ($5.4\%$) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores $47.0/100$. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, $58\%$ reach $99\%$ of the fixed checks, yet only $26\%$ reach $100\%$. Agent capability differs across migration categories: agents score $31.4$ on build toolchain rewrites but only $5.6$ on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.

0.32

On the Risks of using LLM-Generated Tests for Regression Testing

cs.SE ·2026-10-08 ·Mohammadali Charoosaei, Cedric Richter, Mike Papadakis
similarity 0.50
Abstract

Software is under constant evolution: developers continuously add features, fix bugs, and refactor code, and any of these changes may break existing functionality. Regression testing guards against such effects by capturing expected behavior in test cases. LLM-based test generation aims to automate this process by generating regression tests directly from the code under test. This is beneficial when the implementation is correct, but problematic when the code contains faults: the generated tests may then encode and preserve incorrect behavior. To investigate this risk, we apply LLM-based regression test generation to pull requests merged into the main branch of software projects and study the impact of the generated tests on subsequent project evolution. We distinguish between fault-revealing tests, which assert correctly implemented behavior, and fault-enforcing tests, which assert faulty behavior. Across 145 pull requests from SciPy, Qiskit, and pandas, 8%-17% of the generated tests are fault-enforcing, while only 2.4%-4.8% reveal faults. Fault-enforcing tests persist over time: after several subsequent commits, 83%-91% of them are still relevant and pass. They also accumulate: when the faults of all pull requests are combined in one codebase, 83%-92% remain enforced at the end of the commit history, and the developer-written test suite detects only 14%-30% of them. Our results reveal a fundamental risk of LLM-generated regression tests: without manual validation, they may encode faulty behavior as expected behavior, allowing bugs to persist across software revisions and largely evade developer-maintained test suites. LLM-based regression testing can thus give rise to a new form of technical debt.

0.24

Fixed Detectors, Moving Code: How Well Do Static Security Analyzers Hold Up over a Decade of C/C++ Evolution?

cs.CR cs.SE ·2026-11-02 ·Junhua Tang, Zhengsong Zhang. Haipeng Cai
code/data similarity 0.50
Abstract

Static application security testing (SAST) tools are often deployed on the assumption that detectors configured today will remain effective on future code, including previously unknown vulnerabilities. We test this assumption using 20,000 vulnerable and fixed C/C++ cases from 1,116 open-source projects (2016-2025), evaluated with three CodeQL releases, three Infer releases, and one Semgrep configuration. To distinguish temporal effects from shifts in code composition, we compare yearly populations, project- and label-matched year pairs, and within-project trajectories. We investigate the largest within-project declines through repair diffs, detection predicates, alert locations, and execution logs, while accounting for failed, degraded, and costly analyses. Under changed-file case-label scoring, overall F1 ranges from 0.066 to 0.478, but count-based AUROC stays near chance (0.495-0.526). Newer CodeQL releases attain higher F1 alongside nearly parallel rises in recall and false-positive rate, not stronger separation of vulnerable from fixed cases. In project-matched year pairs, CodeQL R2024 has negative mean later-minus-earlier F1 at every source-year gap, whereas Infer v1.1 and v1.3 have positive means throughout. Among selected drops, only 6 of 39 CodeQL and 2 of 620 Infer findings overlap a changed repair line. For Semgrep, 78.96% of cases show parser or timeout diagnostics even though every run reports success. These results suggest that apparent changes in temporal SAST effectiveness can arise from broader alerting, changing analysis coverage, and incidental findings, not solely from losses in vulnerability detection. Evaluations of temporal sustainability should fix detector conditions, control for projects, inspect repair-level evidence, and report failed or degraded analyses.

0.34