CS Paper Radar

What extraction cannot slim, and whether function injection can: a pre-registered study of 580 Java repositories

Daniel Sagenschneider

cs.SE ·submitted 2026-11-07 07:29 UTC · v1
publishedcode/dataunusual

Abstract

A method of 26 code lines or more reads as a Long Method, and Extract Method is the usual fix. Some long methods have no single-exit fix: a return cannot leave an extracted piece, a piece hands back one value, and some methods are knotted so that every cut is blocked. Function injection, as the OfficeFloor framework does it, cuts differently: each piece becomes a function the framework injects and sequences, an early return ends a function, and a function hands on several values. The framework's author pre-registered a test of whether it takes apart what extraction cannot. A stratified random sample of 580 open-source Java repositories, each pinned before AI coding assistants were in general use, was scanned by released tools whose cost models were fixed before the sample was drawn: 3,026,902 methods, 168,466 of them long. Single-exit extraction cannot slim 22.8% of the long methods. The registered hypothesis, that at least a quarter of those stuck methods are ones only function injection takes apart, is refuted: 22.2% (95% CI 20.3 to 24.3). Nearly half of the stuck methods are early-return methods the plan counts as conventionally fixable, and the split takes three in four of those apart as they stand. Where extraction would do, the split chooses injection and costs less for 29.0% against extraction wrapped as one function and 17.2% against plain extraction, almost always as a straight sequence of functions. An exploratory reading of the quarter of the stuck methods that neither takes apart finds a third of it tables and literals, long for their data rather than their logic, and most of the rest blocked by a jump inside a loop, by a nested statement too large to cut, or by values too many for any cut. Three registered studies are proposed: the classes as rules in the tools, a refinement per class, and the same measures on code written with AI assistants.

Comment
19 pages, 1 figure, 10 tables. Pre-registered; every number regenerated by script. Replication package in the ancillary files and at https://github.com/officefloor/long-method-slim-study
DOI
10.5281/zenodo.23991507

Closest papers

Nearest neighbours by meaning across the whole index.

Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction

cs.CL cs.AI ·2026-08-28 ·Qing Ye, Meng-Hsuan Lin
top venuebenchmark similarity 0.52
Abstract · Accepted at EMNLP 2026 Industry Track. 7 pages + appendices

One model passed our fidelity check without ever opening the datasheet. We found it while qualifying models for an internal extraction service: a structured-output constraint had silently disabled tool use, and the model answered anyway, with fabricated source text. Only the per-tool trace exposed it. Fidelity -- whether an extracted value matches the source -- is the standard measure for agentic document extraction, and it scores that run a success. We therefore log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all. From that dispatch record we build two instruments: a rule-based failure-attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value. The detector raises no flag on 207 clean fidelity-passing extractions across three model families, and recovers all 50 planted faults that withhold exactly the tools its rules check. The two results are not symmetric: the first bounds the false-positive rate, the second is recall by construction, and detection power against runs that call their tools and still answer wrongly is unmeasured. A second, independent oracle, a causal chamber that tests whether the datasheet's claims hold under physical measurement, is intentionally partial: it confirms only what the apparatus can exercise, a verifiable envelope of 2 of those 37 claims, and we give a taxonomy of why the rest are not physically gradable. Under a controlled perturbation, fidelity passes throughout while the chamber verdict flips exactly at the measurement uncertainty. Across three deployed model stacks (one destabilised by its serving stack, not by any capability gap) the tool layer buys portability and observability rather than accuracy, and earns its premium only once a document outgrows the context window.

0.37

Pattern-Based Precondition Inference from Method Implementations

cs.SE ·2026-10-16 ·Shanyue Zhou, Martin P. Robillard
code/datahot direction similarity 0.52
Abstract

Method preconditions define the conditions that must be satisfied before a method is called. Often, such conditions are not documented and must be inferred. We present Prex, a pattern-based static analysis approach that infers preconditions from explicit checks in Java source code. For each precondition, Prex reports the condition required for a valid call and the reported exception in the case of a violation. Prex also attempts code transformations to convert the condition detected within a method's implementation into an expression that can be evaluated by client code. It leverages the concept of a mutation boundary to separate precondition checks from exceptions that may occur after the method changes the program state. We evaluated Prex from 46 open source Java projects and manually reviewed independent samples of 150 reports. Among the 150 Prex reports, 149 were correct and 125 of the correct reports were executable by client code. In a systematic comparison with an alternative approach based on path expansion, Prex provided more precise results. In total, 122 of the 149 correct reports had no relevant method documentation, and 41 required nontrivial source code analysis to recover manually. Overall, explicit source-code checks provide precondition information that can be absent from method documentation, and Prex can help recover it.

0.35

A Memorization Floor for LLM Refinement of Decompiled Code

cs.SE ·2026-09-15 ·Muhammad Asjad
similarity 0.51
Abstract · 29 pages. Pre-registered; analysis plan committed before data collection. Replication package: https://doi.org/10.5281/zenodo.21968878

We introduce a memorization floor: a within-item control separating what LLM refinement of decompiler output recovers from its input from what it recovers from its prior. Refine a function, then refine it again from an input whose identifiers have been destroyed, and measure what survives. Because the comparison is within-item, corpus difficulty cannot contribute; it costs twenty API calls. Applied to functions written after our analysis plan was committed, so no released model could have memorized them, it reports two things. Recovery is real: refined output sits +0.072 to +0.137 above an arm-matched permutation null built from its own output vocabulary. But it does not depend on the input we ablate: destroying the input's dataflow changes the naming gain by +0.001 (95% CI [-0.026, +0.026]), and removing type prefixes or permuting names changes it by no more. A second refiner from another vendor, registered in advance and given byte-identical inputs, reproduces this -- twelve contrasts, two models, twelve nulls. Readability stays at ceiling throughout, so a reader is given no signal. The null is bounded, not absolute: contributions under 0.056 are invisible, and the ablation leaves operations intact, so naming from those alone remains a competing reading. No registered hypothesis was confirmed, and we report the five instrument failures behind that in full, including a reassembly harness biased against the treated arm and an equivalence checker we registered without checking it worked on our inputs.

0.26

When LLM Decompilers Recompile More and Preserve Less

cs.CR cs.AI ·2026-09-04 ·Chang Liu, Edward Raff, Kristopher Micinski
similarity 0.51
Abstract

Decompilation recovers high-level source from compiled machine code and serves as a foundation for security tasks such as vulnerability detection and malware analysis. Traditional decompilers like Ghidra and Hex-Rays expose whatever they cannot resolve as visible placeholders and often emit pseudocode that will not compile or execute; LLM-based decompilers produce clean, idiomatic C and are now judged almost entirely by recompilability and re-executability: whether the output builds and passes its shipped input/output tests. We show that these metrics can reward the wrong path: a function may recompile and pass every shipped test yet diverge on other legitimate inputs, and a disclosed vulnerability may disappear from the recompiled code with no visible trace of the crash. Neither failure is caught by existing suites. To address this gap, we propose Decompile-Diverge, a behavioral comparison oracle not relying on fixed or hand-crafted tests: for each function it synthesizes a driver, grows a fuzzing corpus from the reference, and reruns the decompiled code on the same inputs to detect changes in the function's behavior. Across eight systems in nine configurations on established LLM decompilation corpora, candidates that pass every shipped test still diverge from the original on our input corpus: 4.9% overall, and as many as 13% for a single system. On 300 real GitHub library functions and 287 CVE-grounded functions, recompilability and behavioral agreement can come apart: the strongest refinement LLM lifts Ghidra's build rate from 75% to 90%, while its Matched rate falls from 74% to 62%; on disclosed vulnerabilities, up to one tenth exhibit Crash Absence in its output. Source-level analysis traces this divergence to introduced fields, types, callees, and guards that replace the visible unknowns traditional tools leave behind.

0.38

From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents

cs.SE cs.AI ·2026-09-28 ·Tianyu Liu, Dingyuan Dai, Yufan Du, Zhen Yang
benchmark similarity 0.50
Abstract · 14 pages, 3 figures

Can coding agents restore software that no longer runs while preserving its underlying methods, and reconstruct industrial software engines from open specifications? Here we introduce ReviveBench, a benchmark with two task families evaluated by hidden verifiers calibrated against native execution environments, established engineering tools, or purpose-built reference implementations. The revival family comprises ten tasks involving dependency incompatibilities, deleted core modules, legacy builds, and a GPU-based foundation model. Every starting workspace fails verification, and the strongest evaluated model passes all ten tasks in at least one run each. In contamination-control experiments, identifier obfuscation reduces line similarity to the original implementations from 0.51--0.96 to 0.03--0.44 without reducing the observed pass rate of any evaluated model. On repositories created after the stated knowledge cutoffs, the strongest model passes eight of nine runs. The reconstruction family comprises thirteen tasks spanning numerical, geometric, hardware, and transactional systems (e.g. CAD and CRM). Two models meet the benchmark's pass criteria on all thirteen, although our audit shows that the CFD task cannot establish numerical-solver capability. Benchmark construction and auditing uncover 28 verifier defects, including 24 false negatives and two false positives. These findings show that executable verification can itself introduce substantial measurement error. We present three practical checks: test whether prescribed methods can reach the grading thresholds, investigate agreement among independently generated candidates, and recompute diagnostics from submitted artifacts. ReviveBench thus provides both an evaluation of software revival and engine reconstruction, and cases in validating the verifiers used to measure coding agents for software design.

0.29

Complexity concentration as a god-class and god-method detector: a pre-registered test of a scale-free outlier rule against tool definitions and human labels

cs.SE ·2026-10-23 ·Daniel Sagenschneider
unusual similarity 0.50
Abstract · 16 pages, 1 figure, 6 tables. Pre-registered study; ancillary files contain the run data, analysis outputs and readings. Archived at doi:10.…

God-class and god-method detectors mark concentrated complexity with a raw-scale threshold, a constant fitted on a corpus and re-fitted when the population moves. This paper pre-registers a test of a rule with no raw-scale constant: a unit is a god unit when its complexity is a robust outlier for its own codebase, a modified z-score of 3.5 or more. On 445 repositories in ten languages, 171 of them Java, and the 492 Java repositories of MLCQ, the rule is scored against PMD's GodClass, MLCQ's reviewers and two further definitions, with signal and cutoff chosen on half the repositories and read on the rest. LLM-written code may gather its complexity differently, the case for a detector that needs no re-fitting; the first question is whether one can match the scale of existing human-written code. The verdict is inconclusive on both populations, by different arms. The rule ranks PMD's god classes from the per-file maximum of cognitive complexity at per-repository AUCs of 0.916 and 0.908. On the main run it is, at its default, non-inferior within the margin to a cutoff tuned on the same population on recovery, and flags about twice as many non-god classes; on MLCQ it flags about as many and falls short of the margin on pooled recovery. Pooled, the raw value ranks god classes better than the per-repository score, and a cutoff carried between the populations lost almost nothing: both are human-written code on one scale, so the test is of matching that scale, not of surviving a shift. The rule's extra flags are small classes, just over half holding a method at PMD's own method-level threshold; its misses are large classes whose complexity is spread over many methods. Two readers of the source did not agree on what the extra flags are. Concentration finds where the complexity is; whether that place is a god unit is a question about coupling, which a registered follow-on study takes up.

0.53

Pinning Decisions Before Failure: Executable Records of Underspecified Choices in AI-Assisted Code Generation

cs.SE ·2026-10-02 ·Takeharu Mitsui
benchmark similarity 0.50
Abstract · 24 pages

A natural-language requirement leaves questions open, and a model asked to implement it settles them silently: across 600 generated test suites from three models, 42.7% contain no test that distinguishes the competing readings. An acceptance example written before implementing is the usual remedy, but an example both readings satisfy resolves nothing. We insert two steps into that practice: enumerate the requirement's underspecified points by name, then for each construct two throwaway implementations differing only in that point and keep a candidate input only if executing both shows they disagree. The result is recorded as a decision pin: the named point, the confirmed input, and the value the person chose between the two exhibited results. The same record then constrains generation and decides compliance by execution. On a benchmark of 40 tasks with paired reference implementations and two models, a separating input is obtained for 92.5% of decision points and a pin identifying the intended decision for 85-90%. As checks on 703 independently generated implementations, pins agree with the benchmark's classification on 96-97%, with disagreements on three tasks, one where both classifiers erred. As generation constraints, pins are honoured at the pinned input in all 210 generations and are never worse than a prose rule on held-out inputs in 38 cells, though no better than prose stating the same scope. Every compliance failure under a prose rule came from the model deciding the rule's scope itself; one such case silently overturned another recorded decision, was attributed to a single rule by leave-one-out on both models, and was missed by text-level reconciliation but caught by re-running the recorded input. The setting yields too few such conflicts to evaluate a regression step, and we say why.

0.30

ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation

cs.SE cs.LGcs.PL ·2026-10-20 ·Akarsh Ashok, Dinil Mon Divakaran, Vivek Balachandran
similarity 0.50
Abstract

ExeBench is one of the most widely used benchmarks for evaluating LLM-based decompilers. We analyze ExeBench and identify nine issues that affect how faithfully a test measures a decompiler. Three of them prevent LLM decompilers from recovering C code from ExeBench's function. These are ambiguous parameter order when integer and floating-point types are mixed, unused parameters that optimized assembly may drop, and constants stored outside the function's assembly code. Six more affect test validity: non-void functions without a return, placeholder types, static or inline functions removed by the optimizer, functions named main, trivial bodies and truncated definitions. We present ExeBench-FHD, a backward-compatible revision of ExeBench that adds 16 new fields: call-site assembly, data sections, test wrappers, linked-binary disassembly and five warning flags. In ExeBench, 38.9% of functions have at least one of the five flags; dropping records with any of the four warning flags (all except unused_param, which marks functions that the new wrappers make evaluable) keeps 63.9% of the corpus. In an evaluation at -O2 on four architectures, the new fields raise GPT 6 Sol's re-execution rate by up to 26.5%, 19.1% and 21.0% on functions that read data sections, mix parameter types and have unused parameters, respectively.

0.18