CS Paper Radar

Complexity concentration as a god-class and god-method detector: a pre-registered test of a scale-free outlier rule against tool definitions and human labels

Daniel Sagenschneider

cs.SE ·submitted 2026-10-23 10:33 UTC · v1
unusual

Abstract

God-class and god-method detectors mark concentrated complexity with a raw-scale threshold, a constant fitted on a corpus and re-fitted when the population moves. This paper pre-registers a test of a rule with no raw-scale constant: a unit is a god unit when its complexity is a robust outlier for its own codebase, a modified z-score of 3.5 or more. On 445 repositories in ten languages, 171 of them Java, and the 492 Java repositories of MLCQ, the rule is scored against PMD's GodClass, MLCQ's reviewers and two further definitions, with signal and cutoff chosen on half the repositories and read on the rest. LLM-written code may gather its complexity differently, the case for a detector that needs no re-fitting; the first question is whether one can match the scale of existing human-written code. The verdict is inconclusive on both populations, by different arms. The rule ranks PMD's god classes from the per-file maximum of cognitive complexity at per-repository AUCs of 0.916 and 0.908. On the main run it is, at its default, non-inferior within the margin to a cutoff tuned on the same population on recovery, and flags about twice as many non-god classes; on MLCQ it flags about as many and falls short of the margin on pooled recovery. Pooled, the raw value ranks god classes better than the per-repository score, and a cutoff carried between the populations lost almost nothing: both are human-written code on one scale, so the test is of matching that scale, not of surviving a shift. The rule's extra flags are small classes, just over half holding a method at PMD's own method-level threshold; its misses are large classes whose complexity is spread over many methods. Two readers of the source did not agree on what the extra flags are. Concentration finds where the complexity is; whether that place is a god unit is a question about coupling, which a registered follow-on study takes up.

Comment
16 pages, 1 figure, 6 tables. Pre-registered study; ancillary files contain the run data, analysis outputs and readings. Archived at doi:10.5281/zenodo.23581333

Closest papers

Nearest neighbours by meaning across the whole index.

What reviewers call a Long Method is a length: A pre-registered test of coupling divergence as a god-unit signal, and what it settled instead

cs.SE ·2026-11-01 ·Daniel Sagenschneider
similarity 0.68
Abstract · 6 pages, 1 table. Ancillary files: the full registered report and the analysis data. Also deposited at Zenodo, doi:10.5281/zenodo.23749773

A companion study found that a complexity outlier rule shows where complexity concentrates but cannot tell a god unit from a complex algorithm. This paper pre-registered a test of one way to tell them apart: weight a unit's complexity by its coupling divergence, the data it hands on minus the data it receives, so that a leaf is set aside. Coupling was measured by resolving every call in 520 MLCQ repositories, 500 fresh Java repositories and 73 Qualitas Corpus systems. The claim fails: the primary, the pooled MCC of the weighted form minus complexity alone against MLCQ's human Blob, is -0.041 [-0.130, +0.054], no difference resolved and underpowered, and the resolved readings go against it. What the data settle is at method level. What MLCQ's reviewers call a Long Method is an absolute length: a constant of 26 lines or more, chosen on the explore half, reaches MCC 0.777 on the confirm half, +0.080 [+0.016, +0.145] over the best per-repository form; every reviewer with ten or more positives, on their non-leaf reviews, has a best cutoff between 17 and 34 lines; and nothing measured inside a method, its calls, variables, data flow or the cost of cutting it, refines the constant. The constant is a limit of human reading. The next form of the construct is the cost of bringing a long method under it, which is where the construct meets architecture.

0.44

Beyond Coverage and Mutation: A Blind-Spot Analysis of LLM-Generated Unit Tests

cs.SE ·2026-10-23 ·Xiaoyu Shi, Wenqiang Luo, Boyang Yang, Haoye Tian
benchmark similarity 0.53
Abstract

Existing evaluation protocols for unit tests rely on aggregate metrics such as branch coverage and mutation score. These metrics quantify the extent to which a test suite exercises program behavior, but do not identify the behavioral obligations it systematically leaves unchecked. We propose BlindSpotEval, an obligation-based evaluation framework aligned with the Arrange-Act-Assert pattern. The framework comprises three metrics, Boundary-Value Adequacy, Control-Flow Adequacy, and Oracle Strength, complemented by an assessment of execution reliability. We also introduce ObligBench, a complexity-stratified Java benchmark comprising 895 complex and 700 low-complexity functions with source-side obligation annotations. Evaluating 14,320 tests from four LLMs under four prompting strategies reveals systematic blind spots. Incorrect assertions account for 15.6% of test failures, illustrating that coverage alone does not establish oracle correctness. Among executable tests, coverage of null, negative, and empty arguments reaches only 34.8%, 23.8%, and 38.8%, compared with average coverage above 93% for booleans. Catch blocks and default groups have Control-Flow Adequacy scores of only 0.100 and 0.312. Across both dimensions, LLM-generated tests exhibit the happy-path bias observed in human-written tests, concentrating on normal inputs and expected flows while leaving unusual values and exceptional paths unchecked. Oracle strength is high on average but uneven, with weaker assertions for object return property values and exceptions. These blind spots also occur in low-complexity functions, suggesting that code complexity alone does not explain them. BlindSpotEval provides construct-level adequacy profiles that identify under-tested obligation types and can guide targeted improvements beyond those supported by aggregate scores alone.

0.12

Code Detectors Have a Half-Life: Obsolescence and Metric Illusions in LLM-Generated Code Detection

cs.SE ·2026-10-01 ·Alberick Euraste Djire
top venue similarity 0.53
Abstract

Code detectors can become obsolete as code-generating models evolve: a detector validated on one generation of models may not transfer to the next. We call this limited useful life a detector half-life. We evaluate eight general-purpose LLM judges and three dedicated detectors on human-written code and code produced by seven generators across C++, Java, and Python. Our results reveal two problems. First, performance varies considerably across generators and prompting strategies, suggesting that some detectors rely on generator-specific patterns rather than general evidence of code provenance. Second, accuracy can conceal severe prediction bias. DetectCodeGPT and GPT-Sniffer achieved an accuracy of 0.50 but an F1 score of 0.00 across all generators because they classified almost every sample as AI-generated. However, general-purpose LLM judges achieved stronger accuracy and F1 scores. Our results show that general-purpose LLMs are promising training-free judges of code provenance and can outperform dedicated detectors. However, their reliability depends on the judge model, the code generator, and the prompting strategy. We therefore recommend evaluating LLM judges across multiple generators and reporting macro-F1 alongside class-specific precision and recall.

0.53

OEIS Open: How many conjectures can language models turn into theorems?

cs.AI ·2026-08-12 ·Tom Adamczewski
benchmarkcode/datahighly cited similarity 0.50
Abstract · 26 pages, 6 figures. Code: https://github.com/epoch-research/LeanOpenProblems, results: https://github.com/epoch-research/LeanOpenProblems-r…

We construct OEIS Open, a benchmark based on 492 open mathematical conjectures from the OEIS, formalized in Lean by Tsoukalas et al. Whereas these conjectures had previously been attempted only with a bespoke agent, our open-source evaluation code runs any generic language model (LM) against them, and is secure against LM cheating attempts. We find that LMs equipped with a minimal set of tools resolve 147 of these conjectures with a budget of \$50 per attempt, scoring 30% on OEIS Open. OEIS Open Lite is a random subset of 100 conjectures for cheaper evaluation. When evaluated with a budget of \$200 per attempt, the best current LM scores 44% on OEIS Open Lite. Giving LMs access to the mathematics literature via 476,000 papers from arXiv did not increase performance on OEIS Open Lite, and nor did using more sophisticated agent loops. The conjectures covered in this work are of uncertain mathematical significance, and most have likely received little previous attention. Nevertheless, our results show that LMs can resolve open research conjectures autonomously and at modest cost.

0.68

What extraction cannot slim, and whether function injection can: a pre-registered study of 580 Java repositories

cs.SE ·2026-11-07 ·Daniel Sagenschneider
publishedcode/dataunusual similarity 0.50
Abstract · 19 pages, 1 figure, 10 tables. Pre-registered; every number regenerated by script. Replication package in the ancillary files and at https:/…

A method of 26 code lines or more reads as a Long Method, and Extract Method is the usual fix. Some long methods have no single-exit fix: a return cannot leave an extracted piece, a piece hands back one value, and some methods are knotted so that every cut is blocked. Function injection, as the OfficeFloor framework does it, cuts differently: each piece becomes a function the framework injects and sequences, an early return ends a function, and a function hands on several values. The framework's author pre-registered a test of whether it takes apart what extraction cannot. A stratified random sample of 580 open-source Java repositories, each pinned before AI coding assistants were in general use, was scanned by released tools whose cost models were fixed before the sample was drawn: 3,026,902 methods, 168,466 of them long. Single-exit extraction cannot slim 22.8% of the long methods. The registered hypothesis, that at least a quarter of those stuck methods are ones only function injection takes apart, is refuted: 22.2% (95% CI 20.3 to 24.3). Nearly half of the stuck methods are early-return methods the plan counts as conventionally fixable, and the split takes three in four of those apart as they stand. Where extraction would do, the split chooses injection and costs less for 29.0% against extraction wrapped as one function and 17.2% against plain extraction, almost always as a straight sequence of functions. An exploratory reading of the quarter of the stuck methods that neither takes apart finds a third of it tables and literals, long for their data rather than their logic, and most of the rest blocked by a jump inside a loop, by a nested statement too large to cut, or by values too many for any cut. Three registered studies are proposed: the classes as rules in the tools, a refinement per class, and the same measures on code written with AI assistants.

0.56

Detection Is Not Enough: Reasoning Traces as Testable Program Hypotheses

cs.SE cs.AI ·2026-10-21 ·Vaidehi Hegde
code/data similarity 0.50
Abstract · 17 pages, 8 figures, 9 tables. Code and stored outputs: https://github.com/S2V3/Detection-is-not-enough

When a language model is deployed, no reference solution exists, so the reasoning trace it writes is the only evidence for judging its answer. The reference-free substitutes each fail differently: self-assessment degrades the answer it was meant to protect, sampling agreement measures stability rather than correctness, and perturbation studies read the expected answer off a human-written template, grading a model rather than a trace. We ask how far a single trace can be verified with no reference solution, no judge model and no weight update, and find, for a frozen LLaMA-2-7B-chat on GSM8K, that detection is not what limits such a pipeline. Checking a trace against itself reaches an AUROC of 0.741 and stops there for a structural reason: two thirds of wrong traces state no false equation, having executed a mistaken plan correctly, and on wrong-plan failures the detector flags a wrong trace less often than a right one. A candidate-derived metamorphic probe exposes many of these failures: we abstract the trace into a program over the question's quantities, perturb one quantity, execute the program, and ask the model the rewritten question once, so every trace supplies its own test. Valid perturbations exist for 937 of 1,319 test questions (71.0%); among the 847 that also have a parseable first trace, supported traces are correct 60.6% of the time against a 31.6% base rate, although the probe also refutes 45.5% of correct traces. No tested policy yields a statistically significant improvement in accuracy. A flag earns its retry only when detector precision exceeds D/(R+D), where R and D are the rates at which a retry repairs a wrong trace and damages a correct one; our repair rate is so low that, at the same retry budget, even a perfect detector would recover an estimated 66 of 1,319 questions. Repair, not detection, is the open problem.

0.31

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

cs.AI cs.SE ·2026-09-10 ·Qibai Chen, Zeming Liu
publishedbenchmark similarity 0.50
Abstract · Accepted at the 38th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2026). 8 pages, 6 tables

Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.

0.51

PerfArena: Ranking LLMs by Code Performance

cs.PF cs.CLcs.SE ·2026-11-13 ·Rui Rua, Hailemariam Mersha, Joao Paulo Fernandes, Karim Ali
code/data similarity 0.50
Abstract

Industry now relies on LLMs to write a large share of production code. Although that code is mostly, but not always, correct, correctness alone does not determine whether it is fit for repeated and extensive use, which depends on what the code costs in time, memory and energy. Most code-generation benchmarks nonetheless evaluate correctness in isolation, and most performance-oriented benchmarks cover few languages and rarely measure energy consumption. To address this gap, we present PERFARENA, an open-source, reproducible arena that ranks LLMs by the performance of their generated code across multiple languages and competitions. PERFARENA executes the solutions of each LLM on a Unix-like host and reports correctness together with latency, peak memory, and energy consumption. Correctness oracles, performance oracles, profilers, and code-generation workflows are pluggable in PERFARENA, enabling the integration of new judging backends and measurement tools without changing the pipeline. We evaluate PERFARENA on two competitions: CLBGComp, 10 compute-intensive benchmarks from the Computer Language Benchmarks Game, and LeetCodeComp, 99 LeetCode problems whose accepted solutions differ measurably in latency. Currently, PERFARENA holds 9,455 correct and measured solutions from 13 state-of-the-art LLMs across 7 programming languages. Across both competitions, got-6-astra is the only LLM that no other LLM beats significantly in latency, energy consumption, or peak memory, solving 99.7% of LeetCodeComp and 100% of CLBGComp. All PERFARENA leaderboards and measured solutions are publicly available, providing a continuously updated reference that helps developers choose an LLM per language and metric, instead of relying only on correctness rates or pricing.

0.34