CS Paper Radar

When Static Reachability Says No: Checking Joern-Based Negative Verdicts for Python Against Execution

Artur Zięba-Kozarzewski, Bartosz Grzegorzewski

cs.SE cs.CR ·submitted 2026-10-29 07:05 UTC · v1
benchmark

Abstract

A negative reachability verdict removes a finding from human review. We check such verdicts against execution. Our pipeline, SourceLens, maps Semgrep findings to a Python program's call graph, whose edges a query takes from the calls that Joern links in its code property graph. We ran two command-line programs from the PyCG macro-benchmark, in which the benchmark's manual call graph and Joern disagree, under coverage and recorded the calls between their functions. On these edges, 123 functions of asciinema are unreachable from the program's entry points. The project's own integration script executes 86 of them, and all our runs 113. In Sublist3r, 19 of 24 are executed. The latest Joern release gives the same verdicts. The missing calls go through values, such as classes and functions passed around, or have no call expression, as in with blocks and property reads. In asciinema, four missing edges make 50 of the 113 executed functions reachable. The manual call graphs also lack observed calls, in asciinema many of them property reads and with blocks, which the benchmark may not count as calls. A rule that turns negative verdicts into unknown recovers 62 of the 113 and 15 of the 19, mostly through dunder and private method names. It recovers 0 of the 5 security findings with a negative verdict, all in executed functions. Its selectivity cannot be measured, as only 10 and 5 negative verdicts were not executed, and under library entry-point models it degrades every negative verdict.

Comment
8 pages, 5 tables. Artifact: https://doi.org/10.5281/zenodo.23554742

Closest papers

Nearest neighbours by meaning across the whole index.

APG: Agentic Static-Analysis Infrastructure on LLVM-IR Graph Databases

cs.SE ·2026-11-02 ·Zexin Zhong, Yanjun Zhang, Yulei Sui, Xiao Cheng
benchmarkcode/data similarity 0.60
Abstract

LLM-based coding agents increasingly rely on external program knowledge, yet access to static-analysis graphs alone does not ensure appropriate operation selection or evidence interpretation. We present APG, an infrastructure for semantics-grounded static program analysis. Built on SVF and LLVM, it decomposes compiler-derived program analyses into atomic capabilities that LLM agents can select, compose, and orchestrate for goal-directed, cross-graph investigation, with query safeguards and evidence traces preserving inspectable results and provenance. APG provides 38 atomic capabilities. On analysis requests arising during benchmark repair runs, 35 achieve first-attempt query-generation success rates above 50%, including 13 at 100%. Evaluation on 812 C/C++ defects shows higher repair success than the evaluated baselines. Compared with Passive Graph Push, APG improves first-iteration repair success by 17.6 and 18.0 percentage points on the two benchmark collections and reduces average LLM input-token consumption by 90.8%. Our tool is available at https://github.com/JasonZhongZexin/APG-ICSE-Demo and its demonstration video at https://youtu.be/vkuExZUjGnw

0.35

Wake 5: A Python Framework for Solidity Testing, Fuzzing, and Static Analysis

cs.SE ·2026-11-06 ·Josef Gattermayer, Michal Převrátil
benchmarkcode/data similarity 0.58
Abstract

Smart-contract fuzzers differ by an order of magnitude in throughput and by design in how they construct inputs, yet published comparisons rarely separate the two. We present Wake 5, an open-source Python framework for Solidity testing, fuzzing and static analysis, and use it to locate the binding constraint of stateful smart-contract fuzzing. Wake 5 replaces the JSON-RPC connection to an external node with an in-process EVM (revm) and exposes its static analysis to language-model agents through a Model Context Protocol (MCP) server. On the Daedaluzz maze benchmark (153 reachable assertions, guard constants below 2^32), Wake 5 executes 29,931 transactions per second, on par with Foundry (30,186) and 11.5x faster than Echidna (2,600). In eight-hour campaigns, however, the three tools plateau at medians of 75, 71 and 70 assertions, and their union reaches 93. We show that Wake's integer generator places 51% of its probability mass below 2^32 and that, once shallow assertions are exhausted, the generator prior rather than throughput bounds every tool. A harness written by a language-model agent, which derives inputs with a constraint solver, reached all 153 assertions in each of five campaigns executed by Wake, in 33 to 79 minutes. A complementary ablation in an agentic audit of a production Go codebase found that static-analysis tools improved recall for none of three models, raised precision for all three, and increased language-model cost by up to 12.7%. We discuss correlated oracles and non-deterministic completion, and argue that the fuzzer's role is shifting from search to execution.

0.36

Self-Evolving Static Analysis via LLM Pseudo-Labeling

cs.CR cs.AIcs.SE ·2026-10-20 ·Rui Melo, Andre Duarte, Aditya Oke, Shubham Gandhi, Nachiket Kotalwar +3 more
benchmark similarity 0.58
Abstract

Security bugs often slip past static analysis not because they are impossible to detect, but because detection rules have not yet been written for the code in front of us. New frameworks, APIs, and coding idioms continually create such gaps, leaving developers to discover failures and manually encode new rules. We present SESAA, a framework in which an Large Language Model (LLM) agent starts from no rules and iteratively evolves a set of readable and auditable joern queries. The agent receives feedback only from a second LLM critic, which inspects flagged and unflagged code to identify likely false alarms and suspected misses, without access to ground-truth labels. The agent then uses this feedback to revise its queries. On the OWASP Benchmark for Python, SESAA reaches a mean F1 of 0.742 across 8 independent runs, well above the default configurations of five widely used tools and comparable to the strongest of them once it is tuned with access to ground-truth labels that SESAA never sees. In a pilot on real-world Python projects, queries induced from code available before a vulnerability was introduced detect 10 of 19 subsequent CVEs, including 9 detected by rules written for the same repository. The resulting detector is a set of executable, human-readable queries that engineers can inspect, modify, and run independently. These results suggest that LLM agents can turn static-analysis rule maintenance from a manually authored process into an iterative process of automated discovery and repair.

0.25

Beyond Functional Correctness: Static Code Quality Profiling of LLM-Generated Code

cs.SE ·2026-10-13 ·Jue Huang, Tarek Mahmud, Meiru Che, Guowei Yang
publishedbenchmark similarity 0.56
Abstract · Accepted for publication in ICECCS2026

Large language models (LLMs) are increasingly used for code generation, yet most evaluations still focus on whether generated code passes benchmark test suites. Functional correctness alone, however, provides only a limited view of quality: benchmark tests are often incomplete, and passing them does not guarantee that code is maintainable, reliable, or well-structured. This paper {presents a large-scale static quality profiling of} LLM-generated Python code, with a focus on maintainability and reliability. Using 65{,}663 programs from 73 models in LiveCodeBench, we evaluate generated code with established static analysis tools, including Pylint, Radon, and SonarQube. Our results show systematic differences across models that are not captured by correctness alone, with maintainability emerging as the clearest source of variation. We further find that correctness often comes with structural trade-offs rather than uniformly simpler solutions. Overall, our findings motivate evaluation frameworks that assess LLM-generated code beyond pass rates alone.

0.25

AegisAsync: Symbolic Async-Event Simulation for Risks in Python Asynchronous Programs

cs.CR ·2026-10-13 ·Baihong Chen, Wen Li
benchmark similarity 0.56
Abstract

Security and robustness risks in Python async programs depend on event order, object identity, and execution after exceptions or cancellation. Detecting these risks requires connecting related operations across function calls and tracking their effects along execution paths. We present AegisAsync, a bytecode-based analysis of Python async risks for CPython 3.10-3.14. Its Async Event Transition Graph combines bytecode control flow, resolved calls, async events, and symbolic object relations. Bounded path-local simulation tracks event order and object states and explores cancellation at await suspensions. Six detector plugins evaluate the resulting traces and states to produce candidates with supporting witnesses. On AsyncBench, our benchmark of 180 labeled risks, AegisAsync detects all labeled risks with zero false positives on each of the five versions. Across five path budgets on 2,500 PyPI package versions, source review confirms 457 of 910 AegisAsync candidate records (50.2%), representing 457 unique bugs; across three flake8-async samples, 70 of 2,187 review judgments are confirmed (3.2%). By submission, 166 PoCs reproduce AegisAsync findings, and we have reported 157 bugs to developers, of which 16 have been confirmed and fixed.

0.21

CPyGraph: A Version-Aware Static Analysis Framework for Native CPython Bytecode

cs.CR ·2026-09-18 ·Baihong Chen, Wen Li
benchmark similarity 0.55
Abstract

Static analysis of Python packages must recover both program structure and object flow across first-class functions, dynamic dispatch, implicit protocol calls, exceptions, closures, and module execution. Native CPython bytecode provides the executable lowering of these behaviors, but its instruction, call, stack, and exception representations change across releases. This creates a need for a version-aware analysis foundation whose graph products share the same bytecode identities and semantics. We present CPyGraph, a C++ framework for package-level analysis of native CPython bytecode. Version- specific adapters expose stack, control, call, lexical, protocol, and exception semantics through a shared interface while preserving code-object identities and native bytecode offsets. An operand-stack-aware Andersen points- to analysis and call graph grow together to a fixed point. Their shared state supports exception-aware CFGs, block-level CDGs, and interprocedural DDGs, with optional function-level flow, context, and bounded path sensitivity. The framework also records unresolved dynamic behavior through typed coverage summaries. We evaluate CPyGraph with PYGBench, 201 package-level programs and 1,733 fixed candidates. On CPython 3.10, the default analysis reaches 91.40% candidate precision and 100% recall; complete sensitivity reaches 94.97% precision with the same recall. Across CPython 3.10-3.14, 1,328 of 1,334 version-invariant queries agree, and CPyGraph matches PyCG on its 112-program call-graph benchmark.

0.44

Bug Finding With Agentic Symbolic Execution

cs.SE cs.AIcs.LOcs.SC ·2026-11-11 ·George Granberry, Wolfgang Ahrendt, Andy Gordon, Moa Johansson
code/data similarity 0.55
Abstract

Symbolic execution, in tools such as CrossHair for Python, is a powerful technique for finding bugs, but two barriers hinder its adoption: the human effort of annotating code with assertions, and the need to model the execution environment outside Python, such as an external database. To overcome both, we design an agentic harness that orchestrates the symbolic execution tool CrossHair together with an AI coding agent. Without human intervention, it instruments a repository with assertions, synthesizes a constraint-aware symbolic environment for the database, runs symbolic execution, and constructs a concrete test as evidence for each bug it reports. We evaluate our harness on 1,156,044 lines of Python code over 11 popular open-source Python web applications (Zulip, NetBox, Apache Superset, and others). With the synthesized environment, CrossHair executes 81% of the 5,406 contracted functions, against 4.5% without it. Our pipeline finds 1,050 bugs, each reproduced by a test against the unmodified application, 221 of them by CrossHair, and 312 of them are missed by a baseline in which the same LLM inspects the code. The two find different kinds of bugs: CrossHair's are mostly crashes on input-domain corner cases that validation admits, such as empty strings, missing fields and unusual characters, while inspection's are mostly bugs that produce wrong output. Our framework thus lets symbolic execution supplement inspection-based bug finding.

0.32

IRONPROOF: COBOL-to-Python Transpilation with SMT-Based Equivalence Checking

cs.SE ·2026-10-08 ·Dominik Blain
theorycode/data similarity 0.55
Abstract · 13 pages, 6 tables, 1 figure. Code and data: https://github.com/dom-omg/ironproof-cobol

Translating COBOL to a modern language can change what a program computes, and LLM translations carry no guarantee of equivalence. We present IRONPROOF, which parses COBOL into an intermediate representation, generates Python, encodes both as Z3 formulas over shared inputs, and emits either a machine-checkable equivalence certificate (UNSAT) or a counterexample (SAT). On 2,345 COBOL files (GnuCOBOL tests, NIST CCVS85, open-source collections, and programs we generated or wrote), 782 enter the checking path: 606 (77.5%) are proved equivalent, 101 are partially verified, none is refuted, and 75 fail inside our pipeline and stay in the denominator. On independently authored programs the rate is 52.6% (153 of 291), against 92.3% on programs we wrote. Of the 153 independent proofs, 37 cover a single execution of a program that reads input the encoder does not model, and across all 606 proofs only 14 quantify over an input that a proved output depends on. On two public business-application corpora (AWS CardDemo and IBM GenApp), the encoder models no program end to end. An April 2026 LLM-only baseline failed on 54 of 94 independently authored programs, counting translations our encoder could not verify. PIC-bounded overflow detection flags an out-of-range output in 22.1% of the programs it can analyze, an upper bound. An earlier confrontation with GnuCOBOL 3.2.0 found 24 of 49 executable independent proofs disagreeing with the runtime on a variable in the proof's domain. A proof establishes that the generated Python computes what our intermediate representation says the COBOL computes, not equivalence to a COBOL implementation.

0.40