CS Paper Radar

Compilation Is Not Correctness: Scan-by-Scan Evaluation of LLM-Generated PLC Programs

Jinyun Chen, Juntao Hu, Haochen Li, Wentao Song, Ningbo Gu, Jianwei Niu, Qing Gao, Rui Wang

cs.SE ·submitted 2026-10-19 09:16 UTC · v1
benchmark

Abstract

Programmable logic controllers (PLCs) automate industrial processes through repeated scans, each reading inputs, executing control logic, and updating outputs. Engineers write this logic in Structured Text (ST), a high-level textual language defined in IEC 61131-3. Large language models (LLMs) can now generate ST from natural-language requirements, yet compilation success and checks on selected properties or sampled scans can leave behavioral errors undetected. We introduce CycleBench, a benchmark of 301 ST tasks with behavioral requirements, public interfaces, reference programs, and executable scenarios. Its runtime executes each generated program and its reference under aligned inputs, initialization, and logical time, comparing their outputs after every scan. Across eight LLMs and baseline generation methods, execution pass rate trails compilation pass rate by 11.92 percentage points; observed failures include output-update, arithmetic, and timing errors. On fixed traces of programs directly generated by five LLMs, checking outputs only at every fifth scan misses 21.66% of the 157 failures detected by per-scan comparison; across sampling phases, the missed-failure rate ranges from 12.10% to 28.03%. These misses stem from transient output discrepancies, whose detection depends on the sampling phase. Building on this evaluation, we propose CycleGen, which generates multiple candidate ST programs, repairs their compilation errors with bounded compiler feedback, and selects one using runtime self-tests generated in a separate model context. CycleGen improves the execution pass rate over direct generation by 8.68 percentage points. Component ablations distinguish repair and self-test selection; candidate-count analysis shows diminishing returns, with no execution-pass-rate improvement beyond five candidates in our experiments despite 38.8% higher token consumption at eight.

Comment
21 pages

Closest papers

Nearest neighbours by meaning across the whole index.

Trace2ST: Recovering PLC Structured Text from I/O Traces with LLM Generation and Verification

cs.SE cs.AIcs.CYcs.PL+1 ·2026-11-17 ·Hsi Chang, Irfan Khan
code/data similarity 0.76
Abstract · 6 pages, 3 figures, 4 tables. Submitted to the 2027 IEEE Texas Power and Energy Conference (TPEC 2027)

Programmable logic controllers (PLCs), the industrial computers that operate power and process plants, often run programs whose source code is no longer available; however, every input and output of the controller is logged continuously. This paper therefore asks whether a large language model (LLM) can recover an executable program in Structured Text (ST), the standard PLC language of IEC 61131-3, from such input/output records (traces) alone and, equally important, how the recovered program can be verified. To answer this question, we build a generate-replay-feedback pipeline. First, the LLM writes a candidate program from a compressed summary of the trace, without ever seeing the original program. The candidate is then executed one control cycle (scan) at a time and compared with the recording, and whenever a mismatch occurs the corresponding part of the trace is returned to the LLM, for at most three revisions. Because every result depends on the software that executes the candidates, we first validate it against two independent open-source PLC runtimes and find no unexplained difference. On this basis, we evaluate 47 third-party programs from two public benchmarks. The recovered programs reproduce the recorded trace exactly in 43 cases and agree with the original on 94% of scans for unseen inputs. The remaining errors, however, lie mostly in program branches that the trace never exercised; once a trace that targets these branches is added, agreement on unseen adversarial inputs rises from 0.80 to 0.98. By contrast, a classical automata-learning baseline without an LLM fits the recording but falls to 0.40 on unseen inputs, which indicates that recovering program structure is the hard part. Finally, on real data from a public water-treatment testbed, the recovered control laws match unseen measurements closely ($R^2 > 0.99$) and expose the only actuator attack in the dataset.

0.23

PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation

cs.AI ·2026-10-02 ·Yunji Kim, Yunseok Lee, Hyunwoo Seo, Jaerim Choi, Woojin Lee
benchmarkcode/data similarity 0.70
Abstract · 36 pages, 9 figures. Project website: https://yunji0516.github.io/PLCWorld/

Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large language model (LLM)-generated PLC programs satisfy task requirements and safety constraints requires observing how their commands affect device and workpiece states. We introduce PLCWorld, a common closed-loop execution environment and benchmark that couples Structured Text (ST) execution with simulated plant responses and sensor feedback. Grounded in control relations identified in industrial PLC programs and engineering documentation, PLCWorld contains 100 synthetic tasks and 473 registered task-condition pairs across Motion Control and Material Handling, with difficulty defined by control-dependency scope. A common protocol reports Task Success and Safety Violation separately. Validation combines practitioner review, reference and alternative programs, targeted counterexamples, specification-evaluator alignment checks, and comparisons with independent ST runtimes. Reference and alternative programs satisfy their applicable cases, while all 542 targeted counterexamples activate their designated evaluator rules under at least one registered condition. Execution Gap relates submission-profile acceptance to subsequent task failure or observed Safety Violation. Across the constructed task groups, direct GPT-5.5 achieves 82.70% Task Success on Easy cases but 25.10% on Hard cases. Evaluations of six LLMs and four adapted generation-and-verification workflows further expose differences between completion, safety, and generation cost. Our code, simulation environment, benchmark tasks, and baseline implementations are publicly available at https://yunji0516.github.io/PLCWorld/.

0.24

SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation

cs.SE ·2026-08-19 ·Yanlun Tu, Huacan Wang, Jin Xie, Ziyue Zhou, Jie Zhou +9 more
similarity 0.70
Abstract

In industrial practice, AI-generated programmable logic controller (PLC) code must integrate with existing projects and implement the required behavior across scan cycles. Compilation alone does not establish runtime behavior, and subsequent edits can invalidate earlier verification results. We present SemaPLC, a project-grounded, verification-gated agent harness for PLC code generation. The harness combines project context and reusable PLC skills with specification review, compilation, and live-runtime validation. Its completion gate requires recorded checks to meet acceptance criteria for the current implementation. Edits invalidate previous verdicts, and failed checks guide bounded repair. We evaluate SemaPLC on 117 independent-POU tasks and 65 project-context tasks across seven language-model backbones. SemaPLC outperforms the evaluated baselines on both metrics for every backbone. Its mean verified-pass rate and dynamic score reach 72.6% and 52.2, exceeding the strongest baseline means by 8.7% points and 20.8 points, respectively. Verification-layer and failure analyses reveal how build recovery and output observability affect end-to-end results and identify behavioral errors that remain after compilation. We apply the harness in KUKA's iiQworks product environment for PLC development and discuss engineering lessons for its industrial integration and deployment. The industrial use indicates the agent assistance substantially improves PLC development efficiency by 45%.

0.43

VSpector: Specification-Driven Bug Detection for RISC-V CPUs

cs.SE cs.AR ·2026-09-20 ·Tianyu Jia, Zhaoyang Yu, Yuanliang Chen, Wei You, Jianjun Huang +1 more
code/data similarity 0.68
Abstract · 17 pages, 7 figures

Detecting RTL design bugs in open-source RISC-V CPU implementations is critical for ensuring system reliability. Traditional detection approaches inherently rely on predefined artifacts. In this paper, we leverage the official,natural-language RISC-V specifications as an effective information source for bug detection. We present VSpector, a specification-driven bug detection pipeline that directly checks whether CPU register-transfer level (RTL) implementations adhere to official specification rules, without requiring specialized construction of reference models, formal properties, or custom bug patterns. To resolve the key technical trade-off between broad context scope and model reasoning accuracy when using Large Language Models (LLMs), VSpector employs a stepwise context refinement scheme across a four-stage pipeline: rule extraction, implementation localization, candidate identification, and sequential violation auditing. We evaluate VSpector on two industrial-strength RISC-V CPUs, CVA6 and XiangShan. Out of 217 reported candidates, manual inspection confirmed 148 true violations, representing a 68.2% precision. These violations correspond to 73 distinct bugs, including 42 previously unknown bugs. In our comparative experiments, DiveFuzz, a state-of-the-art CPU fuzzer, detected none of these new bugs during 24-hour runs per CPU. All 42 new bugs have been reported upstream, with developers already fixing 19 and confirming an additional 11 (30 in total), demonstrating that specification-driven auditing is a practical and complementary strategy for CPU bug detection.

0.31

More with Less: Efficient Parallel Scientific Code Translation via an Agent-Skill Workflow

cs.DC cs.AIcs.SE ·2026-11-06 ·Jianuo Shen, Pengyi Xu, Bo Zhang
publishedcode/data similarity 0.66
Abstract · Accepted to AgenticAI4HPC 2026 (SC26 Workshops). 10 pages

Translating parallel scientific codes between HPC programming frameworks is a critical but labor-intensive task. Existing LLM-based approaches rely on fixed sequential pipelines with undifferentiated repair strategies and correctness criteria that do not match scientific computing practice. We present an agent-skill framework that separates translation, compilation diagnosis, and execution validation into three autonomous agents coordinated by a verdict-driven orchestrator, routing each failure to a targeted repair skill rather than retrying blindly. An automated output check drives the self-correction loop, complemented by an offline LLM-based verification that exposes false positives escaping compiler and runtime checks. We evaluate the framework on 30 HeCBench kernels translated bidirectionally among CUDA, OpenMP, and SYCL with three open-source LLMs. Our framework attains the higher verified correctness rate in all 18 model and direction combinations over a state-of-the-art baseline while consuming up to 52.4% fewer tokens, remaining consistent across models and directions without architectural modification.

0.43

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

cs.PL cs.AI ·2026-09-29 ·Yuan Li, Hanyun Jiang, Guowei Tian, Chengpeng Wang, Peisen Yao
benchmark similarity 0.66
Abstract · 33 pages

Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require users to state their intent as input-output examples. Large language models (LLMs) suggest a more natural route, text-to-Datalog synthesis from a natural-language question, yet how well they do so has not been systematically evaluated. We present DatalogBench, a benchmark of 136 text-to-Datalog synthesis tasks curated from existing Datalog-based artifacts. Synthesized programs are graded by execution on held-out inputs against an oracle validated by mutation analysis. Across six LLMs and four prompting configurations, exact match peaks at 68.4%, and relation descriptions or an input-output example have only modest, model-dependent effects. Under direct prompting, most failures occur at compile time, typically because a model invents auxiliary predicates that it never declares or types consistently. Two coding agents reach up to 83.8% and eliminate nearly all such failures, leaving mostly semantic errors concentrated in recursive tasks. DatalogBench thus identifies recursive reasoning and decomposition as open challenges for current LLMs and agents, and offers a reliable, execution-grounded measure of both.

0.19

Are LLM-Generated Formal Specifications Faithful to Their Requirements? A Study on Bimodal Formal Specification Generation

cs.SE cs.PL ·2026-10-23 ·Wenhan Wang, Shiming Chen, Zeyu Sun, Shenghua Feng
benchmark similarity 0.66
Abstract

Formal specification generation is a critical step towards formal verification and correctness assurance of software. Recently, large language models have been widely adopted in automatic generation of formal specifications. However, existing LLM-based specification generation approaches often assume that the input program is correct, which deviates from the real purpose of formal verification---to figure out whether the program meets its requirements when its correctness is unknown. Therefore, we conduct an empirical study of LLM-based specification generation under the bimodal setting: given both the natural language requirement and its corresponding code implementation, whose correctness is unknown. We construct a benchmark of 1,403 C programs and evaluate ACSL specification generation with Frama-C verification. Our study reaches several interesting findings: first, when the code implementation is correct, providing the natural language requirement does not further improve the pass rate of generated specifications. Second, when the code implementation is incorrect, LLM-based specification generation may erroneously pass verification by producing weak or implementation-misled specifications, and this cannot be fully alleviated by advanced prompting approaches. Compared with the non-formal HoarePrompt approach, LLM-generated formal specifications are less effective in predicting requirement-code consistency. Our study takes a first look at applying LLM-based formal specification generation to verifying program correctness against requirements, and points to requirement-faithful specification generation as a new direction for future LLM-based formal specification research.

0.43

Large Language Models and Language Server Protocol: a match made in context

cs.SE ·2026-09-02 ·Alessandro Schena, Ilgiz Mustafin, Julia Kotovich
published similarity 0.64
Abstract · Submitted and presented at VERIFAI-2026

This article introduces Eiffel-tools, a language server protocol (LSP) implementation for the Eiffel programming language that uses Large Language Models (LLMs) to aid the development of statically verified software. The tool provides various interactive and non-interactive commands to produce code and specifications. It uses language and project specific knowledge to precisely direct the LLM and verifies the output using a static verifier. It crafts rich programmatic prompts for the input and corrects or rejects the output. Furthermore, it handles the retries until the program passes verification. The tool's bug fixing capability is evaluated on 2 public datasets using 3 models. The tool can fix 76% to 95% of bugs by combining LLMs and a formal verifier depending on the model and prompts used. The results show the trade-off between the number of fixing attempts and the success rate.

0.35