CS Paper Radar

Trace2ST: Recovering PLC Structured Text from I/O Traces with LLM Generation and Verification

Hsi Chang, Irfan Khan

cs.SE cs.AIcs.CYcs.PLeess.SY ·submitted 2026-11-17 19:57 UTC · v1
code/data

Abstract

Programmable logic controllers (PLCs), the industrial computers that operate power and process plants, often run programs whose source code is no longer available; however, every input and output of the controller is logged continuously. This paper therefore asks whether a large language model (LLM) can recover an executable program in Structured Text (ST), the standard PLC language of IEC 61131-3, from such input/output records (traces) alone and, equally important, how the recovered program can be verified. To answer this question, we build a generate-replay-feedback pipeline. First, the LLM writes a candidate program from a compressed summary of the trace, without ever seeing the original program. The candidate is then executed one control cycle (scan) at a time and compared with the recording, and whenever a mismatch occurs the corresponding part of the trace is returned to the LLM, for at most three revisions. Because every result depends on the software that executes the candidates, we first validate it against two independent open-source PLC runtimes and find no unexplained difference. On this basis, we evaluate 47 third-party programs from two public benchmarks. The recovered programs reproduce the recorded trace exactly in 43 cases and agree with the original on 94% of scans for unseen inputs. The remaining errors, however, lie mostly in program branches that the trace never exercised; once a trace that targets these branches is added, agreement on unseen adversarial inputs rises from 0.80 to 0.98. By contrast, a classical automata-learning baseline without an LLM fits the recording but falls to 0.40 on unseen inputs, which indicates that recovering program structure is the hard part. Finally, on real data from a public water-treatment testbed, the recovered control laws match unseen measurements closely ($R^2 > 0.99$) and expose the only actuator attack in the dataset.

Comment
6 pages, 3 figures, 4 tables. Submitted to the 2027 IEEE Texas Power and Energy Conference (TPEC 2027)

Closest papers

Nearest neighbours by meaning across the whole index.

Compilation Is Not Correctness: Scan-by-Scan Evaluation of LLM-Generated PLC Programs

cs.SE ·2026-10-19 ·Jinyun Chen, Juntao Hu, Haochen Li, Wentao Song, Ningbo Gu +3 more
benchmark similarity 0.76
Abstract · 21 pages

Programmable logic controllers (PLCs) automate industrial processes through repeated scans, each reading inputs, executing control logic, and updating outputs. Engineers write this logic in Structured Text (ST), a high-level textual language defined in IEC 61131-3. Large language models (LLMs) can now generate ST from natural-language requirements, yet compilation success and checks on selected properties or sampled scans can leave behavioral errors undetected. We introduce CycleBench, a benchmark of 301 ST tasks with behavioral requirements, public interfaces, reference programs, and executable scenarios. Its runtime executes each generated program and its reference under aligned inputs, initialization, and logical time, comparing their outputs after every scan. Across eight LLMs and baseline generation methods, execution pass rate trails compilation pass rate by 11.92 percentage points; observed failures include output-update, arithmetic, and timing errors. On fixed traces of programs directly generated by five LLMs, checking outputs only at every fifth scan misses 21.66% of the 157 failures detected by per-scan comparison; across sampling phases, the missed-failure rate ranges from 12.10% to 28.03%. These misses stem from transient output discrepancies, whose detection depends on the sampling phase. Building on this evaluation, we propose CycleGen, which generates multiple candidate ST programs, repairs their compilation errors with bounded compiler feedback, and selects one using runtime self-tests generated in a separate model context. CycleGen improves the execution pass rate over direct generation by 8.68 percentage points. Component ablations distinguish repair and self-test selection; candidate-count analysis shows diminishing returns, with no execution-pass-rate improvement beyond five candidates in our experiments despite 38.8% higher token consumption at eight.

0.30

SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation

cs.SE ·2026-08-19 ·Yanlun Tu, Huacan Wang, Jin Xie, Ziyue Zhou, Jie Zhou +9 more
similarity 0.67
Abstract

In industrial practice, AI-generated programmable logic controller (PLC) code must integrate with existing projects and implement the required behavior across scan cycles. Compilation alone does not establish runtime behavior, and subsequent edits can invalidate earlier verification results. We present SemaPLC, a project-grounded, verification-gated agent harness for PLC code generation. The harness combines project context and reusable PLC skills with specification review, compilation, and live-runtime validation. Its completion gate requires recorded checks to meet acceptance criteria for the current implementation. Edits invalidate previous verdicts, and failed checks guide bounded repair. We evaluate SemaPLC on 117 independent-POU tasks and 65 project-context tasks across seven language-model backbones. SemaPLC outperforms the evaluated baselines on both metrics for every backbone. Its mean verified-pass rate and dynamic score reach 72.6% and 52.2, exceeding the strongest baseline means by 8.7% points and 20.8 points, respectively. Verification-layer and failure analyses reveal how build recovery and output observability affect end-to-end results and identify behavioral errors that remain after compilation. We apply the harness in KUKA's iiQworks product environment for PLC development and discuss engineering lessons for its industrial integration and deployment. The industrial use indicates the agent assistance substantially improves PLC development efficiency by 45%.

0.43

PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation

cs.AI ·2026-10-02 ·Yunji Kim, Yunseok Lee, Hyunwoo Seo, Jaerim Choi, Woojin Lee
benchmarkcode/data similarity 0.64
Abstract · 36 pages, 9 figures. Project website: https://yunji0516.github.io/PLCWorld/

Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large language model (LLM)-generated PLC programs satisfy task requirements and safety constraints requires observing how their commands affect device and workpiece states. We introduce PLCWorld, a common closed-loop execution environment and benchmark that couples Structured Text (ST) execution with simulated plant responses and sensor feedback. Grounded in control relations identified in industrial PLC programs and engineering documentation, PLCWorld contains 100 synthetic tasks and 473 registered task-condition pairs across Motion Control and Material Handling, with difficulty defined by control-dependency scope. A common protocol reports Task Success and Safety Violation separately. Validation combines practitioner review, reference and alternative programs, targeted counterexamples, specification-evaluator alignment checks, and comparisons with independent ST runtimes. Reference and alternative programs satisfy their applicable cases, while all 542 targeted counterexamples activate their designated evaluator rules under at least one registered condition. Execution Gap relates submission-profile acceptance to subsequent task failure or observed Safety Violation. Across the constructed task groups, direct GPT-5.5 achieves 82.70% Task Success on Easy cases but 25.10% on Hard cases. Evaluations of six LLMs and four adapted generation-and-verification workflows further expose differences between completion, safety, and generation cost. Our code, simulation environment, benchmark tasks, and baseline implementations are publicly available at https://yunji0516.github.io/PLCWorld/.

0.24

PLC-Bin2Src: Retrieving Corresponding Structured Text Source Files for PLC Binaries

cs.SE ·2026-09-08 ·Ang Jia, Yaxin Duan, He Jiang, Ming Fan, Zhilei Ren +1 more
similarity 0.63
Abstract

PLC developers reuse existing components and third-party libraries to reduce the cost of developing new applications, but this reuse can also introduce security risks. When components are distributed only in binary form, provenance tracing, vulnerability assessment, and security auditing become more difficult. To support these analyses, binary2source matching can retrieve the corresponding Structured Text (ST) source file for a PLC binary from a candidate source repository. However, due to the properties of PLC software, this task faces three challenges: cross-platform compilation heterogeneity, a binary-source representation gap, and inadequate representations of PLC semantics. Therefore, we present PLC-Bin2Src, a cross-platform binary2source matching framework that retrieves corresponding ST source files for binaries produced by CODESYS, GEB, OpenPLC v2, and OpenPLC v3. First, PLC-Bin2Src uses platform-aware binary frontends to recover user control logic across toolchains. Then, ST-to-C conversion and shared normalization are used to reduce syntactic differences between source- and binary-side representations. Next, control-data flow graphs (CDFGs) and function call graphs (FCGs) are constructed to capture program semantics, while recovered symbols provide complementary identity evidence. Finally, PLC-Bin2Src compares these representations and combines their similarity scores to rank source candidates. We evaluate PLC-Bin2Src on PLC-BEAD. The results show that PLC-Bin2Src achieves 95.89% Recall@1, 99.66% Recall@5, and an MRR of 0.9769 across four PLC platforms.

0.24

SERIAL-7: Deriving, Realizing, and Preserving Machine-Verifiable Outputs Across Representations

cs.AI cs.CL ·2026-10-30 ·Youngkyung Seo, MiHyeon Kim, Jisoo Lee, Hwijung Ryu
publishedbenchmark similarity 0.62
Abstract · NeurIPS 2026 Workshop on AI for Verifiable Coding

Large language models increasingly generate structured outputs consumed by programs and downstream systems, where semantic correctness alone does not guarantee reliable execution. We introduce SERIAL-7, a Korean-English benchmark built around an executable verification framework for machine-verifiable generation across seven representations. Using 177 validated canonical artifacts, SERIAL-7 separates semantic derivation (Derive), constrained artifact generation (Realize), and cross-representation preservation (Preserve). Its 43 atomic representation constraints are paired with executable checks over format, structure, and task-specific semantics, enabling progressively stricter verification from representation compliance to full artifact validity. Across ten models, REALIZE-L3 HSR ranges from 42.0% to 93.9%, while Full Compliance exposes additional artifact-level failures beyond correct final answers and valid representations. These results show that reliable structured generation requires explicit, executable verification of how artifacts are realized and preserved across representations.

0.34

ForTress: Bringing Assurance and Determinism for LLM-Driven Fortran-to-Python Translation

cs.SE ·2026-10-15 ·Qinrun Dai, Ruitong Liu, Chien-Wei Huang, Guanhong Tao, Guannan Wei +1 more
similarity 0.62
Abstract

Modernizing mature scientific software from Fortran to Python can improve accessibility and integration with modern computing tools, but manual rewriting is expensive. Large language models (LLMs) make translation more practical, yet generated code can pass regression tests while violating source-language semantics, and contextual corrections may not persist across subsequent translations. We present ForTress, which solidifies recurring LLM translation decisions into guarded executable rules for deterministic reuse and provides semantic assurance for a non-trivial subset of these rules. Its LLM-assisted auditing process expresses universally quantified preservation obligations over mechanized Fortran and Python semantics in Lean. Human experts review semantic fidelity and correspondence with production rules, and Lean's kernel checks proofs and refutations under explicit assumptions. These audit records expose the conditions either supporting correctness or identifying failure cases. We evaluate ForTress on 17 benchmarks from 15 scientific software spanning diverse domains. The translated programs achieve bitwise-identical with native Fortran on all evaluated tests under the recorded execution and initialization configurations. Against these software, ForTress accumulates 193 executable rules and 188 audit records, of which 121 have checked soundness proofs, including 31 established after counterexample-guided refinement. Cross-software reuse exceeds 90% for 14 of the 15 software. Even with additional assurance and determinism, the LLM usage cost is 44.2% lower than direct LLM translation.

0.21

Large Language Models and Language Server Protocol: a match made in context

cs.SE ·2026-09-02 ·Alessandro Schena, Ilgiz Mustafin, Julia Kotovich
published similarity 0.61
Abstract · Submitted and presented at VERIFAI-2026

This article introduces Eiffel-tools, a language server protocol (LSP) implementation for the Eiffel programming language that uses Large Language Models (LLMs) to aid the development of statically verified software. The tool provides various interactive and non-interactive commands to produce code and specifications. It uses language and project specific knowledge to precisely direct the LLM and verifies the output using a static verifier. It crafts rich programmatic prompts for the input and corrects or rejects the output. Furthermore, it handles the retries until the program passes verification. The tool's bug fixing capability is evaluated on 2 public datasets using 3 models. The tool can fix 76% to 95% of bugs by combining LLMs and a formal verifier depending on the model and prompts used. The results show the trade-off between the number of fixing attempts and the success rate.

0.35

PLCBench: Can Autonomous LLM Agents Turn PLC Access into Sustained Physical Impact?

cs.CR cs.AI ·2026-08-27 ·Yitian Zhou, Jingyu Zheng, Qiliang Jiang, Linkang Du, Haoming Liu +4 more
similarity 0.60
Abstract · 36 pages, 13 figures

Industrial control systems (ICSs) rely on programmable logic controllers (PLCs) to connect networked computation with physical control. Tool-using large language model (LLM) agents represent an emerging attack threat: can an autonomous agent convert a network-reachable PLC into sustained adverse physical impact? However, existing evaluations focus on digital tasks or individual stages of PLC testing. In ICSs, evaluations that stop at software exploitation, an accepted write, or tool access may therefore mischaracterize physical risk. We present PLCBENCH, to our knowledge, the first real-PLC hardware-in-the-loop (HIL) framework for characterizing this cyber-to-physical capability and its boundaries. It combines vendor-native interaction, commercial PLC execution, closed-loop reduced-order process simulation, and independent outcome verification. A deterministic evaluator applies fixed rules to runner, communication, PLC-object, and process records to assign six hidden diagnostic flags, distinguishing usable PLC interaction, process-linked manipulation, and sustained physical impact. We instantiate PLCBENCH on four commercial PLCs crossed with four closed-loop workloads. Across five LLM families and 240 real-PLC episodes, 75 episodes (31.3%) sustain their respective physical objectives. Stagewise results show that 98 episodes stop before a valid native read, whereas 62 reach a process-linked write but do not sustain the final objective. Notably, richer process observation is associated with an increase in conditional objective attainment after a process-linked write from 44.2% to 64.0%. These measurements localize failure in configured PLC-process deployments and identify intervention points for future defense evaluation. To support reproducibility, we release the safely disclosable PLCBENCH code and a software-only reproduction pipeline through the accompanying artifact.

0.19