CS Paper Radar

How Well Do SAST Tools Work Across Languages? An Empirical Study Using OCPP Implementations

Zachary Wadhams, Emma Sheppard, Dalton Arford, Clemente Izurieta, Ann Marie Reinhold

cs.CR cs.SE ·submitted 2026-11-10 17:21 UTC · v1
published

Abstract

As electric vehicle (EV) adoption accelerates globally, securing electric vehicle charging station (EVCS) software becomes increasingly critical. The Open Charge Point Protocol (OCPP), widely adopted for EVCS management, places significant responsibility on developers to securely implement specifications that are subject to multiple interpretations. The variety of OCPP implementations across different programming languages provides a unique opportunity to evaluate security tooling in a real-world, multi-language context where implementation variability may introduce vulnerabilities. This study provides the first empirical evaluation of Static Application Security Testing (SAST) tool performance across multiple programming languages using OCPP implementations as a real-world test corpus. We applied both comprehensive multi-language tools (SonarQube and Semgrep) and language-specific analyzers to twelve OCPP 1.6 implementations--each written in a distinct programming language--to examine how detection patterns vary across language ecosystems. Total issues ranged from just two in Rust to 1,689 in Java. Mature programming languages like Java yielded substantially more findings than newer languages like Rust, likely because established languages have more developed SAST tool ecosystems rather than inherently worse code quality. These results suggest that developers should consider available SAST tooling when selecting implementation languages for security critical systems.

Comment
6 pages, 2 tables, Author-accepted manuscript. Published at IEEE CSR 2026
Journal ref
2026 IEEE International Conference on Cyber Security and Resilience (CSR), pp. 250-255 (2026)
DOI
10.1109/CSR69374.2026.11759929

Closest papers

Nearest neighbours by meaning across the whole index.

Evaluating and Improving Static Application Security Testing Tools for Rust

cs.CR cs.SE ·2026-10-15 ·Huaien Zhang, Zhengjie Zhao, Yu Pei, Tian Dong, Minxue Pan +2 more
similarity 0.71
Abstract

Rust has become a popular systems programming language in recent years because of its strong guarantees on memory and concurrency safety. Many types of system software have been re-implemented in Rust. However, Rust programs can still suffer from common memory and concurrency vulnerabilities due to unsafe code and misunderstandings of its complex mechanisms. To mitigate such vulnerabilities, developers often rely on static application security testing (SAST) tools. However, the effectiveness of existing Rust SAST tools on real-world vulnerabilities remains underexplored. In this paper, we first conduct an empirical evaluation of mainstream Rust SAST tools using security vulnerabilities identified in real-world Rust applications. We systematically assess their effectiveness, uncover common weaknesses, and analyze the root causes behind them. Motivated by the study, we propose TRUST, a large language model (LLM)-assisted automated testing framework. TRUST automatically analyzes the official Rust reference to derive and implement semantics-preserving transformations. It leverages these transformations to generate mutant programs for evaluating Rust SAST tools and uncovering potential faults within them. Overall, TRUST has revealed 46 faults across six popular SAST tools, 17 of which have been confirmed. In addition, our study yields ten empirical findings and four practical lessons that provide actionable guidance for developing more effective, robust, and maintainable Rust SAST tools.

0.15

Security Is More Than a Library Call: How Security Features Live in Code

cs.CR cs.SE ·2026-10-05 ·Kevin Hermann, Sven Peldszus, Thorsten Berger
code/data similarity 0.69
Abstract

Implementing security features---functionalities that protect sensitive data or prevent malicious actions by attackers---is important for ensuring the security and integrity of software systems. Correctly implementing access control, cryptography, or other security features is challenging as they require substantial domain expertise, careful design, and custom implementation to integrate them within the software system. Previous work has thoroughly studied security awareness, perception, practices, and expertise via surveys, interviews, and experiments. While they have shown that the implementation of security features are often supported by security libraries and frameworks, they have also shown that developers rarely are security experts, and make mistakes that introduce vulnerabilities into software when using them. Despite this extensive knowledge of security practices and developer behavior, we still lack a comprehensive understanding of how security features actually manifest at the code level across full software systems. We close this gap by conducting a mining study of security features in 9 popular and large open-source Java systems. We manually inspected 2,127,761 lines of code across 19,121 files, identifying and annotating 183,395 lines implementing 561 security features corresponding to 54 security features in our taxonomy. We analyzed the characteristics of the identified security features, such as their size, scattering, tangling, common implementation patterns, and the use of internal and external functionalities. Our findings show that security features are far more than mere calls to external libraries. Security features, such as access control, can grow large in size, scatter across the whole codebase, and frequently tangle with each other. External security libraries are commonly used, but they require substantial amounts of code for integration.

0.33

Combining LLM and Dynamic Analysis for Universal Testing of Static Application Security Testing Tools

cs.CR cs.SE ·2026-10-15 ·Huaien Zhang, Yu Pei, Hao Chen, Shin Hwei Tan
similarity 0.67
Abstract

Static application security testing (SAST) tools help detect potential software vulnerabilities, but they themselves contain faults undermining reliability and usability. Automated testing can address these issues, yet current techniques suffer from limited program transformation diversity and lack mechanisms to verify generated variants, leading to reduced fault detection and high manual effort. We present LLM4SAST, the first universal framework detecting faults in SAST tools using large language models and dynamic feedback. Our key insight is leveraging LLMs to understand and transform code in diverse programming languages. We design novel metamorphic relations with transformations to guide variant generation, using an LLM to produce multi-language variants that satisfy these relations. To reduce erroneous variants and identify unique faults, we propose two heuristics: dynamic-feedback-driven selection and program-differencing-based clustering. For each metamorphic relation, a report-centric test oracle detects variants that fail tests and identifies underlying faults by differentially analyzing SAST reports of the seed program and its variants. LLM4SAST identified 66 unique faults across six SAST tools, with 32 confirmed or fixed after being reported to developers, demonstrating effectiveness. The proposed heuristics eliminate 47.8% of false positives, and LLM4SAST detects 3.4 times more faults while requiring only 39.8% of the variants compared to existing approaches.

0.19

Building Sound Evaluation Practices for Rust-Based Security Frameworks

cs.CR ·2026-10-14 ·David Paaßen, Lucas Davi
similarity 0.66
Abstract

Background: Rust is a modern programming language known for its focus on usability, concurrency, and security. However, empirical studies reveal that Rust is still affected by software vulnerabilities, with hundreds of memory safety issues found in Rust programs, including Rust's standard library. As a result, additional tool-assisted security testing is crucial to enhance the security of Rust code. Evaluating such tools, however, is highly complex. While there is substantial research on evaluating C/C++ security frameworks, including test sets, serious issues in published evaluations have been identified, undermining their scientific soundness and reproducibility. In practice, selecting an inadequate implementation (e.g., a fuzzer) can lead to missed vulnerabilities ultimately compromising a program's security. Aims: Our methodology aims to help researchers to conduct scientifically sound evaluations in order to prevent similar issues with the evaluation of security tools and frameworks in the Rust ecosystem. Method: We systematically analyze existing evaluation literature to develop the first sound methodology for Rust based evaluations. Our guidelines provide enough flexibility to support a variety of use cases. To quantify the performance of a security tool, we introduce an automated approach to inject unsafe code coverage oracles, which require no forward porting, avoid overfitting, and allow updating test sets in the future. We also present methods to quantify the diversity of a Rust test set, ensuring that the results are applicable to a large set of existing Rust programs. We practically apply our methodology in our evaluation of the most popular Rust fuzzers. Results: We show the practical applicability of our design and conduct an evaluation with a diverse test set of 20 Rust projects. [...]

0.18

Barriers to Using Static Application Security Testing (SAST) Tools: A Literature Review

cs.SE cs.CR ·2026-09-01 ·Zachary Wadhams, Clemente Izurieta, Ann Marie Reinhold
top venuesurveypositionhighly cited similarity 0.66
Abstract · 6 pages, 4 figures, published at 2024 39th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)

Developers face a challenging problem with no clear solution. Modern software breaches can wreak havoc on businesses and individuals alike. With code vulnerabilities being a leading cause, securing applications must be a priority for developers. Static Application Security Testing (SAST) has the potential to harden applications by assisting in the identification and resolution of security vulnerabilities. Despite this, many development teams have not adopted SAST tools into their environment. In this paper, we survey the recent literature to uncover why some developers are apprehensive towards SAST and identify what specific problems they encounter when using it. We found a variety of usability problems developers face when using SAST. Some are inherent of the tool and ultimately require some level of developer investment while others are tool shortcomings that SAST tool creators must address. Ultimately, we argue that in order to drive widespread adoption and consistent SAST usage, developers will need to embrace that some investment is required. Simultaneously, developers will be more likely to integrate SAST tools into their workflows if the creators of SAST tools simplify many aspects related to tool usage. Surmounting the primary obstacles preventing the adoption of SAST requires full consideration of both the technical and human factors.

0.65

TRACTOR Benchmark for Evaluating C to Rust Translators

cs.SE cs.CRcs.PL ·2026-09-20 ·Hamed Okhravi, Brandt Ogden, Noah Luther, Ian McQuoid, Howard Mak +1 more
benchmarkcode/data similarity 0.66
Abstract

Memory-safety vulnerabilities remain a persistent source of security risk in critical software, much of which is implemented in memory-unsafe languages such as C and C++. Recent advances in programming languages, program analysis, and artificial intelligence have created new opportunities to modernize these legacy systems through automated translation to memory-safe languages such as Rust. The DARPA Translating All C to Rust (TRACTOR) program seeks to develop scalable techniques for translating large C codebases into safe, performant, and maintainable Rust. MIT Lincoln Laboratory serves as the program's independent test and evaluation organization and has developed a standardized benchmark for systematically assessing C-to-Rust translation tools. This report describes the TRACTOR benchmark, including progressively challenging test batteries and larger milestone projects, as well as the supporting evaluation infrastructure and metrics for assessing correctness, safety, idiomaticity, and performance. The benchmark and associated evaluation infrastructure are publicly available to support the broader development and evaluation of C-to-Rust translation technologies.

0.25

EVSEFuzz : A Tool for Automated Testing of Protocol Implementations

cs.CR ·2026-10-23 ·Soham Gumaste, Michael Khan, Roland Varriale, Rigel Gjomemo
similarity 0.65
Abstract

The rapid adoption and development of electric vehicle (EV) networks has required the expansive deployment of EV charging infrastructure (EVCI) which relies on communication protocols such as the open charge point protocol (OCPP) to coordinate data transfer between charging network operators and the EV supply equipment. In this paper, we present EVSEFuzz, a tool for automatically building fuzzers to test the security of protocol implementations. To achieve this we make a novel use of Python datatype validation packages and tools to ``understand'' datatypes and message structures and generate both compliant and non-compliant messages. To date, EVSEFuzz has been tested successfully with several open and closed-source implementations and has found several vulnerabilities leading to the publication of two CVEs.

0.23

GraftyVul: Synthesising Insecure Programs Through Real-World Vulnerability Grafting

cs.CR cs.SE ·2026-08-28 ·Omri Ram, Mitchell Horner, Ron Van der Meyden, Alsharif Abuadbba, Hammond Pearce
code/data similarity 0.64
Abstract · 13 pages, 8 figures

Vulnerability datasets underpin a wide range of security research, including vulnerability detection, automated remediation, and secure code generation. However, existing datasets sacrifice at least one of three desirable properties: diversity (of language or vulnerability type), reproducibility/executability, or realism. We therefore present GraftyVul, a system that constructs vulnerable programs by grafting real-world vulnerabilities into open-source projects. This grounds the dataset in vulnerabilities observed in real-world contexts while harnessing known good build and test environments, enabling exploit-verification scripts to guarantee that an introduced vulnerability successfully alters a program's behaviour. Using GraftyVul, we generate 212 verified and exploitable vulnerable programs spanning five programming languages (Python, TypeScript, Java, Go, and C#) across 23 CWE categories. To evaluate fidelity, we introduce a language- and context-agnostic semantic embedding that compares vulnerabilities by sink, mechanism and host-feature rather than surface code. This approach outperforms standard code embeddings on cross-language clone and CWE classification. These embeddings demonstrate that GraftyVul samples retain a strong semantic signature to their source vulnerability. We additionally compare GraftyVul against 13 widely used datasets, where it attains competitive diversity while being the only reproducible-exploit dataset with broad language and CWE coverage. Finally, we illustrate GraftyVul's practical utility through an industrial case study evaluating a production vulnerability remediation system.

0.19