CS Paper Radar

DBcover: A White-box SQL Test Generation Framework for Coverage Improvement

Yankai Rong, Shuang Liu, Jinhao Dong, Qiang Yin, Wei Lu, Jianhua Wang, Xiaoyong Du

cs.DB cs.SE ·submitted 2026-08-26 09:36 UTC · v1
top venue

Abstract

Relational Database Management Systems (RDBMSs) are the backbone of modern data-intensive applications, making reliability and robustness critical. However, achieving high coverage in RDBMS testing remains challenging because of large codebases and complex execution logic. Traditional fuzzing relies on random SQL generation and cannot capture the correspondence between SQL inputs and internal execution paths, while symbolic execution suffers from prohibitive cost and scalability limitations. We propose DBcover, an LLM-driven white-box SQL test generation framework based on contextual reasoning. DBcover uses lightweight dynamic analysis to extract SQL-to-path correspondence and call graphs as global context, and collects source-level information around target functions as local context. These contexts are organized in a unified knowledge graph for efficient retrieval and reuse. DBcover then performs two-phase test generation: it first selects a semantically relevant seed whose execution path is close to the uncovered target, and then guides the LLM with global and local context to generate SQL test cases that trigger previously uncovered code regions. Experiments show that DBcover achieves 80.1% and 82.3% coverage on PostgreSQL and MySQL, and is also effective on the enterprise RDBMS KingbaseES, demonstrating its practical applicability to closed-source systems.

Comment
Accepted to the ICSE 2026 Industry Challenge Track. 12 pages, 4 figures, 2 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Distribution-Aware Distributed Database Testing (Extended Version)

cs.DB ·2026-09-16 ·Zhou Zhou, Si Liu, Hengfeng Wei, Min Zhang
published similarity 0.72
Abstract · 18 pages, technical report for Distribution-Aware Distributed Database Testing (VLDB27)

Distributed database management systems (DDBMSs) introduce new challenges for assessing their reliability due to distribution-specific characteristics that affect query execution and optimization. Existing testing approaches, largely designed for centralized DBMSs, often fail to explore diverse distributed execution behaviors and suffer from low executability of generated test queries, thereby limiting their effectiveness in bug detection. We propose DAT (Distribution-Aware Testing), a novel automated approach for detecting query-processing bugs related to distribution strategies and distributed optimizations in DDBMSs, by systematically leveraging distribution-aware information throughout the testing pipeline. DAT builds on a set of techniques that capture diverse combinations of logical schemas and data distribution strategies, and performs guided query mutation to trigger a wide range of distributed query execution behaviors and optimizations, while improving query executability via historical feedback. We implement our approach in a tool, DistRanger, and evaluate it on four widely used production DDBMSs. It uncovers 31 previously unknown bugs, including 28 related to distributed query processing and optimization, and outperforms state-of-the-art testers.

0.24

Agentic Evolution of SQL Test Generators

cs.SE ·2026-10-09 ·Suyang Zhong, Manuel Rigger
similarity 0.71
Abstract

Automated testing has uncovered many logic bugs in Database Management Systems (DBMSs). Existing tools are typically limited to a few SQL dialects and a subset of these dialects' features. Coding agents based on Large Language Models (LLMs) could encode the semantics of additional SQL features into reusable test generators, requiring no additional LLM calls during testing and thus enabling high efficiency and low cost. However, a key challenge is validating the added capabilities in the presence of potential hallucinations. In this paper, we propose Gungnir, an approach for evolving DBMS test generators using coding agents. Our core mechanism is witness replay: the coding agent provides a generation witness recording the database setup and semantic choices needed to generate a target statement. A validator replays these choices through the modified generator and checks that it reproduces the target statement. Historical witnesses serve as regression tests for subsequent modifications. We evaluated Gungnir on SQLite, DuckDB, and PostgreSQL and found 44 bugs confirmed by developers, of which 34 have been fixed. Beyond DBMSs, witness replay could generalize to other systems to guide agents in extending test generators.

0.25

Towards Scalable and Cost-Efficient Vulnerability Detection: A Study on Automatic Query Generation

cs.SE cs.CR ·2026-09-09 ·Ivana Clairine Irsan, Ratnadira Widyasari, Huihui Huang, Ting Zhang, Yue Liu +4 more
similarity 0.68
Abstract

Static analysis remains a cornerstone of software security, yet the effectiveness of tools such as CodeQL is often limited by the substantial manual effort required to develop high-coverage query suites. While large language models (LLMs) have emerged as a potential solution for automated code reasoning, their practical utility in generating structured, executable security queries remains underexplored. In this paper, we conduct an empirical study to evaluate the ability of LLMs to synthesize CodeQL queries using vulnerability data from the National Vulnerability Database. Through this investigation, we explore the potential of using LLMs as an automatic CodeQL query generator. Subsequently, we systematically evaluate the performance of various LLM architectures across a diverse set of real-world vulnerabilities, measuring their ability to improve detection coverage and precision. Our findings reveal that LLM-generated queries significantly enhance the baseline CodeQL queries, yielding 82% improvement in average F1-score. Furthermore, we provide a detailed cost- benefit analysis showing that while direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, leveraging LLMs to synthesize CodeQL queries offers a scalable and cost-effective alternative for large-scale vulnerability detection. Our results suggest that LLMs can effectively bridge the gap between unstructured vulnerability reports and formal static analysis specifications, offering a scalable path toward comprehensive automated vulnerability detection.

0.38

LLM-Based Test Generation: Information Sources, Generation Strategies, and Quality Evidence

cs.SE ·2026-10-04 ·Yunhao Liang, Chengguang Gan, Ruixuan Ying, Hanjun Wei, Zhe Cui +1 more
similarity 0.67
Abstract

Large language models are increasingly used to generate test scenarios, executable test suites, assertions, interaction sequences, and fuzzing infrastructure. These artifacts serve different purposes and rely on different sources of information about correct behavior. A test that increases implementation coverage, a test that agrees with a reference program, and a test that detects a requirements violation therefore provide distinct kinds of evidence. We present a structured narrative survey that connects the generation process to the evidence used to justify test quality. Drawing on 95 curated source records through 29 September 2026, we organize the field along four dimensions: testing objectives and artifacts, information available during training and generation, generation and learning mechanisms, and evaluation evidence. We synthesize work on unit and requirements-based testing, test oracles, API and GUI testing, fuzzing, and learned test generators. The resulting framework explains how execution feedback can improve executability while also affecting the independence of an oracle, why specification provenance and training-time reference supervision matter for comparisons, and how downstream code-selection gains differ from test correctness. We use these distinctions to organize benchmarks and derive a research agenda for independent oracle assessment, budget-aware generation, repository-scale evaluation, and test maintenance.

0.13

Efficient Dialect-Agnostic SQL Equivalence Refutation

cs.DB ·2026-10-15 ·Junwen An, Manuel Rigger
publishedbenchmark similarity 0.67
Abstract

Identifying semantically inequivalent SQL queries is useful for many applications, such as evaluating Text-to-SQL and dialect translation systems. However, existing approaches often struggle with the diversity of SQL features and dialects, requiring significant manual effort and expertise to extend query feature support across Database Management Systems (DBMSs). To address these limitations, we present SQLWitness, an efficient, DBMS-agnostic query equivalence refutation engine. Our core idea is leveraging the DBMS execution engine to determine whether generated instances constitute counterexamples, thereby faithfully capturing the semantics of the SQL dialect and preventing false counterexamples. To efficiently generate test data, our approach explores a constrained data-generation space and uses an integrity-constraint-aware generation strategy. Furthermore, to guide time-efficient termination of this test-data generation process, we pair predicate-coverage signals with a statistical estimator. We extensively evaluated SQLWitness on six benchmark suites across four SQL dialects. SQLWitness refuted 5.34% to 88.4% more pairs than state-of-the-art baselines, and processed query pairs in 0.12 seconds on average, achieving a speedup from 2.1x to 16.4x over the baselines. Furthermore, in a case study, SQLWitness found 8.4% inconsistent translations within a dialect translation benchmark. We believe SQLWitness could be broadly adopted for many use cases.

0.35

Detecting DBMS Bugs by Constructing Equivalent Representations of Intermediate Query Results

cs.DB cs.SE ·2026-08-31 ·Xiaoxu Niu, Gong Chen, Jinfu Chen, Xiaoyuan Xie
code/data similarity 0.66
Abstract · 12 pages, 3 figures

Database Management Systems (DBMSs) support multiple SQL mechanisms for representing intermediate query results, including VIEWs, Common Table Expressions (CTEs), and Temporary Tables (TEMPTs). When these mechanisms are used to represent the same intermediate query result, the corresponding queries are expected to produce consistent results. However, we observe that such queries can return inconsistent results, indicating potential DBMS logic bugs. Existing approaches for detecting DBMS logic bugs have never explored result consistency across such equivalent representations. In this paper, we propose ERIQ, a novel testing approach for detecting DBMS logic bugs from the perspective of checking result consistency across Equivalent Representations of Intermediate Query Results. ERIQ constructs SQL variants using a VIEW, a CTE, or a TEMPT to represent the same intermediate query result, executes these variants, and compares their returned results. We evaluated ERIQ on four widely used open-source DBMSs: MySQL, MariaDB, Percona, and OceanBase. In total, ERIQ detected 64 bugs, 63 of which were confirmed by developers, and two have been fixed. Among the confirmed bugs, 54 were unique and previously unknown logic bugs, and one was a documentation issue.

0.24

NEUROTESTGEN: Neuro-Symbolic Guided Test Generation with Large Language Models

cs.SE ·2026-09-24 ·Ruixin Zhang, Jiho Shin, Hung Viet Pham, Song Wang
benchmark similarity 0.66
Abstract

Ensuring high structural coverage remains a fundamental challenge in automated test generation, particularly for complex software systems where reaching specific lines or branches requires satisfying intricate control- and data-flow constraints. Large Language Models (LLMs) have recently demonstrated strong capabilities in producing human-like test cases; however, they often struggle to generate inputs that satisfy precise path conditions. Conversely, symbolic execution can systematically derive such constraints, but it often fails to construct realistic, executable test cases and is constrained by scalability limitations. In this paper, we introduce NEUROTESTGEN, a hybrid approach that integrates symbolic execution with LLM-driven test synthesis to generate test cases targeting on-demand code coverage. Given a set of target statements within a method, NEUROTESTGEN first employs a symbolic analysis engine (i.e., the Z3 SMT solver) to extract path-specific constraints and construct a symbolic guidance specification for the desired coverage goal. This specification is then used to guide an LLM in synthesizing concrete test cases that are both structurally valid and semantically meaningful. For paths involving complex object-related constraints that are difficult for SMT solvers to handle, NEUROTESTGEN leverages LLMs to infer plausible constraints. Furthermore, NEUROTESTGEN incorporates an iterative feedback loop that validates LLM-generated tests and provides corrective guidance until the target line or branch is covered or a limit is reached. Our empirical evaluation on a widely used benchmark demonstrates that NEUROTESTGEN significantly outperforms the state-of-the-art approach across multiple LLMs, including Llama 3.3 70B1, GPT-4o Mini, Claude 3.5 Haiku3, and Claude Sonnet 4.6.

0.08

PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming

cs.CL ·2026-08-16 ·Marianne Menglin Liu, Leonid Boytsov, Daniel W. Peterson, Pramuditha Perera, Rongguang Wang +11 more
top venuebenchmarkcode/data similarity 0.66
Abstract

We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.

0.84