CS Paper Radar

Why, Where, How: Taxonomy-guided Error Grounding for Code Repair in NL2SQL

Suchan Lee, Woomin Song, Hwanjo Yu, Sangwoo Mo

cs.AI ·submitted 2026-10-04 09:04 UTC · v1

Abstract

SQL queries that large language models write from natural language questions can execute successfully yet produce incorrect results, so execution alone does not reveal what to fix. An error taxonomy says why the query is wrong, but not where to look or how to change it. Existing methods can guide SQL correction through feedback, error reports, or generated plans alongside an unmasked query. We introduce TEG(Taxonomy-guided Error Grounding), which turns a supplied diagnosis into a structured correction input for natural language-to-SQL (NL2SQL) correction. Type-specific rules map each error type to construct classes to reconsider and an edit operation to request. TEG masks the selected constructs in the query when applicable and states that operation in an edit instruction. TEG generates candidate corrections from this input, uses execution feedback to guide candidate selection, and repeats the process one annotation at a time for queries with several errors. On NL2SQL-BUGs, TEG reaches 47.3 single-error execution accuracy and 37.0 overall with Qwen2.5-7B-Instruct. Across the model sizes and thinking modes evaluated in the main comparison, TEG outperforms all evaluated baselines on single-error queries, even when the baselines receive the same error-type annotations. With predicted types, TEG stays above direct LLM correction and ErrorLLM on single-error queries.

Comment
32 pages

Closest papers

Nearest neighbours by meaning across the whole index.

Post-SQLFix: Property-Guided Repair of LLM-Generated SQL with Runtime Verification

cs.SE ·2026-10-30 ·Zhihao Guo, Zonghan Wu, Yilei Shao, Haizhou Du, Nan Tang +2 more
similarity 0.73
Abstract

Large Language Models (LLMs) for Text-to-SQL frequently generate queries with syntax errors, invalid identifier references, or violations of SQL scoping and typing rules. Repairing these errors is a program correctness problem, yet existing approaches rely on Execution Feedback (EF), whose symptom-level, dialect-specific error messages force the LLM to speculate on root causes. We identify an Arbitration Paradox in this paradigm: the system relies on unverified models both to propose repairs and to make the final judgment on whether those repairs satisfy the required properties. We resolve this paradox by reformulating Text-to-SQL repair as property-guided program repair. Our framework, Post-SQLFix, canonicalizes SQL into a dialect-aware Canonical Query Structure (CQS) and validates it through a hierarchical static analyzer with three stratified property layers (Syntactic, Context-Free Structural, Context-Sensitive Semantic). The analyzer produces precise, schema-aware violation diagnostics to guide the LLM's repair proposals. A Verify-Before-Apply Temporal Monitor then accepts only repairs that reduce the violation count, do not reintroduce resolved violations, and satisfy the operator contracts. These runtime invariants are bounded by the declared rule family and operator contracts, while a finite retry budget ensures halting. On three established benchmarks (BIRD, Spider, Mini-Dev) spanning over 3,000 evaluation instances, three SQL dialects (SQLite, MySQL, PostgreSQL), and six model configurations, Post-SQLFix resolves 77.8% of the execution errors that EF leaves unrepaired, while requiring less than half as many repair iterations and 30% fewer tokens. Crucially, accepted transitions under the monitor introduce no statically detectable structural regressions in our evaluation, translating checked repair fidelity into absolute accuracy improvements of up to +11.6 percentage points.

0.20

ACTS-SQL: Agentic and Critic-Oriented Tree-Structured SQL Correctness with Large Language Models

cs.AI ·2026-08-15 ·Xinmei Huang, Jie Song, Peng Li, Fuxin Jiang, Jing Zhang +8 more
benchmark similarity 0.71
Abstract

Large Language Models (LLMs) have been increasingly adopted in Text-to-SQL systems, yet SQL errors remain a major obstacle in real-world Text-to-SQL inference pipelines. Existing SQL correction approaches either rely on large-scale, high-quality training data with substantial overhead, or adopt single-path agentic workflows that are brittle to early mistakes and prone to error propagation. To develop a practical SQL correctness system for industrial scenarios, we present a training-free framework that formulates SQL correction as a plan-guided, tree-structured debugging process. By maintaining multiple correction strategies and enabling backtracking, the framework mitigates error accumulation during iterative refinement. We further integrate execution-based verification and clause-level diagnostic tools to support strategy pruning and precise error localization. We evaluate the system on the BIRD-Critic benchmark and observe consistent accuracy gains over strong LLM backbones and representative agent-based baselines, achieving a 9.42% improvement over the previous state-of-the-art method. The framework is also deployed in the Torch Log Service (TLS) of Volcano Engine to support an online Text-to-TLS API. In production, it improves execution accuracy from 36.77% to 53.61% on real user queries with a representative strong LLM backbone (GPT-5). These results demonstrate the effectiveness and stability of our approach in real-world deployments.

0.04

NL2SQL with Iterative Self-Refinement

cs.DB cs.AI ·2026-10-16 ·Anjali Vemula, Akshat Dasula
similarity 0.70
Abstract

Bridging the gap between human intent and machine logic remains a critical challenge in data access for non-technical users, yet Large Language Models (LLMs) remain unreliable: they hallucinate schema elements, generate syntactically invalid queries, or fail to capture user intent. These limitations are especially critical in domains such as insurance, where KPIs, regulatory constraints, and legacy relational systems demand high precision. We introduce a multi-agent, persona-driven NL2SQL framework that combines (1) domain relevance detection, (2) schema-grounded knowledge construction, (3) KPI-aware SQL synthesis, (4) syntax-level self-repair using sqlite execution feedback, and (5) semantic consistency checking via SQL-to-NL reverse engineering. Our system significantly reduces query hallucination and improves semantic fidelity compared to direct LLM prompting. Evaluated on 60 queries (50 from a HuggingFace text-to-SQL dataset and 10 custom domain queries), the system achieves 90% correctness using Mistral across seven coordinated LLM calls showing that this approach can substantially increase the reliability of NL2SQL pipelines in high-stakes environments.

0.19

PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

cs.CL ·2026-10-15 ·Shuo Lin, Jinzhe Li, Chenlu Guo, Yanhan Li, Yuan Wu +1 more
benchmarkcode/data similarity 0.68
Abstract

Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at https://github.com/vegeet/PremiseSQL.

0.32

DexterSQL: Deep Schema Exploration and Rule-based Correction for Text-to-SQL Generation

cs.DB cs.AIcs.CLcs.IR ·2026-08-12 ·Anik Pramanik, Murat Kantarcioglu, Vincent Oria, Shantanu Sharma
code/data similarity 0.68
Abstract · This version of the paper improved the SQL generation algorithm, increasing the system's overall accuracy. For details, please see the paper

Prompting-based (i.e., non-fine-tuning) Text-to-SQL methods, where underlying large language model parameters are not changed for the task, face three problems: (i) relying on coarse-grained schema information that may not reveal the fine-grained relationships needed to distinguish ambiguous columns, (ii) failing to capture recurring SQL-generation failures, and (iii) suffering from omission or hallucination of components in complex questions. This paper develops DexterSQL, a prompting/non-fine-tuning-based Text-to-SQL system that improves SQL generation with three novel components: (i) deep schema explorator that identifies ambiguous columns, analyzes their individual and joint data distributions to uncover their relationships and the distinct role of each, (ii) database-agnostic rule creator that mines mismatches between generated and gold SQL only on the training database and converts them into database-agnostic corrective rules that capture recurring LLM failure patterns; and (iii) multi-path SQL generation that introduces a dependency-tree-based intermediate representation that uses the question's sentence structure to guide its decomposition into an SQL skeleton for final SQL generation. DexterSQL achieves a higher accuracy compared to the state-of-the-art using both open-source/weight and closed-source/weight models. Particularly, DexterSQL shows a high improvement of at least 5.5% using an open-weight model (GPT-OSS-120B) on BIRDDev, with total accuracy 70.4%. DexterSQL also shows better improvement of at least 1.4% using closed-weight models, with total accuracy 72.1% and 72.9% on BIRD-Dev with GPT-4o and GPT-5.2.

0.20

CypherGuard: Database-Grounded Correction for Reliable Text-to-Cypher

cs.DB ·2026-10-30 ·Zebin Chen, Dong Wen, Makbule Gulcin Ozsoy, Wenjie Zhang
similarity 0.68
Abstract

Text-to-Cypher translates natural-language questions into Cypher queries over property graphs. Graph database systems increasingly adopt this translation as a native query interface. Large language models generate fluent Cypher but stay unreliable. On Cypher-Bench a small model answers only 31.4% of questions correctly, and even a strong model plateaus at 71.76%. The dominant failure modes are structural rather than linguistic. For example, a generated query may parse correctly and follow the schema, yet still return zero rows because it traverses a relationship against the stored graph direction. To address this issue, we present CypherGuard, a training-free framework that detects and corrects these structural faults. Our approach uses two core components: a graph-native error taxonomy to guide repair, and a mined rule library to guide query generation. At inference time, CypherGuard first rewrites schema-detectable faults, then re-executes each draft and repairs it only when the query errors or returns nothing. On CypherBench test questions, across seven graphs, CypherGuard reaches 88.50% overall execution accuracy, +25.08% over the same-backbone zero-shot floor, surpassing an RL-fine-tuned 7B specialist and the strongest training-based system that we compare against.

0.25

SafeQL: Search-based Refinement for Safe and Efficient LLM-based Text-to-SQL

cs.DB cs.AI ·2026-08-10 ·Geonho Lee, Min-Soo Kim
top venue similarity 0.67
Abstract · VLDB 2026

Large language models (LLMs) have advanced Text-to-SQL by enabling natural language interfaces to databases without task-specific fine-tuning. However, existing LLM-based systems remain unreliable, often generating SQL queries that are invalid under the database schema, referencing non-existent tables, attributes, functions, or values. Such errors persist because interactions with the database management system (DBMS) are typically limited to error messages, leaving it in a largely passive role during query refinement. This paper proposes SafeQL, \textit{a search-based refinement paradigm that redefines the role of the DBMS as an active guide in the refinement process}. Instead of regenerating entire queries after execution failure, SafeQL interprets DBMS feedback to incrementally repair only the erroneous components. Each refinement step is formulated as a guided search within a \textit{safe query space}, where candidate queries are progressively validated through DBMS execution, thereby converging to an executable query and preventing repeated regeneration of errors. Experiments on the Bird and Spider benchmarks show that SafeQL significantly improves execution accuracy and efficiency compared to regeneration-based methods.

0.40

FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL

cs.DB ·2026-11-12 ·Alfio Massimiliano Gliozzo, Nandana Mihindukulasooriya, Gaetano Rossiello, Erick Lee, Lisa Amini
benchmark similarity 0.66
Abstract · 17 pages

Existing natural-language-to-SQL benchmarks primarily evaluate queries over bounded relational data, leaving streaming semantics underexplored. We introduce FlinkEval, an execution-grounded synthetic benchmark comprising 3,903 question--SQL pairs across 45 industry domains and 45 semantic labels. We create diverse schemas and domain-plausible, temporally consistent data through iterative generation and validation. SQL patterns collected from the human-written NL2SQL pairs and vendor documentation drive quota-balanced generation, and execution with a repair loop on Confluent Cloud Apache Flink produces reference queries covering windowing, temporal joins, pattern matching, nested types, and data manipulation. The corpus includes paired schema-explicit and business-language requests, with annotations for analysis by query pattern, complexity, and execution mode. A seven-level Question Specification Ladder examines semantics-preserving reformulation and underspecified requests augmented with oracle-derived clues. Baseline evaluations combine execution outcomes with adjudicated LLM judgments, revealing challenges in pattern composition and request specification.

0.10