CS Paper Radar

From Data Querying to Data Investigations: Rethinking Natural Language Interfaces for Databases

Fabian Wenz, Zixuan Chen, Carsten Binnig

cs.DB ·submitted 2026-09-03 14:17 UTC · v1
benchmark

Abstract

Natural language (NL) interfaces to databases have been optimized for the wrong problem. The dominant Text-to-SQL paradigm assumes that users ask questions that can be answered by single SQL queries. In practice, however, users seek assistance with solving data problems. This requires searching a database by sequences of SQL queries while reasoning over intermediate results instead of just running one SQL query. This paper therefore introduces a new paradigm for NL interfaces to data, which we call data investigations. We present D^2, a first prototype of a data investigation system that embodies this vision by autonomously searching, reasoning over, and collecting data to solve data problems. Using a newly constructed benchmark based on the Murder Mystery dataset, we demonstrate the potential of D^2 for tasks that require data investigations with evidence-backed decisions, extending beyond the capabilities of traditional single-query question answering.

Comment
6 pages, 4 figures, 2 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Generating Query Context for Relational Databases

cs.DB ·2026-08-12 ·Alekh Jindal, Jyoti Pandey, Christina Pavlopoulou, Ronith PR, Sharath Prakash +3 more
similarity 0.73
Abstract

Relational databases are the systems of record for business applications, and there is growing demand to query them through natural language interfaces. A key challenge is that AI models need appropriate query context, i.e., sample questions paired with their corresponding data-model fragments, to generate accurate SQL. Today, creating this context is a manual, time-consuming process that requires expertise in both SQL and the database schema, leading to a cold start problem for new databases and an ongoing maintenance burden as schemas and query patterns evolve. We present an automated approach for generating query context for relational databases. Our method defines query flows that capture common patterns of data retrieval and analysis questions, systematically generates data models by traversing these flows, and instantiates them with specific values sampled from the database using weighted strategies that maximize diversity and coverage. The resulting question--data-model pairs can be used to guide natural language interfaces in accurately querying relational databases. We report on our experience deploying this approach across more than 50 real-world databases connected to Tursio.

0.04

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database

cs.CL cs.AIcs.DB ·2026-09-19 ·Vigneshwar Ravi Rao, Rupesh Swarnakar, Fayeq Jeelani Syed†
benchmark similarity 0.71
Abstract

Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL agent that closes this gap end to end: a six-node LangGraph StateGraph checks question relevance, fetches the live schema, generates PostgreSQL, validates it with a dry run, retries on failure, executes the query, and narrates the result set in plain English. The agent uses prompt engineering only; no model was fine-tuned. We evaluated it on the Chicago Crime dataset (approximately 8.5 million records, 22 attributes) against a hand-built benchmark of 100 natural language questions with ground-truth SQL, stratified into 30 Easy, 40 Medium and 30 Hard items. Comparing two prompt revisions of the same agent, the revised system (V2) reached a Valid SQL Rate of 93% (from 87%), an Execution Accuracy of 60% under a hybrid relational equivalence metric (from 47%; 19% from 12% under strict JSON matching), and a mean Synthesis Quality of 4.34 out of 5 (from 3.91). The single largest driver was removing a LIMIT 10 instruction from the system prompt, which had been truncating multi-row answers. Error analysis attributes the residual failures to relevance-checker false rejections, ambiguous question semantics, and free-tier API rate limits rather than to the language generation step. We report no comparison against an external baseline system or a public benchmark; the study is a single-model engineering evaluation.

0.12

DataWeave: Deploying Human-LLM Analytics for Exploratory Structured Data Analysis

cs.AI cs.HC ·2026-10-02 ·Raquib Bin Yousuf, Harith Laxman, Vitaliy Shkremetko, Eunice Son, Shambhavi Verma +8 more
similarity 0.70
Abstract

Data journalism, the practice of using data analysis to surface newsworthy stories, depends increasingly on the ability of reporters and investigative journalists to uncover trends, disparities, and accountability narratives. In practice, exploring large structured datasets remains slow and brittle: journalists must navigate hundreds of variables across many datasets over years, understand data coding conventions, and write non-trivial analysis code while hypotheses evolve. Although LLMs are often touted as "ask in English, get SQL/answers," real newsroom workflows expose recurring failures, e.g., schema mismatches and drift, misread domain semantics and units, and silent assumptions. We present DataWeave, a system that addresses these needs by combining conversational interaction, schema grounding, analytical planning, and executable query generation to support exploratory analysis over structured data. Rather than treating LLMs as autonomous answer engines, DataWeave frames them as interactive partners whose outputs can be inspected, corrected, and steered as hypotheses shift. We present a case study with professional journalists using our system to analyze the U.S. Department of Education's Integrated Postsecondary Education Data System (IPEDS), a high-stakes public dataset with substantial domain semantics and frequent schema updates. We also report how deployment experience and iterative refinement shaped the current DataWeave architecture and its analytical workflow. Our findings distill design principles and deployment lessons for trustworthy human-LLM collaboration in structured data analysis.

0.10

RDL-Bench: A Benchmark for Task-Aware Question Answering over Relational Databases

cs.LG ·2026-10-15 ·Yao Cheng, Siqiang Luo
top venuebenchmark similarity 0.70
Abstract · This paper has been accepted by ICDE 2027

Relational databases (RDBs) remain the primary infrastructure for storing and managing structured data across enterprise and scientific domains. Natural-language (NL) interfaces to RDBs aim to let users ask questions without writing SQL, but real user needs go beyond SQL-style lookup: some questions require reasoning over the observable database state, while others require predictive inference from historical relational records under temporal constraints. Existing benchmarks address these goals in isolation. To address this gap, we introduce RDL-Bench, a benchmark for task-aware question answering (QA) over real RDBs. RDL-Bench organizes diverse user questions into reasoning-oriented questions, whose answers are determined by the current database state, and prediction-oriented questions, whose answers depend on future outcomes under temporal constraints. Each instance contains an NL question, task metadata, schema information, temporal context, either executable evidence or target-construction logic, and denotation-level ground truth. RDL-Bench supports two evaluation settings: native solvers under their intended capabilities, and task-aware modular systems where a router dispatches each question to an appropriate execution module.

0.35

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

cs.AI cs.CL ·2026-08-29 ·Yunfan Zhou, Qiming Shi, Yizhou Yang, Di Weng, Yingcai Wu
top venuebenchmarkcode/data similarity 0.70
Abstract · Accepted at Findings of the Association for Computational Linguistics: EMNLP, 2026

While recent Large Language Model (LLM)-based text-to-SQL systems achieve impressive performance on standard benchmarks, they struggle when user queries implicitly rely on domain-specific knowledge, such as business logic, data conventions, and analytical practices, that is neither captured by the schema nor explicitly stated in the natural language question. Historical SQL query logs offer a valuable source of such knowledge, yet existing benchmarks do not adequately support evaluation of history-driven approaches. To address this gap, we introduce BIRD-History, a benchmark consisting of 1,393 tasks across 11 databases, designed to evaluate text-to-SQL systems' ability to ground underspecified natural language questions using historical SQL scripts. Each task is annotated with ground-truth labels specifying which historical queries contain relevant knowledge and which SQL clauses encode it, enabling systematic evaluation of both retrieval effectiveness and knowledge utilization. Alongside the benchmark, we propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation. The retriever integrates seamlessly into existing few-shot text-to-SQL pipelines without requiring prompt modifications. Experiments demonstrate consistent improvements across four text-to-SQL systems, highlighting the value of leveraging historical query logs for handling underspecified queries. Dataset and code are open-sourced on https://github.com/zjuidg/BIRD-History.

0.62

Carnot: Interpretable, Interactive, and Optimized Execution of Deep Research Queries

cs.DB cs.AI ·2026-08-10 ·Matthew Russo, Yash Agarwal, Tianyu Li, Zhuohan Gu, Michael Cafarella +3 more
top venue similarity 0.69
Abstract · 4 pages, 2 figures, published as a demo paper in VLDB 2026

Enterprises increasingly seek to query data lakes using natural language via AI-driven tools like semantic operators or deep research agents. However, the latter operates as an opaque black box, hiding its intermediate reasoning and data retrieval steps, and failing to expose controls for managing API costs and execution latency. Meanwhile, the former can be prohibitively expensive for enterprise-scale data lakes. Consequently, analysts using these systems lack the agency to intercept hallucinated premises, verify intermediate results, or correct the system's trajectory. We present Carnot, an interactive execution engine for AI-driven analytics. Carnot compiles natural language requests into physical execution graphs and surfaces them through an interactive notebook interface. Rather than waiting blindly for a final output, users can critique the plan, incrementally execute operators, inspect intermediate data, or directly edit the underlying code or semantic operator instructions. Carnot's query optimizer will optimize the query with respect to cost or latency constraints provided by the user. Our demo will showcase how Carnot helps users achieve efficient and verifiable insights on workloads motivated by real enterprise use cases.

0.72

TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databases

cs.CL ·2026-09-28 ·Fei Lyu, Zhiyi Peng, Jiaming Liu, Yixuan Yang, Changjian Chen +3 more
benchmarkcode/data similarity 0.68
Abstract

Large language models (LLMs) have significantly advanced natural language querying over relational databases, yet their ability to query time-series databases (TSDBs) remains largely unassessed. Existing benchmarks fail to adequately capture the non-unified query syntaxes, diverse application domains, and unique time-specific query intents inherent to TSDBs. To address this gap, we introduce TQTS-BENCH, a multi-syntax benchmark for evaluating text-to-query capabilities over TSDBs. TQTS-BENCH contains 6,125 high-quality question-answering (QA) pairs spanning 97 TSDBs, 23 distinct query syntaxes, 22 application domains, and 4 types of time-specific query intents. It is constructed through a human-centric AI-assisted workflow, where all QA pairs are carefully reviewed and revised by domain experts to ensure quality and correctness. Extensive evaluations of advanced LLMs and state-of-the-art text-to-query methods reveal challenges in querying TSDBs. Even the best-performing model evaluated, Claude-Opus-5, achieves only 48.98% execution accuracy, while humans reach 87.34%. Error analysis reveals that this performance gap mainly stems from the heterogeneous query syntaxes across different TSDBs, misinterpretation of time-specific intents, and incorrect schema linking. These findings highlight new opportunities to narrow the gap between current LLM capabilities and the requirements of TSDB queries in real-world applications. The benchmark is available at: https://anonymous.4open.science/r/TQTS-Bench-00CD.

0.20

FALCON: A Model and Dataset Agnostic Framework for Synthetic Data Generation for NL2SQL Pairs

cs.CL cs.DBcs.LG ·2026-10-02 ·Darian Lee, Shannon Rumsey, Jack St. Clair, Xinyi Tang, Aditya Bansal +1 more
publishedbenchmark similarity 0.68
Abstract · Accepted to AKBC Workshop, EMNLP

Relational databases are among the most widely deployed forms of structured knowledge, and natural language access to them requires grounding language onto schema entities and relations while handling the ambiguity inherent in how people phrase requests. Existing synthetic NL-to-SQL data generation methods largely ignore this ambiguity and produce oversimplified queries that fail to prepare models for the complexity of real-world structured knowledge access. We present FALCON, a framework that generates realistic, ambiguity-aware NL-to-SQL data matching the complexity of challenging real-world benchmarks, at low cost using compact open models. Our approach combines reserved-word SQL seeding and persona-based prompting to generate structurally complex queries, while alignment-based filtering preserves difficulty by distinguishing genuinely incorrect examples from complex but valid queries. Human evaluation confirms consistent high quality across model sizes, and our generated data exceeds existing benchmarks in both SQL complexity and natural language richness. Difficulty-stratified analysis shows models trained on FALCON data increasingly outperform baseline-trained models as query complexity increases, validating our pipeline's success in generating challenging training data. When combined with a small proportion of existing benchmark data, mixed training recovers performance on simpler queries while preserving these advantages on complex ones. The model- and database-agnostic design enables organizations to generate high-complexity NL-to-SQL training data locally without external APIs.

0.20