CS Paper Radar

TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databases

Fei Lyu, Zhiyi Peng, Jiaming Liu, Yixuan Yang, Changjian Chen, Zhuo Tang, Jiapeng Zhang, Kenli Li

cs.CL ·submitted 2026-09-28 09:55 UTC · v1
benchmarkcode/data

Abstract

Large language models (LLMs) have significantly advanced natural language querying over relational databases, yet their ability to query time-series databases (TSDBs) remains largely unassessed. Existing benchmarks fail to adequately capture the non-unified query syntaxes, diverse application domains, and unique time-specific query intents inherent to TSDBs. To address this gap, we introduce TQTS-BENCH, a multi-syntax benchmark for evaluating text-to-query capabilities over TSDBs. TQTS-BENCH contains 6,125 high-quality question-answering (QA) pairs spanning 97 TSDBs, 23 distinct query syntaxes, 22 application domains, and 4 types of time-specific query intents. It is constructed through a human-centric AI-assisted workflow, where all QA pairs are carefully reviewed and revised by domain experts to ensure quality and correctness. Extensive evaluations of advanced LLMs and state-of-the-art text-to-query methods reveal challenges in querying TSDBs. Even the best-performing model evaluated, Claude-Opus-5, achieves only 48.98% execution accuracy, while humans reach 87.34%. Error analysis reveals that this performance gap mainly stems from the heterogeneous query syntaxes across different TSDBs, misinterpretation of time-specific intents, and incorrect schema linking. These findings highlight new opportunities to narrow the gap between current LLM capabilities and the requirements of TSDB queries in real-world applications. The benchmark is available at: https://anonymous.4open.science/r/TQTS-Bench-00CD.

Closest papers

Nearest neighbours by meaning across the whole index.

BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

cs.AI cs.CL ·2026-08-29 ·Yunfan Zhou, Qiming Shi, Yizhou Yang, Di Weng, Yingcai Wu
top venuebenchmarkcode/data similarity 0.74
Abstract · Accepted at Findings of the Association for Computational Linguistics: EMNLP, 2026

While recent Large Language Model (LLM)-based text-to-SQL systems achieve impressive performance on standard benchmarks, they struggle when user queries implicitly rely on domain-specific knowledge, such as business logic, data conventions, and analytical practices, that is neither captured by the schema nor explicitly stated in the natural language question. Historical SQL query logs offer a valuable source of such knowledge, yet existing benchmarks do not adequately support evaluation of history-driven approaches. To address this gap, we introduce BIRD-History, a benchmark consisting of 1,393 tasks across 11 databases, designed to evaluate text-to-SQL systems' ability to ground underspecified natural language questions using historical SQL scripts. Each task is annotated with ground-truth labels specifying which historical queries contain relevant knowledge and which SQL clauses encode it, enabling systematic evaluation of both retrieval effectiveness and knowledge utilization. Alongside the benchmark, we propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation. The retriever integrates seamlessly into existing few-shot text-to-SQL pipelines without requiring prompt modifications. Experiments demonstrate consistent improvements across four text-to-SQL systems, highlighting the value of leveraging historical query logs for handling underspecified queries. Dataset and code are open-sourced on https://github.com/zjuidg/BIRD-History.

0.44

TSSDA: An Autonomous Agent for Constructing a Time-Series Text-to-SQL Benchmark

cs.LG cs.DB ·2026-11-01 ·Zhihan Zheng, Shaoxu Song, Haitao Yuan, Minxiao Chen
benchmark similarity 0.70
Abstract

As agents increasingly undertake data querying, Text-to-SQL serves as a foundational capability for database access. However, for text-to-SQL in time-series databases, high-quality benchmarks and large-scale training corpora remain scarce, limiting the systematic evaluation and domain adaptation of existing models and agents. Existing automated SQL generation approaches lack sufficient autonomy, leaving key challenges unresolved. First, numerous combinations of temporal syntax and functions complicate comprehensive coverage of valid queries. Second, as the corpus grows, generating valuable and diverse new data becomes increasingly challenging. Third, schema and data heterogeneity hinder reuse of temporal query structures, limiting generation efficiency. To address these challenges, we propose TSSDA, a Time-Series SQL (TSSQL) Generation Data Agent framework, and construct TERN, a comprehensive Time-sERies text-to-SQL beNchmark. First, to achieve broad coverage of valid queries, we systematically model TSSQL syntax and semantics to construct a data generation space. Next, to explore the space and improve diversity, we develop a Data Agent that autonomously orchestrates data generation. The agent comprises a planning and decision-making module and a query generation module, with online adaptation for continuous self-improvement. Finally, we introduce a temporal-aware memory that enables adaptive reuse of experience across heterogeneous data instances, improving generation efficiency. Comprehensive evaluations on TERN reveal the limitations of existing methods in performing Text-to-TSSQL tasks. Furthermore, extensive experiments demonstrate that TSSDA substantially improves coverage, diversity, and complexity.

0.18

RDL-Bench: A Benchmark for Task-Aware Question Answering over Relational Databases

cs.LG ·2026-10-15 ·Yao Cheng, Siqiang Luo
top venuebenchmark similarity 0.69
Abstract · This paper has been accepted by ICDE 2027

Relational databases (RDBs) remain the primary infrastructure for storing and managing structured data across enterprise and scientific domains. Natural-language (NL) interfaces to RDBs aim to let users ask questions without writing SQL, but real user needs go beyond SQL-style lookup: some questions require reasoning over the observable database state, while others require predictive inference from historical relational records under temporal constraints. Existing benchmarks address these goals in isolation. To address this gap, we introduce RDL-Bench, a benchmark for task-aware question answering (QA) over real RDBs. RDL-Bench organizes diverse user questions into reasoning-oriented questions, whose answers are determined by the current database state, and prediction-oriented questions, whose answers depend on future outcomes under temporal constraints. Each instance contains an NL question, task metadata, schema information, temporal context, either executable evidence or target-construction logic, and denotation-level ground truth. RDL-Bench supports two evaluation settings: native solvers under their intended capabilities, and task-aware modular systems where a router dispatches each question to an appropriate execution module.

0.35

FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL

cs.DB ·2026-11-12 ·Alfio Massimiliano Gliozzo, Nandana Mihindukulasooriya, Gaetano Rossiello, Erick Lee, Lisa Amini
benchmark similarity 0.69
Abstract · 17 pages

Existing natural-language-to-SQL benchmarks primarily evaluate queries over bounded relational data, leaving streaming semantics underexplored. We introduce FlinkEval, an execution-grounded synthetic benchmark comprising 3,903 question--SQL pairs across 45 industry domains and 45 semantic labels. We create diverse schemas and domain-plausible, temporally consistent data through iterative generation and validation. SQL patterns collected from the human-written NL2SQL pairs and vendor documentation drive quota-balanced generation, and execution with a repair loop on Confluent Cloud Apache Flink produces reference queries covering windowing, temporal joins, pattern matching, nested types, and data manipulation. The corpus includes paired schema-explicit and business-language requests, with annotations for analysis by query pattern, complexity, and execution mode. A seven-level Question Specification Ladder examines semantics-preserving reformulation and underspecified requests augmented with oracle-derived clues. Baseline evaluations combine execution outcomes with adjudicated LLM judgments, revealing challenges in pattern composition and request specification.

0.10

Generating Query Context for Relational Databases

cs.DB ·2026-08-12 ·Alekh Jindal, Jyoti Pandey, Christina Pavlopoulou, Ronith PR, Sharath Prakash +3 more
similarity 0.68
Abstract

Relational databases are the systems of record for business applications, and there is growing demand to query them through natural language interfaces. A key challenge is that AI models need appropriate query context, i.e., sample questions paired with their corresponding data-model fragments, to generate accurate SQL. Today, creating this context is a manual, time-consuming process that requires expertise in both SQL and the database schema, leading to a cold start problem for new databases and an ongoing maintenance burden as schemas and query patterns evolve. We present an automated approach for generating query context for relational databases. Our method defines query flows that capture common patterns of data retrieval and analysis questions, systematically generates data models by traversing these flows, and instantiates them with specific values sampled from the database using weighted strategies that maximize diversity and coverage. The resulting question--data-model pairs can be used to guide natural language interfaces in accurately querying relational databases. We report on our experience deploying this approach across more than 50 real-world databases connected to Tursio.

0.04

From Data Querying to Data Investigations: Rethinking Natural Language Interfaces for Databases

cs.DB ·2026-09-03 ·Fabian Wenz, Zixuan Chen, Carsten Binnig
benchmark similarity 0.68
Abstract · 6 pages, 4 figures, 2 tables

Natural language (NL) interfaces to databases have been optimized for the wrong problem. The dominant Text-to-SQL paradigm assumes that users ask questions that can be answered by single SQL queries. In practice, however, users seek assistance with solving data problems. This requires searching a database by sequences of SQL queries while reasoning over intermediate results instead of just running one SQL query. This paper therefore introduces a new paradigm for NL interfaces to data, which we call data investigations. We present D^2, a first prototype of a data investigation system that embodies this vision by autonomously searching, reasoning over, and collecting data to solve data problems. Using a newly constructed benchmark based on the Murder Mystery dataset, we demonstrate the potential of D^2 for tasks that require data investigations with evidence-backed decisions, extending beyond the capabilities of traditional single-query question answering.

0.22

TL;QR: Querying Temporal Discursive Patterns in Textual Corpora

cs.CL cs.CYcs.DBcs.LO ·2026-10-11 ·François Hublet, Vladimir Krsmanovic, Jesus Solano, Ahmed Bouhoula, Ana Oliveira da Costa
code/datahot direction similarity 0.67
Abstract

Many questions about large textual corpora concern patterns that unfold over time, such as a political group whose stance on a topic drifts from support to criticism over several years of debates. We present TL;QR (Too Long; Query with logical temporal Reasoning), a declarative query language and open-source engine for finding such temporal discursive patterns. In TL;QR, a single query combines natural-language predicates, whose truth is judged by a Large Language Model (LLM), with regular expressions and the operators of a temporal logic. The LLM answers only local questions about individual text passages; a symbolic engine then aggregates these judgments over time and links every match to the recorded passages and predicate decisions behind it. With this division of labor, the temporal and relational reasoning stays exact and auditable even over long time horizons. In addition, when a query includes a regular-expression pre-filter, only a small fraction of the corpus ever reaches the LLM. On a suite of controlled synthetic benchmarks with temporally and semantically complex patterns, TL;QR is substantially more accurate than LLM-only baselines. In a case study on ten years of European Parliament debates (442,593 multilingual speeches), TL;QR identifies candidate shifts in the political groups' stances on the GDPR privacy regulation.

0.37

Guided Table Retrieval for Structured Data Search

cs.DB ·2026-08-12 ·Alekh Jindal, Jyoti Pandey, Christina Pavlopoulou, Ronith PR, Sharath Prakash +3 more
benchmark similarity 0.67
Abstract

Answering natural language questions over structured databases requires identifying the relevant tables and determining how to join them---a task that demands both schema knowledge and semantic understanding of the user's intent. We present guided table retrieval, a four-phase pipeline that combines deterministic grounding via hash-based predictors, structural exploration of join-graph reachability, LLM-powered disambiguation of sources and targets, and algorithmic merging into minimal, topologically ordered join trees. By decomposing the problem into phases with distinct responsibilities--- determinism, coverage, semantic reasoning, and coherence--- the pipeline avoids the brittleness of end-to-end LLM approaches while leveraging LLMs where their contextual judgment is most needed. We evaluate on BIRD-DEV and the enterprise-scale BEAVER benchmark, achieving 94% and 70% precision respectively, with 92% and 53% F1---substantially outperforming existing baselines on precision and F1 while producing exact join trees that can be directly consumed by downstream query compilers.

0.04