CS Paper Radar

LoStBench: Benchmarking Longitudinal Studies Data Integration

Pratik Pokharel, Oliver Kennedy, Abhi Ramtel, Poonam Kumari, Jeff Good, Katarzyna Kordas, Marianthi Markatou, Andrew Talal

cs.DB ·submitted 2026-11-06 21:27 UTC · v1
benchmarkcode/data

Abstract

A longitudinal study observes the same subjects or variables, repeatedly over an extended period of time, allowing researchers to examine changes, trends, and relationships as they develop. Such studies typically evolve over their life cycles, as improved understanding of the study's domain, or external factors trigger the need for revisions. The resulting heterogeneity among collected data must be resolved before the survey's results may be analyzed. Although this problem is a form of schema matching, it is a form of schema matching that differs significantly from prior workloads explored by the database community. For instance, rather than the short attribute names and data instances that many schema matching tools rely on, a stronger signal in longitudinal studies arises from longer prose questions that these tools are less equipped to handle. This paper documents the joint efforts of a team of computer scientists and domain experts to adapt existing schema matching algorithms to our new use case of longitudinal data integration. We present a case study in the context of a representative workload: The American National Election Studies (ANES) dataset, a nearly eight-decade survey with publicly available data and a canonical domain-expert curated schema matching. Using ANES, we develop LoStBench (Longitudinal Studies Benchmark for data integration), a medium-scale, subject-matter-expert-annotated benchmark. Using LoStBench, we conduct a comprehensive empirical study evaluating state-of-the-art schema matching algorithms, and expose limitations of these algorithms when applied in the context of longitudinal studies. We propose improvements that address some of these limitations, and identify key challenges to motivate future data integration work to address the others.

Closest papers

Nearest neighbours by meaning across the whole index.

Genuine Information Needs of Social Scientists Looking for Data

cs.IR ·2026-09-02 ·Andrea Papenmeier, Thomas Krämer, Tanja Friedrich, Daniel Hienert, Dagmar Kern
publishedhighly cited similarity 0.59
Abstract

Publishing research data is widely expected to increase its reuse and to inspire new research. In the social sciences, data from surveys, interviews, polls, and statistics are primary resources for research. There is a long tradition to collect and offer research data in data archives and online repositories. Researchers use these systems to identify data relevant to their research. However, especially in data search, users' complex information needs seem to collide with the capabilities of data search systems. The search capabilities, in turn, depend to a high degree upon the metadata schemes used to describe the data. In this research, we conducted an online survey with 72 social science researchers who expressed their individual information needs for research data like they would do when asking a colleague for help. We analyzed these information needs and attributed their different components to the categories: topic, metadata, and intention. We compared these categories and their content to existing metadata models of research data and the search and filter opportunities offered in existing data search systems. We found a mismatch between what users have as a requirement for their data and what is offered on metadata level and search system possibilities.

0.56

Benchmarking Hybrid Deep Research Across Database Querying and Web Search

cs.CL ·2026-09-08 ·Ruofan Wu, Peiran Xu, Xiaolong Li, Fan Shu, Soyoung Yoon +7 more
benchmarkcode/data similarity 0.59
Abstract

While autonomous agents have made significant strides in "deep research" by iteratively navigating the open web to synthesize information, real-world problem-solving is rarely confined to a single environment. Complex analytical tasks inherently require agents to weave together evidence from both ambiguous unstructured text (e.g., the open web) and highly precise structured data (e.g., relational databases). However, existing benchmarks evaluate these modalities in isolation, failing to capture the critical "handoff" - the ability to preserve constraints when moving evidence between systems. We introduce HybridDeepResearch, to our knowledge the first deep-research benchmark that requires both web search and SQL to form a complete, verifiable answer. The benchmark contains 380 tool-dependent tasks grounded in LiveSQLBench-Base-Lite databases and public web corpora, validated through automated checks and human review, and covering three reasoning patterns: SQL2S, S2SQL, and Parallel. Evaluations across proprietary and open-weight models under various agentic scaffolds reveal that even state-of-the-art models like GLM-5.2, Claude-Sonnet-4.6 and GPT-5 achieve only about 50-54% Pass@8 on the hard subset. Notably, results show that directional reasoning is substantially more difficult than parallel intersection, highlighting that bridging structured and unstructured information spaces without losing constraints remains a major open challenge for agentic systems. Code and datasets are publicly available at GitHub (https://github.com/Snowflake-AI-Research/HybridDeepResearch) and Hugging Face (https://huggingface.co/datasets/Snowflake/HybridDeepResearch).

0.51

Reading the Data Back: Enriching Variable-Level Metadata for Model-Data Consistency Checks

cs.DL cs.DB ·2026-09-20 ·Eryk Kulikowski
code/data similarity 0.59
Abstract · 21 pages, 8 tables, 2 figures. Submitted to the International Journal on Digital Libraries. Code, metadata profile and census artifacts: htt…

Purpose: Research data repositories often lack variable-level metadata. Model-choice screening, checking whether a reported model suits the values of its outcome variable, also requires summaries of those values and links to the analyses that use them. We ask how repositories can represent and acquire them as metadata with evidence and review histories. Methods: We propose a metadata application profile compatible with the Data Documentation Initiative Cross-Domain Integration (DDI-CDI) model, linking versioned variables and empirical profiles to reported analyses, estimators, and analysis roles. Extraction provenance and review decisions are recorded separately. We evaluate streaming data profiling, deterministic extraction from Stata and R code, and language-model extraction of analysis records from papers. Results: A worked export of one replication package illustrates the profile: it passes shape and interchange checks and answers four queries, one of which returns the analyses that raise no alert. Screening 4,868 replication datasets from six political-science journals links outcomes to profiled columns in 1,440 deposits and flags 965, including 14 of the 21 deposits with a hand-verified linear model on a count, proportion, binary, or ordinal outcome. AI-assisted adjudication yields 55% precision on a near-balanced sample of 119 count and proportion candidates; screening rules were developed on the same corpus. Conclusion: The profile makes analysis-variable relationships queryable while preserving evidence and review history; screening yields review candidates with context. Code, metadata, and measurements are released.

0.23

Back to the Future: The Evolving Universal Relation Model

cs.DB ·2026-10-27 ·Nan Tang, Haixun Wang
similarity 0.58
Abstract

Document backed analytics need databases that can be built as queries arrive. Enterprises increasingly ask analytical queries over messy document collections such as contracts, invoices, reports, emails, policies, and tickets, where the relevant structure and facts are often not known in advance. RAG can retrieve and summarize supporting evidence, but it does not by itself produce reusable, joinable, governed database state. Schema first construction offers the opposite strength: once a database is built, it supports joins, aggregation, constraints, indexes, optimization, and transactions. Yet for document analytics, fixing the schema and extracting all possible fields upfront is both expensive and brittle, because later queries and new documents often surface concepts that were not modeled before. We propose the Evolving Universal Relation Model (EUR), a query first model for document backed analytics. Users ask natural language queries as if the document corpus exposed one conceptual relation over application concepts. Internally, the DBMS turns each query into controlled database evolution: it updates the logical schema when new concepts are needed, extracts and integrates the facts required to answer the query, and decides what durable database state should remain for future queries. Thus, in EUR, a query is not only evaluated; it also helps construct the database. Document backed analytics becomes query first universal access plus workload-driven database schema and instance evolution.

0.26

DataWeave: Deploying Human-LLM Analytics for Exploratory Structured Data Analysis

cs.AI cs.HC ·2026-10-02 ·Raquib Bin Yousuf, Harith Laxman, Vitaliy Shkremetko, Eunice Son, Shambhavi Verma +8 more
similarity 0.58
Abstract

Data journalism, the practice of using data analysis to surface newsworthy stories, depends increasingly on the ability of reporters and investigative journalists to uncover trends, disparities, and accountability narratives. In practice, exploring large structured datasets remains slow and brittle: journalists must navigate hundreds of variables across many datasets over years, understand data coding conventions, and write non-trivial analysis code while hypotheses evolve. Although LLMs are often touted as "ask in English, get SQL/answers," real newsroom workflows expose recurring failures, e.g., schema mismatches and drift, misread domain semantics and units, and silent assumptions. We present DataWeave, a system that addresses these needs by combining conversational interaction, schema grounding, analytical planning, and executable query generation to support exploratory analysis over structured data. Rather than treating LLMs as autonomous answer engines, DataWeave frames them as interactive partners whose outputs can be inspected, corrected, and steered as hypotheses shift. We present a case study with professional journalists using our system to analyze the U.S. Department of Education's Integrated Postsecondary Education Data System (IPEDS), a high-stakes public dataset with substantial domain semantics and frequent schema updates. We also report how deployment experience and iterative refinement shaped the current DataWeave architecture and its analytical workflow. Our findings distill design principles and deployment lessons for trustworthy human-LLM collaboration in structured data analysis.

0.10

Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?

cs.IR cs.AIcs.CLcs.DB ·2026-09-07 ·Daniel Gomm, Maarten de Rijke, Madelon Hulsebos
survey similarity 0.58
Abstract

Democratizing access to the knowledge held in large corpora of tables such as data lakes is emerging as a central research challenge. Research in this space is advancing and broadening in scope, increasingly supplying the components to satisfy a person's insight need end-to-end. Yet these efforts remain fragmented across communities that frame the problem under their own conventions, such as table question answering, text-to-SQL, and data analysis agents, with works six times as likely to cite within the same task label as across labels. To bring these communities onto common ground, we establish a holistic framework for this pursuit, which we refer to as Open Tabular Insight Extraction (OpenTI). We formalize OpenTI from first principles around the analytical knowledge a person needs, the procedure for deriving it from a corpus of tables, and how well a result serves the person who sought it. In doing so we consolidate frameworks and terminology across information retrieval, natural language processing, machine learning, databases, and human-computer interaction, and apply this grounding in a systematic review and analysis of systems and benchmarks that work towards OpenTI. We find that current systems do not cover the end-to-end scope of OpenTI, mainly focusing on the analysis itself, and that benchmarks are largely unfit for evaluations in an open setting as inputs presuppose knowledge of tables, and validation mechanisms do not match the setup. Finally, we distill a research agenda towards OpenTI systems, evaluation, and interaction paradigms that surface the insights users need. An interactive companion to our paper is available at https://open-tabular-insight-extraction.github.io.

0.27

AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research

cs.AI ·2026-09-15 ·Bernie Boscoe, Srinath Saikrishnan, Vikram Seenivasan, Jack Stark, Andrew Lizarraga +4 more
benchmarkcode/data similarity 0.58
Abstract · 14 pages, 1 figure

Scientific research increasingly relies on large, heterogeneous data sources, motivating interest in retrieval-augmented generation (RAG) systems that provide natural language access to scientific knowledge and research workflows. Researchers are exploring the viability of these systems as natural language interfaces for document search and for generating analysis code and pipeline components. At the same time, concerns about data privacy and control over research infrastructure have motivated interest in open-weight models and open-source deployments hosted within research institutions. In astronomy, this development follows a long history of computational infrastructure development, from archival databases and Structured Query Language (SQL)-based systems to large language model (LLM)-assisted research tools. This paper presents a domain-expert evaluation of faithfulness for AquiLLM, an open-weight, offline RAG-LLM platform designed to support scientific research groups in the use and preservation of tacit and formal knowledge. We define faithfulness as the extent to which generated responses remain grounded in retrieved scientific context without unsupported claims or omissions. We report results from an astronomy case study evaluating AquiLLM across retrieval and scientific analysis tasks. AquiLLM performs most reliably on explicit retrieval-oriented questions grounded in the RAG collection, while faithfulness degrades for queries requiring synthesis or ambiguity resolution. These results highlight both the promise and limitations of open-weight RAG-LLM systems for scientific research and demonstrate the importance of domain-expert evaluation beyond standard benchmark leaderboards.

0.52

Data.Readme: Documenting Table Relationships for Diverse Data Tasks

cs.DB ·2026-10-18 ·Jiaming Liang, Yeye He, Haoyu Dong, Zachary Ives, Surajit Chaudhuri
similarity 0.58
Abstract

Large language models (LLMs) can perform diverse data tasks when given a small set of relevant tables. However, for complex tasks requiring multiple tables over large data lakes, identifying relevant tables for the task becomes a prerequisite, but remains challenging. This is because there are often many tables in a data lake with similar table and schema names, and limited additional semantic context, thereby forcing agents to spend substantial time exploring tables at online query time, greatly affecting models' ability to perform tasks accurately, while increasing cost/latency substantially. We present Data.Readme, a novel table documentation approach that uses lightweight offline processing to infer and index rich \emph{table-to-table relationships}, including confusable, joinable, and union-able tables. We further develop a suite of agentic tools for LLMs to work with the index, to help agents retrieve relevant tables and perform tasks much more efficiently and effectively. Finally, we study how to optimize our index construction with a budget constraint. On data lakes containing up to 12K tables, Data.Readme improves task accuracy by as much as 40 percentage points across a range of data discovery, data analysis, and data science tasks, outperforming diverse baselines, including powerful coding agents with unlimited time and monetary budgets. We also show that equipping existing methods (such as coding agents) with Data.Readme can not only lead to consistent quality gains, but also significantly reduce their latency by up to 6x, demonstrating the potential of Data.Readme.

0.19