CS Paper Radar

fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets

Sanket Mhatre, Sumit Gulwani, Vu Le, Yasharth Bajpai, Gust Verbruggen

cs.LG ·submitted 2026-10-17 14:49 UTC · v1
benchmark

Abstract

Spreadsheets are among the most widely used tools for computations, and formulas are their core building blocks. Besides agentic spreadsheet tools, proactive formula suggestions enable significant productivity gains and are now available in popular spreadsheet software (Microsoft Excel and Google Sheets). Such suggestions pose interesting challenges: the right context must be collected and represented for a machine learning model, which must then be able to write a correct formula, in a way that is fast and cheap enough to trigger on every = (or subsequent keystrokes for auto-completion) without taking users out of their editing flow. Surprisingly, there is no public benchmark for formula suggestions on real spreadsheets. In this paper, we therefore (1) introduce fxBench as a manually curated benchmark of 503 formula suggestion tasks based on Sheetpedia, (2) break down the formula suggestion problem into four capabilities (gather context, understand context, understand intent, write formula) that a formula suggestion system requires, and (3) evaluate existing and new approaches to understand bottlenecks in these capabilities that pose interesting research directions. Specifically, we annotate each benchmark with the cells that describe intent to determine the theoretical performance limit of a given context selection strategy, we evaluate performance of small and large models on different representations of context, including a dense format with high theoretical coverage at low token counts, and we fine-tune smaller language models to show that we can learn to understand such dense formats. Additionally, we perform quantitative and qualitative analysis on failure modes to understand bottlenecks in the formula writing capability; the best configuration only achieved 58.8% correct suggestions.

Closest papers

Nearest neighbours by meaning across the whole index.

A Compact Formula-Aware Spreadsheet Abstraction for Large Language Models

cs.SE cs.AI ·2026-10-30 ·Tie Cheng
published similarity 0.75
Abstract

Large language models (LLMs) are increasingly used to assist spreadsheet users with tasks such as spreadsheet model construction, formula generation, and data analysis, yet their interaction with real-world spreadsheets remains limited by the mismatch between the structure of spreadsheets and the sequential textual representations typically provided to LLMs. We propose a compact formula-aware spreadsheet abstraction that represents large homogeneous ranges through their positions, types, R1C1 formula patterns, and selected boundary values. An efficient algorithm detects and structures these ranges before a textual serialization is supplied to an LLM. The abstraction is provided to the LLM before detailed spreadsheet content is submitted, improving the accuracy and efficiency of spreadsheet processing while reducing token consumption. We evaluate the method on 50 spreadsheet-query cases. The evaluation shows that the abstraction alone was sufficient to answer some queries. When additional spreadsheet content is required, the abstraction reduces the amount of data that must be submitted to the LLM, thereby reducing token consumption. The abstraction layer also improves the reliability and accuracy of spreadsheet processing. We conclude by discussing further techniques for combining spreadsheet static analysis with artificial intelligence to improve the reliability, accuracy, and efficiency of AI assisted spreadsheet tasks.

0.26

A Taxonomy and Taxonomy-Guided Benchmark Reduction for Spreadsheet Tasks

cs.SE ·2026-11-16 ·Bogdan Vasilescu, Emerson Murphy-Hill, Anikesh Parashar, Aayush Kumar, Gustavo Soares +2 more
surveybenchmark similarity 0.68
Abstract

Spreadsheets let millions of end users without programming training do computational work, and teams are increasingly building AI agents to help them. Evaluating each candidate agent configuration across many tasks is costly. Benchmark reduction can lower this cost by selecting a small subset of tasks whose outcomes predict full-benchmark scores, but selecting tasks only for predictive accuracy can leave whole categories of spreadsheet work untested. We contribute a taxonomy of spreadsheet tasks, grounded in a review of benchmarks and refined with the Excel Agent product team at Microsoft, and a benchmark-reduction method that favors taxonomic diversity among similarly predictive choices. On benchmark logs from Microsoft, taxonomy-guided selection broadens coverage at little or no cost in predictive accuracy and often improves prediction on configurations excluded from selection. Broader coverage can therefore support, rather than compromise, predictive accuracy. Generalization remains limited across time: subsets selected using little recent history predict poorly for configurations much later in time. Beyond benchmark reduction, the taxonomy provides a reusable framework for auditing gaps in task coverage, as teams at Microsoft have done.

0.29

MBABench-2.0: Evaluating AI Agents on Constructing Financial Spreadsheets Beyond Simple Verifiability

cs.SE cs.AI ·2026-10-22 ·Julian Poeltl, Thomson Yen, Patrick Shea, Shubham Anil Varma, Dhruv Rastogi +3 more
benchmark similarity 0.67
Abstract

Spreadsheets are the lingua franca of modern finance, driving valuation, forecast, audit, and many other core workflows of the industry. As LLM agents advance to automate knowledge work, spreadsheet construction is a natural and pertinent testbed. However, most existing spreadsheet benchmarks evaluate agents on fill-in-the-blank style tasks, where an agent performs minor edits in otherwise complete spreadsheets. This protocol fails to capture the most central and time-consuming part of the spreadsheet workflow: the modeling decisions, format design, and sheet layout regular knowledge workers need to toil through to build a workbook from scratch. We introduce MBABench-2.0, a benchmark of 101 original spreadsheet construction tasks, the hardest of which is estimated to take 33 hours for a modeler with 3+ years of experience. Unlike coding or math, where a unit test or a final answer can verify an agent's output, the quality of a financial spreadsheet that concerns readability, maintainability, or auditability, is not so simply verifiable. To holistically evaluate spreadsheet construction, we introduce 129 criteria spanning 12 categories that operationalize quality beyond the accuracy of the spreadsheet. Evaluating over 20 agents across 8 models and 4 harness types, we find that frontier agents rarely construct spreadsheets that are both accurate and meet our quality criteria: the top agent achieves a pass rate of only 16% on the benchmark. The benchmark demonstrates a substantial gap in spreadsheet building capability of current agents.

0.22

Walk Before You Run: The Importance of Data Exploration for Data Analysis Agents

cs.DB cs.AI ·2026-08-17 ·Yike Yuan, Virum Ranka, Tina Lasisi, Lin Ma
publishedbenchmark similarity 0.64
Abstract · 9 pages, 6 figures. Accepted to VLDB 2026 Workshop: DASHSys: Systems for Data-centric Agents with Human-in-the-loop

LLM-based data-analysis tools are increasingly used to help users analyze messy spreadsheets and workbooks, from answering questions over uploaded files to generating code, summaries, and visualizations. These systems are often evaluated by the correctness of their final downstream answers. However, reliable data analysis also depends on an earlier step: understanding what the dataset contains before solving the requested task. For complex workbooks, this Data Exploration step includes identifying the logical tables behind physical sheets, interpreting column semantics, recovering keys and relationships, and detecting quality issues. In current tools and benchmarks, this step is usually left implicit, creating a gap between downstream task performance and the dataset understanding needed for reliable, human-checkable analysis. Our key contribution is to identify this overlooked gap, make Data Exploration a first-class evaluation target, and show through downstream experiments that stronger Data Exploration support improves task performance. To evaluate dataset understanding directly, we introduce two benchmark settings: a real multi-sheet workbook benchmark based on a Vitamin D study dataset, and an extension of DSBench with schema-fixed Data Exploration artifacts. In both settings, systems are evaluated by the quality of a structured artifact capturing tables, columns, semantic roles, relationships, and profiling signals. Our results show that strong LLMs and data-analysis agents still miss important logical structure even when they read spreadsheet content. Furthermore, explicit Data Exploration support often improves downstream correctness, suggesting it should be treated as a first-class, inspectable stage in LLM data-analysis workflows and a natural human-in-the-loop checkpoint where domain experts can review and correct the artifact before downstream analysis proceeds.

0.41

FICO: Find-Then-Compute for Corpus-Level Spreadsheet Question Answering

cs.IR ·2026-10-02 ·Sandarsita Guntupalli, Lu He, Kang Li
similarity 0.64
Abstract

Question answering over spreadsheet collections requires finding the correct workbook and computing over complete tables. We introduce Find-then-Compute (FiCo), which retrieves document summaries, disambiguates similar workbooks, and executes constrained Structured Query Language (SQL) over the selected full table. On DataBench (80 datasets, 1,810 questions), FiCo reaches 76.2% accuracy: 9.9 points above a strong TableRAG-style baseline on the same frozen workbook choices (66.3%) under the tracks' prespecified evaluators, and 63.4 points above prefix RAG (12.8%). On 508 MiMoTable questions, FiCo reaches 79.7%, versus 22.2% for prefix RAG. Giving the strong baseline the gold workbook raises it from 66.3% to 76.3%, exposing a 10.0-point source-selection cost under fixed compute. Despite 95.1% document recall and 98.5% executable SQL, only 81.3% of questions execute on the gold workbook. FiCo's advantage comes from integrating semantic source selection with exact, schema-grounded computation.

0.27

MidTool: Mid-training Data Synthesis for Agentic Tool Use

cs.AI ·2026-08-20 ·Fengqing Jiang, Yite Wang, Boyi Liu, Zhaoyang Wang, Canwen Xu +3 more
highly cited similarity 0.64
Abstract · Data & Model: https://hf.co/collections/MidTool/midtool-release

Mid-training is increasingly recognized as a critical stage for shaping the capabilities of large language models. Recent work has shown that targeted mid-training can strengthen reasoning-intensive abilities such as math and science, and can also improve agentic capabilities in software-engineering settings. In this work, we study the parallel but less explored agentic capability: general tool use. We present MidTool, an open corpus construction pipeline for agentic tool-use mid-training that combines large-scale web, PDF, and code data with synthesized supervision from real-world tool APIs, MCP skills, and document-grounded workflows. MidTool is designed to teach models how to recognize tool affordances, ground arguments from context, compose tool call workflow, and recover from incomplete information. We mid-train Qwen3-4B-Base and Qwen3-8B-Base on MidTool-Mix, and then apply follow-up post-training with both supervised fine-tuning and reinforcement learning. Compared with baselines, MidTool-Mix consistently improves downstream performance under both SFT and RL on BFCL, tau2-Bench, and MCP Universe. These results suggest that general tool use, like other important LLM capabilities, benefits from dedicated mid-training rather than being left entirely to post-training.

0.43

Carnot: Interpretable, Interactive, and Optimized Execution of Deep Research Queries

cs.DB cs.AI ·2026-08-10 ·Matthew Russo, Yash Agarwal, Tianyu Li, Zhuohan Gu, Michael Cafarella +3 more
top venue similarity 0.64
Abstract · 4 pages, 2 figures, published as a demo paper in VLDB 2026

Enterprises increasingly seek to query data lakes using natural language via AI-driven tools like semantic operators or deep research agents. However, the latter operates as an opaque black box, hiding its intermediate reasoning and data retrieval steps, and failing to expose controls for managing API costs and execution latency. Meanwhile, the former can be prohibitively expensive for enterprise-scale data lakes. Consequently, analysts using these systems lack the agency to intercept hallucinated premises, verify intermediate results, or correct the system's trajectory. We present Carnot, an interactive execution engine for AI-driven analytics. Carnot compiles natural language requests into physical execution graphs and surfaces them through an interactive notebook interface. Rather than waiting blindly for a final output, users can critique the plan, incrementally execute operators, inspect intermediate data, or directly edit the underlying code or semantic operator instructions. Carnot's query optimizer will optimize the query with respect to cost or latency constraints provided by the user. Our demo will showcase how Carnot helps users achieve efficient and verifiable insights on workloads motivated by real enterprise use cases.

0.72

Cellery: Gradual, Composable Structure for Spreadsheets

cs.HC ·2026-09-02 ·Jacob Yim, Sathwika Peechara, Brian Hempel, Michael Coblenz
top venue similarity 0.63
Abstract · To appear at the 39th Annual ACM Symposium on User Interface Software and Technology (UIST '26)

Structure in spreadsheets is typically restricted to ranges of cells and flat tables. However, replacing the open-ended two-dimensional grid with a more structured alternative risks limiting users' freedom and impeding exploratory workflows. How can we enable expressive structure in spreadsheets while preserving their flexibility? We propose a model that allows users to gradually add structure on top of the traditional grid. To identify common structural patterns, we conducted a content analysis of 30 real-world spreadsheets. Based on our findings, we developed Cellery, a novel spreadsheet system that makes these structures explicit. In Cellery, users gradually define the structure of their spreadsheets using named groupings of cells, composing together four structural primitives: Atoms, Lists, Tables, and Blocks. Once defined, users can directly manipulate structures, dynamically reference them in formulas, and incorporate them in reusable abstractions. We evaluate Cellery with an expressiveness study reproducing spreadsheets from our sample, demonstrating that it is capable of representing types of structure that traditional spreadsheets cannot. In a preliminary user study of five participants, we find that they successfully use Cellery to design spreadsheets with explicit structure, and we uncover opportunities for further work on the formula language.

0.28