CS Paper Radar

Plottery: A Code-Integrated GUI for Matplotlib

Brian Hempel, Ruanqianqian Huang, Devamardeep Hayatpur, Sorin Lerner, Haijun Xia

cs.HC ·submitted 2026-10-09 20:32 UTC · v1
published

Abstract

Scientists have many tools to author plots: graphical direct manipulation apps like a spreadsheet's chart builder, text-based programming tools like Matplotlib and ggplot2, or AI chat interfaces. Textual code is the most capable of these, but writing code, even with AI assistance, is a potentially poor fit for graphical chart design. We present Plottery, a Jupyter notebook extension that leverages API type annotations to add graphical direct manipulation to the popular Matplotlib plotting library. User interactions in Plottery's in-situ graphical interface are immediately reified in the user's code, so they may seamlessly mix-and-match between GUI, code, and AI interactions. In a user study, participants seamlessly alternated modalities, with the GUI used for straightforward visual changes, code for quick edits or duplicating code, and the AI for broad support. Plottery showcases graphical interaction that retains code's benefits.

Comment
Preprint. Final version is published in the proceedings of the 2026 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

ggaction: A Grammar of Graphical Actions

cs.HC cs.GR ·2026-09-13 ·Hyeon Jeon, Jinwook Seo
code/data similarity 0.69
Abstract · 21 pages

A chart may be declarative; authoring it is not. Visualization grammars often describe charts as finished specifications, whereas people construct them through a sequence of authoring actions. This mismatch can make visualization code difficult for humans to interpret and for machines to generate from human intent. ggaction addresses this gap by modeling the chart authoring process itself. In ggaction, individual authoring actions are abstracted as functions, and the authoring process is expressed as a chain of these functions. This representation more closely aligns chart designers' authoring intent with code specifications, making the code easily understandable to both humans and machines, including language models. Through a series of evaluations, we show that ggaction is sufficiently expressive to capture common chart authoring intents and outperforms widely used visualization grammars, including Vega-Lite and ggplot2, in both human and machine interpretability. ggaction is available at github.com/ggaction/ggaction.

0.24

ChartRevise: A Dataset and Evaluation Protocol for Exact Chart Editing via Code

cs.AI cs.CV ·2026-09-29 ·Jiaxiang Tang, Yi Zhou, Chad DeLuca, Rogerio Feris, Ahmed Khalil Omran +3 more
dataset similarity 0.63
Abstract

Chart editing requires cross-modal edit grounding, realizing a requested visual change in the code that draws it, with necessary related updates and without altering unrelated content. Existing benchmarks emphasize either code executability or chart quality, but their metrics do not clearly distinguish request completion from missed coupled updates and gratuitous changes. We introduce ChartRevise, a structured dataset and evaluation protocol for exact program-grounded chart editing. For dataset construction, we build on the grammar of graphics to systematically cover chart-editing operations, using source-program checks to verify their applicability across chart types and libraries. To improve edit exactness, our pipeline checks individual requirements and guides repair or exclusion when they are unmet. The resulting dataset contains 92,438 records covering 344 edit types across 20 chart types and three plotting libraries. For evaluation, our reference-free protocol separately measures atomic requirement completion, identifies gratuitous changes, and detects missed coupled updates. These checks are combined with successful execution and rendering to determine exact-edit success. Across five models and four external benchmarks, fine-tuning yields relative gains of 16\% in mean requirement recall and 22\% in mean exact-edit rate.

0.08

RAGE-Vis:A Relation-Aware Generative Editing Interface for Natural Language-Based Chart Editing

cs.HC ·2026-08-12 ·Ziyao Kang, Yiping Sun, Linxuan Tian, Henghuan Qu, Wei Zeng +1 more
published similarity 0.63
Abstract · Accepted by ChinaVis'26

Natural language offers an easy way for users to express chart editing intents, which are often composite and cross-component (e.g., adjusting style, extending categories, highlighting values). However, existing methods typically map instructions to a single operation or widget, limiting their ability to handle high-level requests and often producing locally plausible but globally inconsistent results due to a lack of awareness of relationships between chart components. To address these challenges, we introduce RAGE-Vis, a Relation-Aware Generative Editing interface for natural language-based chart editing. The system supports bitmap chart images as input and converts them into an editable parameterized intermediate representation. Instead of mapping instructions to a single edit or widget, RAGE-Vis parses composite intents, identifies targets and scopes, and generates hierarchical editing panels for underspecified requests, enabling users to adjust both global settings and local parameters. Furthermore, RAGE-Vis identifies potentially affected fields based on visual encoding relations, structural relationships, and expressive consistency relations, and organizes them into actionable widgets to support cross-component coordinated controls. Through two case studies, we demonstrate the applicability of RAGE-Vis in complex editing tasks, including style adjustment, data extension, order rearrangement, legend layout, and color mapping. A user study further shows that participants can effectively handle underspecified requests, explore candidate alternatives, and maintain cross-component consistency with RAGE-Vis.

0.19

A Chat Assistant for Software Exploration in a 3D Software Visualization

cs.SE ·2026-10-07 ·Malte Hansen, Karim Issa, Wilhelm Hasselbring
top venue similarity 0.62
Abstract · 11 pages, 5 figures, accepted at VISSOFT 2026

We present a chat assistant for interactive software exploration, embedded in the 3D software visualization tool ExplorViz. The assistant builds upon current Large Language Models (LLMs) and enables users to ask questions about the currently visualized software system and trigger actions that change the visualization through natural language. We integrate the chat assistant in our ExplorViz frontend using the CopilotKit libraries such that probabilistic LLMs are combined with deterministic and tool-based actions similar to implementations using the Model Context Protocol (MCP). The chat assistant is also enabled to restructure the software system by adding, removing, or modifying parts of the software system in the visualization. An empirical experiment with eleven participants evaluated both perceived comprehension support and the tool calls that were triggered by the chat assistant. Participants rated the assistant's generated summaries and explanations as largely correct. They also reported high usability for actions like highlighting entities in the visualization and the creation of new color themes. In contrast, open-ended chat-assisted software restructuring in the visualization showed mixed results. This suggests a need for stronger guardrails for the employed LLM and better incorporation of user feedback. Overall, the assistant was perceived as usable and promising for reducing interaction overhead during exploratory program comprehension tasks. We provide a video presenting the chat assistant's use and a reproduction package of the software system that was used in our evaluation.

0.46

VisEditBench: Can Vision-Language Models Edit Visualization Code from Multimodal Feedback?

cs.CL ·2026-08-11 ·Mizanur Rahman, Arshia Azimlu, Shadikur Rahman, Md Tahmid Rahman Laskar, Amran Bhuiyan +2 more
top venuebenchmarkcode/datahighly cited similarity 0.61
Abstract

Vision-language models (VLMs) have shown strong capabilities in generating visualization code from textual or visual specifications. However, real-world visualization authoring is inherently iterative: users frequently revise existing visualizations to repair flawed charts or adapt them to desired styles. Existing benchmarks primarily evaluate generation from scratch, leaving visualization code editing from multimodal feedback largely unexplored. We introduce VisEditBench, a benchmark of 1,395 human-annotated visualization code-editing tasks grounded in realistic visualization workflows and failure cases. VisEditBench covers two practical settings: feedback-guided repair, where models revise visualization code using buggy or marked charts together with textual feedback, and reference-guided restyling, where models modify code to match a target chart image. Evaluating 20 state-of-the-art VLMs reveals that visualization code editing remains challenging: Claude-4.6-Sonnet achieves the best overall pass rate of 74.46%, while most open-source models remain below 50%. Performance is particularly weak on visually grounded style adaptation, where Claude-4.6-Sonnet achieves only 55.71%. To establish a strong baseline, we further propose VisEditAgent, a render-grounded editing framework that iteratively generates, executes, validates, and refines candidate edits. Built on GPT-4o, VisEditAgent improves overall pass rate from 55.75% to 67.99%, demonstrating the importance of render-grounded feedback for faithful visualization editing. We will release VisEditBench at https://github.com/vis-nlp/VisEditBench.

0.93

The CAPED Framework: Characterizing User Skills in Chart Authoring through the Lens of Tasks and Tools

cs.HC ·2026-10-26 ·Bongshin Lee, Zhicheng Liu, John Thompson, Chenglong Wang
published similarity 0.60
Abstract · Accepted at IEEE VIS 2026, to appear in IEEE Transactions on Visualization and Computer Graphics (TVCG)

In this paper, we introduce a new framework for characterizing the user skills required for chart authoring. Despite growing attention to authoring systems, we lack a conceptual framework that characterizes the diverse and multifaceted skills involved and helps identify opportunities for system design and development. Our initial investigation revealed that skill is a complex and elusive concept, and that existing theories on skills from other disciplines are too high-level to capture the importance of tasks and tools in data visualizations. To address this gap, we formulate the CAPED framework that characterizes skill as the ability to create a chart involving five dimensions pertinent to tasks and tools: input data, output chart, paradigm, abstraction, and expression. We demonstrate the framework's descriptive power by highlighting different skills involved in similar authoring situations and identifying the types of knowledge associated with chart authoring. We further present examples illustrating how the framework can be used to identify new research opportunities in supporting users with varying skills.

0.37

IChart2Code: Benchmarking Multimodal Large Language Models for Interactive Chart Code Generation

cs.SE ·2026-09-26 ·Xu Zhang, Hongzhang Zheng, Zhili Huang, Yaoyi Wang, Ling Xu +1 more
benchmark similarity 0.60
Abstract · 9 pages, 3 figures, 5 tables

Interactive chart code generation requires models to reproduce a reference chart's appearance and underlying data and correctly implement the state changes triggered by specified user interactions. Existing chart-to-code benchmarks focus on static outputs and lack task representations or evaluation protocols for interaction specification, browser execution, and post-interaction verification. We introduce IChart2Code, a benchmark comprising 377 tasks across 20 chart forms and 13 data families, with 1209 interaction requirements in six families. Each task provides a reference screenshot, task-local data, and natural-language interaction requirements, with executable HTML/JavaScript code as the target output. We further develop a browser-based evaluation protocol with an Executability gate and three rubric-guided dimensions: Data Fidelity, Static Visual Correctness, and Interaction Correctness. The protocol tests runtime viability, consistency with task-local data, fidelity of the initial rendering to the reference screenshot, and interaction-induced state changes in a sandboxed browser. A rubric-guided MLLM judge evaluates task-specific items for the three scored dimensions using the collected browser observations and achieves an overall item-level F1 score of 0.8844 against adjudicated human labels. We also propose TRAIL, a trajectory-guided dual-agent framework for interactive chart code generation. An Inspector derives task-specific inspection checks, executes them in the browser, and uses the resulting trajectories to diagnose failures and produce structured repair feedback. Averaged across four MLLMs, TRAIL improves the four evaluation dimensions over direct prompting by 7.89, 4.98, 3.45, and 4.47 percentage points, respectively.

0.09

From Visual Widgets to UI Code: Efficient Tool-Grounded Generation

cs.CV cs.LG ·2026-08-12 ·Houston H. Zhang, Tao Zhang, Li Gu, Linfeng Ye, Yuanhao Yu +3 more
published similarity 0.59
Abstract · ECCV2026 (MUCG Workshop)

Existing screenshot-to-code systems face a trade-off between flexibility and controllability. Direct multimodal generation can hallucinate visible details, whereas structured pipelines reduce such errors through component-wise decomposition, predefined templates, and customized intermediate representations. These structures, however, introduce additional generative orchestration and restrict outputs to designs covered by the representation. We investigate whether selective tool grounding can improve the fidelity--efficiency trade-off of direct widget-to-code generation. We introduce \textbf{WidgetGen}, a lightweight tool-grounded framework that extracts observable text and color evidence, performs high-level layout and optional chart reasoning, and directly generates executable JavaScript XML (\emph{JSX}). This design reduces reliance on component-wise generation while avoiding a fixed UI schema. Across six multimodal models and \(1{,}000\) held-out widgets, WidgetGen outperforms direct prompting and the structured Widget2Code pipeline on most visual reconstruction metrics, with consistent gains in area, legibility, and style. Finally, reconstruction-derived image-code pairs improve six Qwen-family open-weight models across every reported metric through supervised fine-tuning. These results establish WidgetGen as a strong lightweight baseline and show that selective evidence grounding offers an effective alternative to extensive representation constraints.

0.49