CS Paper Radar

How Do Artificial Agents Make Decisions in Light of Visual Data?

Keyty Loza, Shayan Monadjemi

cs.AI cs.HC ·submitted 2026-11-03 21:58 UTC · v1

Abstract

Vision-language models (VLMs) have demonstrated significant advances in reasoning with visual data, which is particularly important given that much of human knowledge is encoded visually in scientific articles and other sources. However, a gap remains in understanding how artificial agents, powered by VLMs, perceive and navigate decision-making under uncertainty in the presence of visual data. This study explores this gap by evaluating and analyzing autonomous decision-making processes. Building upon prior work in visualization literacy and decision theory, we conducted two complementary studies: one examining how agents perceive and quantify uncertainty encoded in visualizations, and another investigating how artificial agents make decisions based on visual data in the form of images. We analyzed perception accuracy and decision performance across models and visualization types. Our results show that GPT-4o demonstrates the highest accuracy in perceiving uncertainty in pie and bar charts among the tested models. For decision-making, autonomous agents showed the best alignment with rational decision theory when uncertainty was represented through circle charts, icon arrays, or text.

Comment
Presented at the Workshop on GenAI, Agents, and the Future of VIS, held in conjunction with IEEE VIS, 2026

Closest papers

Nearest neighbours by meaning across the whole index.

A Stevens's Power Law Check-up of GPT-5.5's Implicit Reading of Visual Encoding

cs.CV ·2026-10-06 ·Kaichun Yang, Jian Chen
similarity 0.66
Abstract · 9 pages, 6 figures, including supplementary material. Accepted by the VISxGenAI workshop at IEEE VIS 2026

We adapt Stevens's power law to measure the implicit ability of AI models to read visualizations, which can reveal the built-in perceptual mechanisms of algorithmic models. In our pilot study, AIs see no legend. In the color conditions, no colormap name is provided either. GPT-5.5 first views a reference visual representation and estimates its magnitude, then estimates the magnitude of each subsequent image of the same representation relative to that reference. Our evaluation of twelve visual variables makes how algorithmic models read visual encodings measurable, comparable with human perception, and more transparent to humans.

0.31

AI Agents and the Future of VIS

cs.HC ·2026-08-14 ·Chen Zhu-Tian, Nam Wook Kim, Saeed Boorboor, Shivam Raval, Pan Hao +2 more
similarity 0.62
Abstract · workshop proposal

Recent advances in agents (i.e., autonomous, goal-driven AI systems that iteratively observe, act, and learn from their environments) offer a fundamentally different approach from traditional AI models that passively respond to input. These AI agents are rapidly reshaping how we approach data-intensive tasks and providing new opportunities for the VIS community. Imagine an agent autonomously generating visualizations to analyze complex data, discovering patterns collaboratively, testing hypotheses, and communicating visual insights at a speed and scale beyond human capability. Yet, the emergence of these powerful systems raises critical questions that the VIS community must address: Could autonomous agents eventually replace human data scientists, and if not, how might they best collaborate? Are current visualization techniques and interfaces, originally designed for human analysts, suitable for agent interactions? How can VIS designers effectively integrate agents into their workflows without compromising human agency? And to what extent should agents help shape and educate the next generation of visualization researchers? Through a mix of keynote talks, paper presentations, and an agentic VIS challenge, this workshop invites researchers and practitioners to share innovative ideas, explore these questions, and discuss strategies to transform the impact of VIS for a future where human and AI agents co-exist.

0.41

ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

cs.CV ·2026-08-13 ·Mahsa Khoshnoodi, Sarah Adel Bargal
benchmark similarity 0.62
Abstract

Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (\emph{perception}), locating them and binding them to their labels (\emph{grounding}), and performing single-step computations such as ranking, totals, and differences (\emph{simple reasoning}). We introduce \textbf{ChartProbe}, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.

0.04

Evaluating Human-LLM Alignment on Visualization Preference

cs.HC ·2026-11-04 ·Tanmay Bakshi, Chenglong Wang, Simra Shahid
similarity 0.62
Abstract · IEEE VIS 2026 (VISxGenAI Workshop)

There are often multiple valid visualizations, spanning different chart types, visual encodings, and aesthetic choices, that can all support a user's goal of analyzing a dataset. Determining which of these candidate visualizations is most appropriate, however, remains challenging. Prior work has leveraged expert-curated taxonomies to guide visualization design and subsequent quality evaluation, a practical evaluation model must also account for human-centric factors. Latest visualization tools integrate Large Language Models (LLMs) to generate and refine visualizations, using LLM-as-a-judge to provide design feedback. Without a deeper understanding of whether, and in what ways, human and LLM perceptions of visualizations align, there is a real risk of deploying low-quality agents that do not reflect human judgment. We study this question through a pilot experiment - we formulate the problem as a preference task over curated chart pairs and compare the judgments of state-of-the-art vision models with those of a panel of human annotators. We establish a cautionary tale: such judges risk filtering out informative, compelling displays. We find that frontier models collapse human subjectivity, a rich signal that reveals contrasting design choices, into near-deterministic verdicts.

0.19

Colour Blinded by the Noise

cs.HC stat.ME ·2026-08-18 ·Harriet Mason, Rachel Rogers, Alison Kleffner, Dianne Cook
published similarity 0.61
Abstract · To be presented at IEEE VIS 2026 and published in IEEE TVCG in 2027

Uncertainty visualisation is important for data transparency, especially for map visualisations where data is often aggregated. Despite the importance of this area, studies evaluating uncertainty visualisation lack consensus and produce conflicting results. This work introduces a new evaluation approach for uncertainty visualisation that attempts to assess uncertainty as noise, rather than signal. We evaluate five methods of visualising uncertainty: standard choropleth maps, value/variance bivariate maps, value-suppressing uncertainty palettes, overlaid sampling, and pixelated sampling maps. Built on principles of implicit testing, we put an 'uncertainty visualisation' spin on the classic Ishihara colourblind test to create a novel test that is able to evaluate uncertainty as noise. We compare signal visibility to conventional hypothesis tests at various levels of group separation. By building our experimental design on top of established graphics theory, we isolate the plot components that facilitate successful signal suppression and establish foundational theory for the perception of uncertainty visualisation.

0.53

Healthy skepticism in AI: a data visualization research agenda

cs.HC cs.AI ·2026-10-07 ·G. Elisabeta Marai, Marc Baaden, Michael Behrisch, Michael Krone, Pere-Pau Vázquez
position similarity 0.61
Abstract

Research in data visualization of artificial intelligence (AI) models has historically focused on enhancing trust through visual explanations of AI. The trustworthiness line of work was built at least partially on an assumption that humans were critical users unlikely to adopt AI technology. It is increasingly clear that human trust levels in AI span, in fact, a wide range from critical to over-reliant. There is an urgent need to support both trust and healthy skepticism in AI solutions. We argue that it is healthy for humans to adopt a skeptical view both on the results of AI models and on the use of such AI models. We share our thoughts on the rising phenomenon of over-reliance on AI models, the risks and opportunities in using AI models, and the role of data visualization in over-reliance situations where humans are not motivated to engage in critical thinking.

0.24

Visualizing Uncertainty-to-Action Composition for Human Oversight

cs.HC cs.AI ·2026-08-17 ·Chisom Anyabolu, Akshat Dubey, Georges Hattab
similarity 0.60
Abstract · 5 pages, 2 figures, IEEEVis 2026 UncertaintyVis workshop

Artificial intelligence systems often disclose uncertainty, yet they rarely make clear what response that uncertainty should trigger. Most uncertainty visualizations encode uncertainty in model outputs, leaving users to discern the most appropriate course of action. A second region of the design space--uncertainty in the decision process itself, including how multiple uncertainty conditions compose into an oversight response-- remains comparatively underexplored. We address this gap with two coupled contributions. First, we introduce an uncertainty-to-action binding framework that composes multiple uncertainty conditions into a single oversight response under a precedence policy with a contextual safety modifier. That response concerns whether and how an AI-supported decision may proceed, not the substantive domain decision itself. Second, we present ActionCue, a process-transparency visualization that renders that composition explicit. We demonstrate the approach through a three-way comparison with confidence-only and data-level uncertainty displays, using worked cases from healthcare, credit assessment, and disaster forecasting. Together, the framework specifies how uncertainty conditions are resolved into an oversight response, and the visualization makes that resolution inspectable rather than implicit.

0.32

Bridging Learned Visual Perception and Symbolic Belief-Space Planning

cs.AI cs.RO ·2026-09-15 ·Guy Azran, Michael Navat, Sarah Keren
publishedhot direction similarity 0.60
Abstract · To appear in the Proceedings of the 3rd International Conference on Neuro-Symbolic Systems (NeuS), 2026

In partially observable settings, agents must act without full knowledge of the world state and rely on uncertain state-estimation pipelines. Obtaining grounded and verifiable symbolic plans under such uncertainty remains a key challenge. Recent work has integrated Vision-Language Models (VLMs) to bridge perception and symbolic reasoning, following two main paradigms. The first, VLM-as-planner, maps images directly to action sequences, and the second, VLM-as-grounder, grounds observations into symbolic predicates used as the initial state by off-the-shelf planners. Both approaches ignore uncertainty in the planning process, compromising robustness. We introduce a third paradigm, VLM-as-probabilistic-grounder, a novel approach that captures the uncertainty of VLM predicate groundings as a probability distribution over symbolic states. This enables planning in belief space and producing robust plans under uncertainty. Experiments in simulated household robot settings show improved robustness and task success over deterministic grounding, underscoring how our approach leverages foundation models for reliable planning under uncertainty.

0.52