CS Paper Radar

The Whole Warps the Parts: Attentional Certainty Modulates Ensemble-Induced Bias in Visualizations

Jennifer E. Corbett, Brian W. Anthony

cs.HC ·submitted 2026-11-17 14:55 UTC · v1

Abstract

Data visualizations often require viewers to extract both ensemble statistics and individual values from collections of graphical marks. Yet individual items embedded within an ensemble can be systematically biased by that context. Prior studies have reported both repulsion away from and attraction toward ensemble statistics, but these effects have been observed across different conditions. We test whether task or attentional certainty explains these opposing forms of ensemble-induced bias. In Study 1, observers briefly viewed ensembles of differently sized circles and then judged the size of an individual circle embedded within the ensemble. Using a common stimulus generation framework, we tested both identification and discrimination when the target was specified before or after the ensemble, and an additional no-cue identification condition without a specified target. Under the shared cued conditions, effects were predominantly repulsive or near zero in both identification and discrimination. In contrast, eliminating target-specific information reversed ensemble-induced bias from repulsion to attraction. Identification precision also decreased systematically from pre-cue to post-cue to no-cue conditions. Study 2 replicated this cue-dependent pattern in graphical displays of startup valuations and precinct vote contributions. Our findings isolate attentional certainty as a key determinant of whether ensemble context repels or attracts individual-item representations, extending the design question beyond precision to how attention directing choices determine which bias the viewer will be operating under.

Closest papers

Nearest neighbours by meaning across the whole index.

Divided Attention Amplifies the Importance of Expectation-Aligned Visualization Design

cs.HC ·2026-08-10 ·Jiho Kim, Anna L. Chinni, Karen B. Schloss, Michael Gleicher
published similarity 0.60
Abstract · 33 pages, 22 figures (11 pages, 5 figures for the main text; 22 pages, 17 figures for the supplementary material), to be published in IEEE T…

Studies have shown that visualization design affects interpretability when visualization interpretation is the user's sole task. However, in real-world settings, users often engage with visualizations while performing concurrent tasks, such as when users simultaneously monitor alerts or respond to messages. Such divided attention may alter how users interpret visualizations, potentially increasing the importance of designs that align with viewer expectations. We investigated this possibility through two experiments comparing visualization interpretation under single-task and dual-task conditions. Specifically, we examined how well-established inferred mappings between color, spatial position, and semantic concepts affect interpretation when users perform a concurrent task, both with unlimited viewing time (Exp. 1) and under limited viewing time (Exp. 2). Our results show that divided attention amplifies the performance gap between expectation-aligned and expectation-violating designs, affecting response time, interpretation accuracy, and the ability to produce a judgment under time constraints. To explain these results, we model the user's decision-making process using a Linear Ballistic Accumulator (LBA) framework. Our findings highlight the increased importance of aligning visualization designs with viewer expectations under divided attention and introduce a process-oriented modeling approach to understanding how expectation and multitasking shape visualization interpretation.

0.22

Which Attention Heads are like the Human Head? Not the Ones that Compute

cs.AI q-bio.NC ·2026-09-29 ·Christopher Pinier, Gustaw Opiełka, Hannes Rosenbusch, Taylor Webb, Michael D. Nunez +1 more
similarity 0.59
Abstract · 25 pages, 16 figures, including appendix

Brain-AI alignment is often interpreted as a sign that model and brain perform similar computations. Whether the aligned units are causally involved in model computation is rarely checked. On an abstract pattern-completion task (AAABAAA $\rightarrow$ B), we compare LLM attention-head representations with human EEG and test how ablating those heads affects task performance. Alignment and causation dissociate: brain-aligned heads contribute to performance, but their removal is substantially less disruptive than removal of heads selected via attribution patching. We compare two head sets that prior interpretability work defines without reference to the brain: concept vectors (CVs), which represent abstract patterns across formats, and function vectors (FVs), selected for their contribution to correct-answer prediction. Brain alignment shows little association with FV scores, while its association with CV scores varies across models. Among brain-aligned heads, we find recurring attention profiles: one emphasizes distinctive elements (novelty heads), the other repeating elements (repetition heads). The novelty family tracks salience and attends to the same elements that humans look at, yet its removal is less damaging than random ablation on average. Repetition heads contribute modestly to performance and are associated with abstract-pattern representation (CVs). Across 17 models spanning 3B-72B parameters, FV-ranked removal is substantially more disruptive than brain-ranked removal. Brain alignment thus captures how the model reads the stimulus, and only faintly captures how it represents the pattern and solves the task.

0.35

Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models

cs.CL cs.AIcs.CV ·2026-09-29 ·Sumin Hong, Katsumi Ibaraki, Renee Shi, David Chiang, Toby Jia-Jun Li
similarity 0.58
Abstract · 9 pages

Cross-modal associations are systematic pairings of features across modalities, such as the association of 'bouba' with round shapes and 'kiki' with sharp shapes. Prior work has compared humans and vision-language models (VLMs) on such associations, but often using different stimuli or tasks between humans and models. Here, we ask whether VLMs align with humans not only in choices, but also in where they look when making those choices. We study both VLMs and humans (N = 53), presenting them with the same stimuli, a pseudo-word and two images, and record participants' choices and eye movements, which we release. We find choice alignment in a few larger VLMs, but their saliency matches human gaze less closely than a center-bias baseline, a fixed Gaussian at the center of each image. Fine-tuning small VLMs on human choices brings their choice alignment to the level of a human majority-vote reference on unseen words and images, yet their attention still matches human gaze less closely than this baseline. Training model attention on human gaze raises attention-gaze correlation without improving choice alignment, and a single average gaze map per image position raises it by a similar amount. Matching human choices, or even human gaze patterns, is therefore not sufficient evidence of human-aligned cross-modal processing.

0.22

Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

cs.CV cs.AI ·2026-09-01 ·Xurui Song, Weishi Wang, Zhongqi Yue, Kuluhan Binici, Tao Bai +3 more
top venue similarity 0.57
Abstract · EMNLP 2026

Whether attention weights faithfully reflect model reasoning has been actively debated in NLP, yet this question remains largely unexplored for the visual modality in Vision-Language Models (VLMs). We address this gap through causal perturbation analysis on current VLMs, evaluating both the comprehensiveness and sufficiency gap of attention-ranked visual tokens. Our analysis reveals that visual attention faithfulness is heterogeneous, manifesting in three distinct processing modes: Faithful-Sufficient, where top-$k$ attention tokens are both necessary and sufficient for prediction; Faithful-Distributed, where they are necessary but broader visual context remains required; and Non-Focal, where no localized attention region is individually necessary while visual information remains an essential trigger for prediction. Furthermore, human-annotated ground-truth regions satisfy comprehensiveness in only $\sim 60$% of cases compared with model attention rankings, revealing systematic divergence between model visual reliance and human intuition. We demonstrate these patterns across both general VQA on VQAv2 and document tasks on VRDU and ChartQA, showing that visual attention faithfulness varies systematically with processing demands and model architectures rather than being uniformly faithful or unfaithful.

0.61

Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

cs.CL cs.AIcs.CV ·2026-10-14 ·Josh Barua, Kyle Mahowald, William Rudman
similarity 0.57
Abstract

Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.

0.12

One Affinity, Two Axes: Bidirectional Slot Attention for Iterative Aggregation across the Structure--Detail Spectrum in Visual Retrieval

cs.CV ·2026-11-18 ·Zihang Wang, Wenkai Zhu, Lingdong Kong, Teng Yan, Alan Liang +4 more
hot direction similarity 0.56
Abstract

Aggregating dense patch tokens into compact and discriminative global descriptors is central to visual retrieval. Although foundation models provide rich local features, existing aggregation heads still rely on single-pass mappings and lack an explicit mechanism to coordinate structural and fine-grained cues. Therefore, we propose Bidirectional Slot Attention (BiSA), which reformulates aggregation as an iterative coupled system. Cross-Axis Consensus (CAC) progressively refines learnable slots by interpreting a single shared token--slot affinity along two complementary axes. Along the competition axis, slots vie for each token under a temperature shared across all of them, determining token ownership and inducing a non-redundant scene decomposition. Along the selection axis, a slot-specific temperature governs the breadth of each slot's response, spanning a spectrum from broad layouts to sparse fine-grained details. Their consensus couples non-redundant decomposition with multi-granular slot specialization. Furthermore, the two axes expose intrinsic token-reliability cues at no extra cost: a token's claim strength under competition and its salience under selection. Progressive Token Filtering (PTF) converts these cues into a retention probability that down-weights unreliable tokens before the next CAC iteration, without any segmentation branch or external prior, forming a self-reinforcing loop of competitive decomposition, selective focusing, and feedback purification. Across all three retrieval tasks, BiSA leads on five of six place recognition benchmarks (+4.2 R@1 on Baidu Mall), tops all 16 metrics of drone--satellite geo-localization (+8.4 R@1 at 150 m), and improves instance-level retrieval on ROxford-Hard by 4.5 mAP points. Notably, the selection axis adds only 20 learnable scalars, yet improves average R@1 by 2.8 points.

0.34

When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models

cs.CL ·2026-09-28 ·Yizhou Fang, Siyue Chen, Zimo Qi, Zhiyu Xue, Xi Chen +1 more
similarity 0.56
Abstract · 22 pages, 9 figures. Preprint

Despite substantial progress across downstream applications, vision-language models (VLMs) remain susceptible to language bias, often prioritizing linguistic cues over visual evidence and consequently producing incorrect predictions. Prior studies have proposed various approaches to understanding and mitigating language bias in VLMs, yet their findings often conflict due to the difficulty of tracing how language bias propagates within black-box VLMs. Building on the word completion task, we trace how language bias propagates through VLM inference by (1) proposing a diagnostic framework that decomposes the inference process into four distinct yet interdependent stages to trace the propagation of language bias; and (2) examining how two key factors underlying language bias, i.e., linguistic priors and cross-modal coverage, evolve across these stages and ultimately give rise to incorrect predictions. The linguistic prior captures the strength of statistical bias induced by the language model component of a VLM and represents the origin of language bias, whereas cross-modal coverage measures the extent to which linguistic cues cover the visual content. By decomposing inference into four stages and characterizing the interplay between linguistic priors and cross-modal coverage across these stages, we propose a systematic framework for tracing the propagation of language bias throughout the inference process; and uncover the underlying mechanism of language bias by revealing the interplay between linguistic priors and cross-modal coverage.

0.21

Attention Capture Is Not Detection: A Two-Stage Account of How Humans Miss Localized AI Image Edits

cs.CV ·2026-08-14 ·Chiao-Chieh Deng
similarity 0.55
Abstract · 9 pages, 4 figures

As AI-generated image edits proliferate, the platforms meant to curb the resulting disinformation treat detectability as a single, undifferentiated property: an edit either gets a warning or it does not. We show this is the wrong model. Across a controlled eye-tracking study ($N=59$, Latin-square design, four conditions crossing edit area and semantic plausibility), a mixed-effects analysis reveals that whether an edit is noticed and whether it is correctly judged as fake are dissociable stages, governed by different factors: edit area drives attention capture ($p<0.001$) while semantic plausibility drives judgment accuracy and look-but-fail-to-see (LBFS) error rates ($p<0.001$). This dissociation survives correction for multiple comparisons; a secondary interaction between the two factors does not. This two-stage account extends a long-standing distinction in visual attention research (between pre-attentive capture and effortful recognition) into the new domain of AI-edit detectability. We then test whether a generative eye-movement model can computationally operationalize the attention-capture stage: a Transformer trained to generate scanpaths tracks per-image attention with strong discriminative power (Pearson $r=0.77$--$0.82$ across held-out stimuli) and, on the harder task of predicting LBFS incidence, modestly outperforms a two-parameter linear baseline even without access to the plausibility label ($r=0.52$ vs. $r=0.48$). We report this comparison, our ablations, and our method's limitations (a single fixed train/validation split, not leave-one-subject-out) without inflation, consistent with responsibly communicating what a machine learning system can and cannot do to help curb AI-driven disinformation.

0.10