CS Paper Radar

AVAtt: A Free-Viewing Eye-Tracking Dataset and Benchmark for Visual Attention on Paintings

Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani, Alessandro Bruno

cs.CV cs.HCeess.IV ·submitted 2026-10-21 22:02 UTC · v1
publishedbenchmarkdatasetcode/data

Abstract

Saliency and scanpath models are developed almost exclusively on natural photographs, yet paintings are composed by artists to guide the viewer's eye and differ from photographs in texture, colour and abstraction. We introduce \avatt, a free-viewing eye-tracking dataset on \nImages{} paintings spanning sixteen art movements and five centuries, with \nScanpaths{} scanpaths and \nFixations{} fixations from \nParticipants{} participants (\spPerImageMin--\spPerImageMax{} observers per painting). For every painting we release individual scanpaths with fixation timing and eye--screen distance, fixation maps, saliency maps smoothed at one degree of visual angle, temporal saliency videos and rich art metadata. We benchmark classic and deep saliency models against human agreement, revealing substantial headroom for art-aware models, and rank 44 scanpath-similarity measures, including a new language-free family of image-encoder metrics built on ten CNN, ViT and self-supervised encoders. With controls that remove viewing biases, we show that faces are fixated \faceEnrich$\times$ more than expected and within half a second, that local structure draws the eye, that composition rules fitted from image content predict where people look, and that art style shapes how much viewers agree. Dataset, code and a Python package are available at: https://huggingface.co/datasets/Amine-CV/AVAtt.

Comment
Extenstion of 10.1145/3649902.3655656 published at ETRA 2024

Closest papers

Nearest neighbours by meaning across the whole index.

Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters

cs.CV ·2026-10-02 ·Susmit Agrawal, Rebecca Wanner, Juliane Verwiebe, Matthias Tangemann, Matthias Bethge +1 more
benchmarkdatasetcode/data similarity 0.67
Abstract

Video saliency prediction is inherently harder to model than static image saliency due to the additional temporal dimension. Video saliency benchmarks rest on the premise that predicting gaze on video requires utilizing temporal activity distributed across frames. Prior work has challenged this, showing that static baselines recover a significant fraction of the explainable gaze information on LEDOV, a popular video saliency dataset, and that video saliency models fail in the same places as this static baseline. We verify that this diagnosis still stands: under a more capable gold standard than the original analysis, and an updated panel of recent architectures, the strongest temporal architecture in the panel still does not substantially improve over a fine-tuned static baseline. However, it remains unclear whether the marginal gain reflects limitations of current temporal architectures or a lack of temporal patterns in the benchmark itself. We introduce SalTempto, a video saliency benchmark with greater dynamism: 224 clips of highly dynamic content, sourced from the HACS-Segments dataset so that each clip contains an event together with its lead-up and aftermath, with gaze recordings from up to 16 subjects and a training split for adapting pretrained models. On SalTempto, the static baseline recovers only about 13\% of the headroom above the centerbias, against more than half on LEDOV. A fine-tuned temporal architecture shows a substantial gain in performance over the static baseline, indicating that it does capture meaningfully more temporal information, which LEDOV fails to measure. Yet, even this SoTA model still leaves nearly half of SalTempto's headroom unexplained, indicating room for improvement in video saliency modelling. Examination of SalTempto also lets us describe human tendencies that models miss. SalTempto link: https://huggingface.co/datasets/bethgelab/video_saliency.

0.32

Assessing Representation Semantics Under Language Pretraining in Fine-Grained Classification

cs.CV cs.AI ·2026-10-20 ·Md Raisul Kibria, Sébastien Lafond
similarity 0.66
Abstract

Understanding how supervision signals shape semantic visual representations remains challenging, particularly in fine-grained, low-data settings. Standard evaluations of vision models rely on downstream accuracy, which often fails to capture structural differences in learned representations. We propose a compact explainability-driven framework for comparing encoders trained with classification-only and caption-generation supervision on CUB-200. Using a controlled two-stage protocol in which both encoders are ultimately optimized for the same classification task, we enable a direct comparison of the resulting representations. Saliency maps are generated using attention rollout and gradient-weighted rollout and evaluated with five metrics covering faithfulness (positive and negative perturbation) and localization (keypoint emphasis, background attribution, and Earth Mover's Distance). Across random seeds and explanation methods, caption supervision yields representations that are more sensitive to the removal of salient regions, better aligned with annotated object parts, and less reliant on background regions, whereas classification supervision produces representations that are slightly more robust to the removal of low-saliency regions. An ablation study on the Oxford-IIIT Pet dataset shows consistent trends across key metrics. These results demonstrate that the proposed framework makes supervision-induced representational differences observable and quantifiable beyond downstream accuracy alone. More broadly, it provides a systematic basis for assessing learned representations and comparing alternative training and transfer-learning strategies.

0.18

Semantic Saliency from Multi-Modal Large Language Model Scene Understanding Maps

cs.CV ·2026-10-11 ·Ziqi Wen, Shravan Murlidaran, Jonathan Skaza, William Wang, Miguel P. Eckstein
code/data similarity 0.64
Abstract

Image-computable low-level image saliency has shaped perceptual psychology and computer science, but is limited by its inability to capture human high-level cognitive factors that influence attention and eye movements. Here, we propose a new fully automated method to compute semantic saliency by quantifying each object's contribution to scene comprehension without fitting any parameters to human data: the Automated Scene Understanding Map (AUTO-SUM). AUTO-SUM automatically segments and removes individual objects from a scene, uses multimodal large language models (MLLMs) to describe the original and modified scenes, and measures the semantic similarity between these descriptions using text embeddings. We show that AUTO-SUM can predict which object is most important in a human-measured scene understanding map (H-SUM) estimated using human scene descriptions. We also show that AUTO-SUM can predict the object most fixated by human observers during free viewing and scene description tasks better than a low-level saliency model (GBVS, Itti-Koch), a saliency model trained based on human meaning ratings (Meaning Map), and comparable to DeepGaze, which is trained on human fixation data. We contend that AUTO-SUM can be used as a semantic saliency model that complements lower-level saliency models. Code are available at https://github.com/starsky77/AUTO-SUM-Source-Code .

0.24

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

cs.CV cs.LG ·2026-09-29 ·Sheng Zhao, Weikai Lin, Yuhao Zhu
top venue similarity 0.64
Abstract · Accepted at NeurIPS 2026. 25 pages

Egocentric gaze prediction enables many downstream applications but remains challenging, as human gaze is inherently stochastic. This stochasticity is constrained by structured temporal dynamics alternating between fixations and saccades, top-down influences from tasks, and bottom-up visual saliency. Based on this observation, we introduce GazeFlow, a framework that directly models gaze as a joint distribution of temporal gaze positions conditioned upon both top-down and bottom-up information. In particular, GazeFlow uses conditional flow matching (CFM): a learned velocity field iteratively transports a Gaussian noise sample into a plausible gaze trajectory drawn from this joint distribution. The velocity field is conditioned on bottom-up visual features extracted by a video encoder and on top-down task information obtained by globally querying these features. On standard datasets, GazeFlow achieves state-of-the-art performance on per-frame metrics, and the generated trajectories align better with human gaze temporal dynamics.

0.47

[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning

cs.CV ·2026-10-10 ·Anton Nuzhdin, Andrei Gruzitski, Balázs Egyed, Aleksandr Raudvee
published similarity 0.64
Abstract

This paper presents a reproducibility study of ``Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning'', published at ICLR 2025, which proposes an unsupervised adversarial fine-tuning (AFT) method with norm regularisation to enhance the visual interpretability of CLIP's image encoder. We attempt to reproduce the key claims regarding improved saliency map quality, increased concept alignment, transferability to out-of-distribution datasets, and the trade-off with zero-shot accuracy. Beyond reproduction, we propose a saliency-guided regularisation extension that introduces an Energy Pointing Game loss, directly supervising the spatial alignment of Simple Gradient saliency maps with target objects. We evaluate our extension across a range of saliency-loss weights and show that explicit saliency supervision \rev{substantially} improves localisation metrics at a modest cost: the reduction in zero-shot accuracy and adversarial robustness is small at the lowest saliency-loss weight ($w_s=0.2$) and moderate at the strongest setting ($w_s=0.6$). We restructure the codebase for configurable, systematic experimentation.

0.28

HiRA-CAM: Preserving Fine-Grained Spatial Relevance in Gradient-Based Visual Explanations

cs.CV cs.AIcs.LGcs.NE ·2026-08-19 ·Manasi Nerurkar, Ali A. Minai
published similarity 0.63
Abstract · IEEE World Congress on Computational Intelligence, Maastricht, Netherlands, June 2026

Deep Learning models can include billions of parameters or more, making it difficult to explain their internal transformations and outputs. However, explainability is increasing in importance due to the use of AI in crucial applications. This paper focuses on the interpretability of convolutional neural networks (CNNs). Building on the popular gradient based method LayerCAM for extracting internal features in CNNs, we propose an improved method named HiRA-CAM, and show that it outperforms both LayerCAM and Grad-CAM on creating useful saliency maps for object classification. The main feature of HiRA-CAM is its adaptive use of activation maps from all the layers of the CNN to arrive at a more focused saliency map.

0.21

TRINITY: A Multi-Perspective Benchmark for Personal-Style Video Highlight Detection

cs.CV ·2026-08-30 ·Qianqian Chen, Hyun Bin Kim, Denzel Elden Wijaya, Yang Yi, Bo Liu +1 more
top venuebenchmarkcode/data similarity 0.62
Abstract · 32 pages, 9 figures. Accepted to ECCV 2026

Traditional video highlight detection relies on a narrow, event-centric definition of saliency, which often fails to generalize to unconstrained personal videos where highlights are heterogeneous and perspective-dependent. To address this, we introduce TRINITY, a multi-perspective benchmark that decomposes highlight saliency into three complementary dimensions, Event, Emotion, and Nature, within a unified temporal framework. Leveraging this multi-faceted view, we propose a shared-backbone multi-branch architecture designed for parallel multi-perspective prediction via view-specific experts. Comprehensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines, achieving gains of +7.15/+3.62 mAP (rho=15%/50%) on Mr. HiSum and +10.82 mAP on YouTube Highlights. These results validate that multi-perspective modeling provides a more robust and comprehensive formulation of video saliency, especially for complex real-world scenarios. The benchmark and relevant codes will be released upon acceptance. The benchmark is available at https://huggingface.co/datasets/vanilladucky/TRINITY and the code is available at https://github.com/vanilladucky/TRINITY.

0.57

Beyond Endpoints: Bézier Paths for Vision Foundation Model Adaptation

cs.CV ·2026-11-13 ·Chenzhi Zheng, Xi Xiao, Chih-Ting Liao, Qizhen Lan, Jingwen Lin +3 more
code/data similarity 0.62
Abstract · 47 pages

Frozen vision foundation models can be adapted by learning a few prompt tokens shared across all images. When prompts are updated as data shift or tasks change, the distribution of image features moves from one endpoint to another. Prior theory asks which endpoints prompts can reach; we ask whether they can travel between them along the shortest path. Because shared prompts compete with each image's own content for attention, they often cannot. We quantify this restriction as the extra transport cost that shared attention imposes over the shortest path. In an exactly solvable attention model, the reachable features form a cone; unrolling it into a plane gives the exact minimum action for every pair of endpoints and explicit optimal prompts on short arcs. The analysis reveals a sharp threshold in content normalization, which measures how much attention each image's own tokens absorb: for same-sign endpoints, at most two distinct levels allow prompts to follow the shortest path, whereas three or more force a strictly positive extra cost unless the endpoints differ only by rescaling. This gap persists with more prompts, small content perturbations, and aligned multihead outputs. On ViT-B/16 with VTAB-1k, a quadratic Bézier prompt path that minimizes feature action reduces held-out transport action relative to linear and cross-entropy-trained paths between the same endpoints and improves intermediate and online accuracy under unlabeled distribution shift, while cross-entropy-trained paths retain higher minimum accuracy when labeled calibration data are available. Reachable endpoints do not guarantee a reachable shortest path: how prompts move matters, not only where they end. Project page: https://xixiaouab.github.io/Beyond-Endpoints/.

0.36