CS Paper Radar

Can We Use Saliency Maps for the Evaluation of Generative Image Models?

Eleonora Ristori, Luca Bindini, Marco Lippi, Paolo Frasconi

cs.CV ·submitted 2026-10-16 13:05 UTC · v1

Abstract

Evaluating generative image models remains challenging, as commonly used metrics depend on fixed pretrained feature extractors optimized for different tasks that do not always align with human realism perception across datasets and model families. To address this limitation, we investigate a human-inspired evaluation framework that avoids representations learned for unrelated tasks. Specifically, we train a generator-specific classifier to distinguish real images from samples produced by a given model and analyze its decisions using saliency maps. Across diverse benchmarks, we observe that these classifiers learn generator-specific artifacts rather than a universal notion of "fakeness", exhibiting strong accuracy and limited cross-generator transfer. This makes saliency analysis particularly informative: the resulting attribution maps highlight visual cues and structural irregularities characteristic of each model's outputs. By combining classification accuracy and saliency information, we obtain a composite score aligned with human judgments, without relying on large pretrained backbones and using only real and generated samples from the evaluated model.

Closest papers

Nearest neighbours by meaning across the whole index.

[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning

cs.CV ·2026-10-10 ·Anton Nuzhdin, Andrei Gruzitski, Balázs Egyed, Aleksandr Raudvee
published similarity 0.70
Abstract

This paper presents a reproducibility study of ``Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning'', published at ICLR 2025, which proposes an unsupervised adversarial fine-tuning (AFT) method with norm regularisation to enhance the visual interpretability of CLIP's image encoder. We attempt to reproduce the key claims regarding improved saliency map quality, increased concept alignment, transferability to out-of-distribution datasets, and the trade-off with zero-shot accuracy. Beyond reproduction, we propose a saliency-guided regularisation extension that introduces an Energy Pointing Game loss, directly supervising the spatial alignment of Simple Gradient saliency maps with target objects. We evaluate our extension across a range of saliency-loss weights and show that explicit saliency supervision \rev{substantially} improves localisation metrics at a modest cost: the reduction in zero-shot accuracy and adversarial robustness is small at the lowest saliency-loss weight ($w_s=0.2$) and moderate at the strongest setting ($w_s=0.6$). We restructure the codebase for configurable, systematic experimentation.

0.28

Beyond Resemblance: Image Fusion Reimagined

cs.CV ·2026-11-19 ·Hao Zhang, Yumin Huang, Linfeng Tang, Junjun Jiang, Jiayi Ma
similarity 0.69
Abstract

Image fusion often lacks ground-truth fused images, forcing existing deep methods to indirectly define fusion objectives by requiring the fused image to simultaneously resemble multiple source images. However, the distinct imaging characteristics of different modalities can impose competing constraints on the fused result, potentially limiting the effective integration of their complementary information. To address this long-standing challenge facing the image fusion community, we propose ApexFuse, a novel generative image fusion paradigm that moves beyond source-resemblance constraints toward learning a general distribution of high-quality scene representations. Specifically, we develop a human-LMM-in-the-loop pipeline that leverages large multimodal generative models and iterative human feedback to translate abstract fusion objectives into reliable visual references, thereby defining the target distribution of ideal fusion outcomes. Through target-distribution-guided adaptation, we transfer the rich generative priors of pretrained FLUX to image fusion, enabling high-quality scene reconstruction conditioned on multimodal observations. Extensive experiments demonstrate that ApexFuse harmoniously integrates salient thermal targets, natural colors, and fine textures, achieving superior visual quality as well as semantic segmentation and object detection performance over previous source-resemblance methods. Moreover, ApexFuse exhibits remarkable cross-task generalization, extending from infrared-visible fusion to multi-exposure fusion, multi-focus fusion, and visible-depth fusion without additional fine-tuning.

0.20

Amplify What You Gaze At: Target Saliency Boosting in Text-to-Image Generation

cs.CV ·2026-09-25 ·Shengqi Dang, Zhengxi Yu, Feilin Han, Xingyu Lan, Nan Cao
dataset similarity 0.67
Abstract

Text-to-image generation has advanced in controlling what, where, and how objects appear, yet how visual attention is distributed among objects remains largely unexplored. In this paper, we introduce Target Saliency Boosting, a new task aimed at boosting the visual saliency of a specific object during text-to-image generation without requiring any visual priors. Our key insight is that visual saliency is inherently relative: boosting the saliency of a target object also depends on the global saliency distribution across all objects in the scene. Based on this insight, we propose GazeME, a lightweight framework that uses saliency-marked prompts, inserting learnable marker tokens around object descriptions to indicate which objects to visually emphasize or suppress. To learn these markers, we construct a saliency-semantics dataset that associates objects in image--prompt pairs with object-level saliency scores, and propose Saliency Prior Marker Activation (SPMA), a saliency-aware stochastic marker activation strategy that exploits relative saliency relationships for robust training. During inference, GazeME automatically inserts appropriate markers into the prompt, thereby directly enhancing the visual saliency of the target object. Extensive experiments demonstrate that GazeME effectively boosts target saliency while preserving both semantic alignment and image quality.

0.11

Learning Continuous Source Responses For Generalizable AI-Generated Image Detection

cs.CV ·2026-09-13 ·Manni Cui, Ruiqi Liu, Zijian Yu, Hao Tan, Zibo Wei +6 more
code/data similarity 0.67
Abstract

Advances in image generation have made synthetic images increasingly difficult to distinguish from real photographs, raising concerns about the trustworthiness of visual media. Existing AI-generated image detectors often perform well on in-domain data, but their robustness and cross-generator generalization remain limited. These limitations are commonly attributed to overfitting to shortcut cues. Although many methods seek to suppress shortcut learning, most retain binary classification as the training task without reconsidering how the task itself shapes the learned representations. We introduce CuRe, a framework for learning Continuous Source Responses that revisits authenticity detection from the perspective of the training task. CuRe reformulates backbone adaptation as regression of real-generated mixing ratios, providing finer supervision that encourages the model to capture authenticity-related variation beyond binary endpoint separation. We further select a compact source-response subspace to suppress nuisance variation and limit the final classifier's access to potential shortcut cues. Across ten public benchmarks, CuRe achieves an average balanced accuracy of 89.7%, exceeding the second-best method by 5.2 percentage points. Further experiments demonstrate consistent generalization gains across visual backbones and strong robustness to common image degradations. Code is available at https://github.com/manic-cui/CuRe

0.24

ConEx: Human-Interpretable Saliency Maps via Concept-Aware Attribution

cs.CV cs.AIcs.LG ·2026-10-03 ·Yehonatan Elisha, Oren Barkan, Ziv Weiss Haddad, Noam Koenigstein
published similarity 0.67
Abstract · ICML 2026

Many visual explanation methods in computer vision highlight pixel importance but struggle to link these low-level cues to semantically meaningful concepts, limiting their interpretability and trustworthiness. We introduce Concept-based Explanations (ConEx), a novel framework that bridges saliency visualization with concept-based reasoning to provide both faithfulness and interpretability. ConEx automatically discovers class-specific concepts and represents them through concept activation vectors (CAVs), learned without manual supervision using an architecture-specific masking mechanism that reduces noise introduced by the segmentation masks to enhance concept purity. ConEx generates faithful saliency maps that reveal where each concept appears in the image and how it contributes to the prediction. To evaluate the reliability of these learned concepts, we propose two complementary metrics, Vector-Concept Match (VCM) and Concept-Class Match (CCM), that quantify concept alignment and enable direct comparison with existing methods. Extensive experiments across diverse settings demonstrate that ConEx achieves state-of-the-art performance on faithfulness, segmentation, and concept-quality benchmarks. Overall, ConEx advances the field toward truly interpretable and concept-grounded explanations in vision models.

0.45

Relative Patch Response Learning for Generalizable AI-Generated Image Detection

cs.CV ·2026-10-08 ·Tianyu Wang, Ouxiang Li, Yanbin Hao, Zhenhua Tang, Shuo Wang
similarity 0.67
Abstract · 19 pages, 6 figures, 13 tables

Generative models can now synthesize highly realistic images, simultaneously increasing the risks of misinformation and visual forgery. Therefore, detecting AI-generated images becomes more essential, and a reliable detector must generalize to unseen generators and stay robust to unseen perturbations in the wild. Existing detectors are typically trained on either independently collected real and generated images or aligned real-generated pairs designed to mitigate content bias. Building on aligned pairs, recent methods form a mixed view by replacing some patches of the real image with their generated counterparts. However, we find that self-attention lets real and generated patches interact, so the feature of each patch no longer reflects its own source alone. This contextual shift makes a per-patch source label an imprecise target. To this end, we propose Relative Patch Response Learning (PRL). Instead of labeling each patch, PRL compares the same patch across two mixed views of an aligned pair and learns from its patch response, the change of its score between the views. (i) To give precise supervision under the contextual shift, a relative response objective measures the responses of source-changed patches against those of source-unchanged patches, which respond to the shift alone. (ii) To provide a reliable reference for the shift, a reference coherence objective keeps each group of source-unchanged patches moving as a whole. (iii) Since the two views contain different amounts of generated content, an area ranking objective asks the view with the larger generated area to have a higher mean patch score. Extensive experiments demonstrate the superior performance of PRL, which surpasses the best prior methods by 4.3% and 5.9% in average balanced accuracy across eight standard and three in-the-wild benchmarks, respectively.

0.20

TRIG: Toward Measuring Texture Realism in Image Generation

cs.CV ·2026-10-10 ·Zhiwei Xu, Jiali Cui, Zhiqiang Lao, Heather Yu, Liang Peng
similarity 0.66
Abstract · 22 pages, 19 figures

This paper studies the backbone used by metrics for evaluating the quality of generated images. Although generative models have advanced rapidly, some outputs still exhibit unrealistic texture characteristics, and these differences are difficult to measure. Widely used evaluation metrics provide limited insight into texture realism, since they often rely on a backbone trained for another objective, such as image classification or image-text alignment. Such objectives typically encourage invariance to low-level variations that carry texture, so the resulting features primarily encode semantic and global structure. As a result, texture-specific discrepancies may not be explicitly captured. We propose TRIG, a metric directed at the texture realism of generated images. Its backbone is instead the discriminator of a generative adversarial network pre-trained on a texture dataset, which provides a learned prior for the local statistics of natural textures. A plausibility head is fine-tuned to identify local texture degradations, while preserving natural defocus and sensor noise as positive cases. On the features of this backbone, we compute a standard distributional distance between a reference set of real images and a set of generated ones. We apply TRIG to several generative models and observe closer agreement with human realness ratings than baseline metrics.

0.27

ScenePercept: Scenario-Conditioned Perceptual Assessment of Real-World User-Generated Content Images

cs.CV ·2026-10-13 ·Shuhan Xia, Yu Chen, Xiang Wang, Shuai Li, Peipie Li +1 more
benchmark similarity 0.66
Abstract · Revise some experimental results

The rapid development of mobile imaging and social platforms has led to explosive growth in user-generated content images, making their systematic and comprehensive perceptual assessment increasingly important. Existing image perception benchmarks are typically task-oriented, curating specific scenarios for quality or aesthetic assessment and applying a uniform perceptual criterion across diverse scenarios. This limits their ability to capture the scene diversity and perceptual complexity of real-world UGC images. To address these limitations, we introduce ScenePercept-Bench, a benchmark designed to reflect the distribution of real-world UGC images, comprising 23,100 images across 21 categories and 105 fine-grained scenes. To account for the influence of scene context on human perception, we design a hierarchical, scene-aware annotation protocol, covering visual quality, aesthetic perception, and scene semantics, with 15 attributes in total. Each image is annotated with an overall score, 15 attribute-level scores, and structured explanatory reasoning. Building on this benchmark, we further introduce Scenario-Conditioned Perceptual Alignment, a training framework that tackles the heterogeneity of perceptual criteria across scenarios. SCPA jointly optimizes absolute score prediction, preference ranking, and cross-scenario calibration, preserving the model general perceptual assessment ability while encouraging it to explore and adapt perceptual criteria across different scenarios. Extensive experiments demonstrate that SCPA achieves state-of-the-art performance in UGC image perception and generalizes effectively to established IQA and IAA benchmarks.

0.20