CS Paper Radar

Shaping Useful Noise: Energy Distributions Predict Visual Pretraining Quality

Ching Lam Choi, Antonio Torralba, Phillip Isola, Stefanie Jegelka

cs.LG cs.AIcs.CV ·submitted 2026-11-09 06:31 UTC · v1
top venue

Abstract

Synthetic data can scale visual pretraining, but current selection criteria only partially explain which datasets train strong representations. We propose using the energy profile of a dataset--the distribution of log-probability levels it occupies under a normalised real-image density model--as an audit and curation principle for synthetic data. To operationalise this view, we introduce Energy-Level Distribution Distance (ELD), which compares real and synthetic energy distributions. Real images span sparse high-likelihood scenes and dense low-likelihood textures, so useful synthetic data should cover this energy range rather than collapse to likely but low-variation samples. Across diverse synthetic datasets, ELD exposes mismatches missed by FID and predicts pretraining performance beyond FID, LPIPS variation, and feature-space recall, including across CNN and ViT self-supervised settings. We then use ELD to resample and mix synthetic sources, matching the real-image energy histogram while preserving diversity and improving downstream transfer.

Comment
Accepted as an oral presentation to NeurIPS 2026 (Sydney)

Closest papers

Nearest neighbours by meaning across the whole index.

Scaling Transformer Energy-based Models For High Quality Image Synthesis

cs.LG cs.CV ·2026-10-22 ·Jungsoo Lee, Alexi Gladstone, Yilun Du
similarity 0.70
Abstract

Energy-Based Models (EBMs) are a flexible class of generative models that decouple probabilistic modeling from the generation process. This decoupling directly enables compositional generation and inference-time scaling, where sample quality can be refined simply by increasing sampling steps. However, severe challenges in high-dimensional sampling and training instabilities with deep architectures have restricted EBMs to low-dimensional regimes. To resolve these bottlenecks, we introduce the Transformer Energy-Based Model (TEM), a scalable transformer energy function operating entirely within a latent space. On top of this backbone, we extract energies at multiple layers and sum them, composing complementary constraints from local texture to global semantics. Furthermore, two components make this high-quality at scale: Positive Representation Alignment (PRA), which supervises TEM's intermediate representations with a vision foundation model, and advanced sampling strategies that improve sample quality at inference. TEM-XL achieves an FID-50K of 3.82 on class-conditional ImageNet 256$\times$256 without classifier-free guidance, showing that a contrastive EBM can generate high-fidelity images. On ImageNet 512$\times$512, TEM achieves an FID-50K of 8.24, demonstrating that the approach scales to higher resolution. We further validate the core promises of EBMs: compositional generation, zero-shot inpainting, and out-of-distribution detection.

0.14

Beyond Resemblance: Image Fusion Reimagined

cs.CV ·2026-11-19 ·Hao Zhang, Yumin Huang, Linfeng Tang, Junjun Jiang, Jiayi Ma
similarity 0.68
Abstract

Image fusion often lacks ground-truth fused images, forcing existing deep methods to indirectly define fusion objectives by requiring the fused image to simultaneously resemble multiple source images. However, the distinct imaging characteristics of different modalities can impose competing constraints on the fused result, potentially limiting the effective integration of their complementary information. To address this long-standing challenge facing the image fusion community, we propose ApexFuse, a novel generative image fusion paradigm that moves beyond source-resemblance constraints toward learning a general distribution of high-quality scene representations. Specifically, we develop a human-LMM-in-the-loop pipeline that leverages large multimodal generative models and iterative human feedback to translate abstract fusion objectives into reliable visual references, thereby defining the target distribution of ideal fusion outcomes. Through target-distribution-guided adaptation, we transfer the rich generative priors of pretrained FLUX to image fusion, enabling high-quality scene reconstruction conditioned on multimodal observations. Extensive experiments demonstrate that ApexFuse harmoniously integrates salient thermal targets, natural colors, and fine textures, achieving superior visual quality as well as semantic segmentation and object detection performance over previous source-resemblance methods. Moreover, ApexFuse exhibits remarkable cross-task generalization, extending from infrared-visible fusion to multi-exposure fusion, multi-focus fusion, and visible-depth fusion without additional fine-tuning.

0.19

Learning-State-Aware Dynamic Generative Data Augmentation on Small-Scale Datasets

cs.CV cs.AI ·2026-08-19 ·Ting Xiang, Chenxi Deng, Jinhui Zhao, Bingting Jiang, Ke Zhang +2 more
similarity 0.68
Abstract

Small-scale image classification is often limited by the scarcity of training data. Generative data augmentation (GDA) based on pretrained generative models has emerged as an effective solution. However, existing methods rely on task-agnostic augmentation strategies that overlook downstream model needs. Although recent dynamic GDA methods incorporate model feedback to guide augmentation, they still struggle to reliably determine sample-specific augmentation strengths and adapt augmentation strategies to different image regions while balancing image diversity and class semantics. To address these issues, we propose learning-state-aware dynamic generative data augmentation (LSADA). Specifically, LSADA constructs a learning state for each sample based on its current loss and loss-decrease rate, which is then mapped to a sample-specific augmentation strength. Furthermore, LSADA introduces a decoupled data augmentation and diffusion fusion strategy that applies strength-controlled transformations to class-relevant regions and generates diverse class-irrelevant regions, progressively fusing them to improve image diversity while preserving class semantics. Experiments on nine public datasets show that LSADA outperforms the existing SOTA dynamic GDA method by an average of 4.5% on six natural image datasets and 2.5% on three medical image datasets.

0.15

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

cs.CV cs.AI ·2026-09-26 ·Aleksandr Gushchin, Khaled Abud, Georgii Bychkov, Ekaterina Shumitskaya, Artem Filippov +3 more
benchmarkdataset similarity 0.67
Abstract

AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or processed images. Existing datasets provide limited support for evaluating these challenges jointly across diverse visual content. We introduce REALIS, a dataset of 1.43 million real and synthetic images generated by 42 modern text-to-image models, including the latest proprietary systems such as Nano Banana 2. REALIS combines prompts derived from real images, quality filtering, and stratified sampling to reduce class-specific shortcuts while preserving content diversity. We further introduce REALIS-Expert, a stress-test subset for high-quality synthetic images, where real and generated samples are selected with closely matched semantic and visual characteristics. We also propose a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing. Based on REALIS, our benchmark evaluates pretrained detectors, fine-tuned models, and zero-shot vision-language models under generator and post-processing shifts. On the hardest processed split, the best pretrained conventional detector achieves 0.550 ROC-AUC, compared with 0.752 for the best REALIS-trained detector. REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use.

0.10

Spectra: Multi-Scale Implicit Decomposition for Deepfake Detection

cs.CV cs.AI ·2026-10-22 ·Dhananjaya Jayasundara, Kartik Narayan, Vishal M. Patel
position similarity 0.67
Abstract

Deepfakes increasingly undermine trust in digital media, yet existing detectors often struggle to generalize across unseen manipulation techniques, compression levels, and dataset shifts. Most approaches rely on RGB appearance, frequency-domain artifacts, or their combination, learning cues present in the final synthesized image. While effective in-distribution, such cues are frequently generator-specific and can diminish as synthesis models evolve. We argue that a complementary perspective can be obtained through implicit neural representations (INRs), which expose how an image is reconstructed rather than merely how it appears. By fitting an INR to an image and analyzing its internal activations, one can reveal reconstruction dynamics that are difficult to observe in the image domain alone and may capture more intrinsic differences between genuine and synthetic content. To this end, we propose Spectra, a deepfake detection framework that combines semantic visual representations with artifact-aware cues derived from INR reconstruction. Spectra extracts multi-level reconstruction heatmaps from image-specific INRs and jointly models them with semantic features through a bidirectional fusion mechanism. This enables direct interaction between image semantics and reconstruction artifacts, allowing the model to identify forgery cues that are less dependent on specific generation pipelines. Extensive experiments on multiple deepfake benchmarks demonstrate improved robustness and cross-dataset generalization, consistently outperforming detectors based solely on RGB appearance or semantic representations. Furthermore, we show that the proposed representation extends beyond deepfake detection to face presentation attack detection, highlighting its potential as a unified framework for facial attack detection.

0.37

Source-Anchored Visual Autoregression for Extreme Image Compression

cs.CV ·2026-11-08 ·Bingxin Wang, Yuan Lan, Yang Xiang, Yiqing Huang
similarity 0.67
Abstract

In the extreme-low bitrate range below $0.01$ bpp, image compression becomes a constrained generation problem: the reconstruction depends on inferring plausible missing details while staying close to the input image. Autoregressive token models provide an efficient coarse-to-fine prior for this setting, but the transmitted early-scale tokens often leave object identity and texture underdetermined, which can lead to source-inconsistent reconstructions. We introduce Source-Anchored Visual Autoregression (SAVAR), a generative codec built on Visual Autoregressive (VAR) modeling. SAVAR transmits early-scale tokens and a compact caption. The tokens provide image-specific layout information, while the caption supplies global semantics. SAVAR reconstructs a partial VQ-VAE latent from the retained scales, maps it to the start-of-sequence initialization, and uses the same latent to condition the caption embedding before adaptive layer normalization, separating two roles that are typically driven by a shared condition in VAR-style generation. On the Kodak and CLIC2020 datasets in the extreme-low range, SAVAR reduces perceptual distortion relative to matched ARPC retained-scale operating points, lowering LPIPS by about $8$--$9\%$ and DISTS by up to $22\%$ at a lower bpp. It reconstructs a $512 \times 512$ image in about $218$ ms on one NVIDIA L20X GPU, retaining fast autoregressive decoding in the evaluated setting.

0.14

Data Density Scaling Laws for Image Self-Distillation

cs.LG ·2026-10-13 ·Alvard Barseghyan, Ani Vanyan, Hakob Tamazyan, Hrant Khachatrian
similarity 0.67
Abstract · 32 pages, 13 figures

Current scaling law formulations suggest that increasing the number of unique samples in a dataset consistently improves model performance, while repeated exposure to the same samples provides diminishing value. In this work, we challenge this assumption for image self-distillation models. We train iBOT models on random subsets of seven datasets spanning satellite imagery, driving scenes, egocentric video, and natural images, and evaluate their frozen representations on downstream tasks. We propose a density-interaction scaling law that combines unique-image count $U$ and repetition $R$, fitting it on runs with at most 50 million image presentations and testing it at larger budgets $C$. On Maxar dataset, a random subset containing approximately 1.5% of the training pool improves average nearest-neighbor classification accuracy by 3.11 percentage points over training on the full pool at the same budget of 200 million image presentations. Smaller subsets also outperform the full pool on SATLAS and Ego4D, whereas ImageNet favors larger pools at higher budgets. At a fixed subset size, further training can improve performance, lead to a plateau, or reduce performance. Our law captures smaller-subset advantages that the compared baselines cannot represent and improves prediction at higher budgets on Maxar, SATLAS, and Ego4D under nearest-neighbor evaluation. These findings show that both the number of unique images and how often they are repeated matter when allocating a training budget.

0.33

Discriminative Score Function: Turning Pretrained Models into Functional Generative Priors

cs.CV cs.LG ·2026-10-14 ·Junhoo Lee, Hyeonjin Kim, Sangbum Han, Nojun Kwak
top venuecode/data similarity 0.67
Abstract · 20 pages. Accepted at the Fortieth Annual Conference on Neural Information Processing Systems (2026)

Discriminative models are trained with task-specific objectives such as classification, detection, or representation learning, yet their practical utility depends on generalizing beyond the training task to unseen samples from the same domain. This suggests that a discriminative model may retain an implicit trace of the data manifold beyond its task labels. We ask whether this trace can be exposed as a generative signal without task-specific guidance. We introduce the Discriminative Score Function (DSF), which extracts an image-space generative field from the loss-gradient geometry of a pretrained discriminative model. DSF provides a training-free update rule that moves images toward model-consistent visual regions without retraining or task-specific targets. We apply DSF to diverse pretrained discriminative models, including ResNet classifiers, DETR detectors, and DINOv2 ViT encoders. DSF generates unconditional images from noise and supports conditional guidance, editing, inpainting, and explanation through the same field. DSF-generated samples also provide effective zero-shot quantization calibration when real images or labels are unavailable, preserving feature geometry better than data-free calibration baselines. These results show that discriminative training leaves a generative functional signal for synthesis and calibration. Project page: https://junhoo.me/dsf.

0.69