CS Paper Radar

WildFakeToolBench: Benchmarking AI-Generated Image Detectors with Web-Sourced Image Processing

Mingjie Li, Yukun Jiang, Yicong Tan, Yang Zhang

cs.CV cs.CR ·submitted 2026-10-22 12:09 UTC · v1
benchmark

Abstract

As AI generated images are getting popular these days, discussions on such generations are widely existed in current public forums, software repositories, and photography tutorials. Moreover, with the help of LLM-assisted code translation, malicious actors can adopt such information to evade detectors without specialized knowledge or local GPU execution of the resulting tools. To evaluate such risks, we introduce \textsc{WildFakeToolBench}, a benchmark suite of $156$ executable image-processing tools derived from $220$ public sources across $142$ domains. Each tool retains its source evidence and passes syntax, execution, and output-distortion checks without using feedback from the evaluated detectors. On $105$ generated images from five generators, we evaluate six state-of-the-art detectors with images processed by the tools in our collected bench to assess their effectiveness against such in-the-wild risks. The strongest individual tools achieve attack success rates of $25.25\%$--$100\%$ across detectors, exceeding the strongest of six selected image-processing baselines for each detector. Evaluating subsets of an eight-tool pool yields worst-case coverage above $80\%$ for every detector. Effectiveness is sparse and detector-specific: high coverage across candidates does not imply reliable one-shot evasion. These findings not only highlight the potential of such web-sourced risks, but also show the potential on establishing AI-image detector robustness evaluation framework based on those in-the-wild information.

Closest papers

Nearest neighbours by meaning across the whole index.

Video-Index: A Curated Meta-Benchmark for Video Understanding

cs.CV ·2026-10-01 ·Enxin Song, Yinuo Xu, Shusheng Yang, Wenhao Chai, Jiatao Gu
benchmarkcode/data similarity 0.67
Abstract · Blog: https://www.enxinsong.com/blog/video-index/ GitHub: https://github.com/Espere-1119-Song/Video-Index Hugging Face: https://huggingface.…

A video benchmark should reward the capability it claims to measure, yet models can exploit answer options, question text, or partial visual evidence. We introduce the attack pyramid, five levels of shortcut attacks with increasing access to each item, and audit 115 video benchmarks with it. On 35 benchmarks, attackers that never see a frame approach full-video accuracy. On 51 benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy. Near-duplicate questions make up at least half the items in 63 benchmarks. We screen 505,518 question-answer pairs from 112 of them into an audited pool. Agents turn evaluation requests into specifications, and a deterministic selector with a red-team gate composes reproducible benchmarks. We release Video-Index, the 210 hardest verified items under these attacks in each of four capability groups, 840 items from 76 sources. With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy. Blog: https://www.enxinsong.com/blog/video-index/ GitHub: https://github.com/Espere-1119-Song/Video-Index Hugging Face: https://huggingface.co/datasets/Video-Index/Video-Index

0.55

Tricking Synthetic Image Detectors with Amortized Generative Attacks

cs.CV ·2026-10-18 ·Mohamed Awad, Khaled Elhady, Yasser Rohaim, Samuele Poppi, Nils Lukas
similarity 0.67
Abstract

Rapid advances in image synthesis have increased the need for reliable synthetic-image detectors, yet their robustness to transferable attacks remains incomplete and understudied. Existing evaluations generally examine whether an attack constructed against one detector transfers to another, but rarely isolate how its effectiveness changes when the attacked images originate from different generators. We conduct a systematic evaluation of three representative attacks: two iterative perturbation-based methods, MIG and MuMoDIG, and the amortized generative attack Tricker. Across four detectors and five generators spanning diverse architectures, we introduce a two-axis evaluation protocol that independently varies the surrogate detector used to construct or train the attack and the generator supplying its input images. We assess attack effectiveness, cross-detector transferability, perceptual quality, runtime, and interaction with common post-processing operations. We revisit Tricker and show that its data and training requirements can be substantially reduced, enabling broader and more efficient evaluation. Our results demonstrate that transferability is not a detector-only property: the input-generator distribution strongly influences both evasion effectiveness and attacked-image quality. MIG and MuMoDIG exhibit pronounced dependence on the detector--generator pair, whereas Tricker provides more uniform white-box evasion and image quality. We additionally observe that post-processing maintains the asymmetric transferability topology. These findings expose a neglected dimension of attack evaluation and motivate transfer-aware protocols that jointly account for detector and generator variation. We will release our code, evaluation data, and trained attack models with a responsible-use statement.

0.08

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

cs.CV cs.AI ·2026-09-26 ·Aleksandr Gushchin, Khaled Abud, Georgii Bychkov, Ekaterina Shumitskaya, Artem Filippov +3 more
benchmarkdataset similarity 0.65
Abstract

AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or processed images. Existing datasets provide limited support for evaluating these challenges jointly across diverse visual content. We introduce REALIS, a dataset of 1.43 million real and synthetic images generated by 42 modern text-to-image models, including the latest proprietary systems such as Nano Banana 2. REALIS combines prompts derived from real images, quality filtering, and stratified sampling to reduce class-specific shortcuts while preserving content diversity. We further introduce REALIS-Expert, a stress-test subset for high-quality synthetic images, where real and generated samples are selected with closely matched semantic and visual characteristics. We also propose a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing. Based on REALIS, our benchmark evaluates pretrained detectors, fine-tuned models, and zero-shot vision-language models under generator and post-processing shifts. On the hardest processed split, the best pretrained conventional detector achieves 0.550 ROC-AUC, compared with 0.752 for the best REALIS-trained detector. REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use.

0.10

Training-Free Reconstruction-Based AI-Generated Image Detectors Are Inherently Vulnerable to Adversarial Examples

cs.CV ·2026-08-17 ·Roman Demchenko, Jonas Ricker, Asja Fischer
published similarity 0.65
Abstract · Accepted at AI4MFDD (AI for Multimedia Forensics & Disinformation Detection) Workshop, ECCV 2026

The impressive visual quality and ubiquity of AI-generated images call for reliable and robust detection methods. Reconstruction-based detectors have emerged as a promising direction for transparent and training-free identification of synthetic images. However, due to their fundamentally different mode of operation (compared to standard, classifier-based methods), little is known about their adversarial robustness. In this work, we propose two novel attack methods targeted at detectors that leverage autoencoder reconstruction error. We find that by constructing imperceptible adversarial examples, the distance between original and reconstruction can be artificially increased, causing fake images to be wrongly classified as real. Our evaluation including images from three state-of-the-art generators and three detectors demonstrates that detection performance is significantly decreased, even if attacked images additionally undergo real-world degradations. Critically, our adversarial examples naturally transfer across detectors, as they all share the same principle, pointing towards an inherent vulnerability of reconstruction-based detectors.

0.20

WildAIGC-Bench: Can AI Image Detectors Handle the Wild? A Benchmark of Generated and Edited Images in Online Circulation

cs.CV ·2026-10-20 ·Zijian Yu, Hangwei Zhang, Jun Lan, Huijia Zhu, Weiqiang Wang +1 more
benchmarkcode/data similarity 0.65
Abstract · 19 pages, including references and supplementary appendix

AI-generated images linked to news events and online claims can mislead audiences, enable fraud, or cause public controversy. We introduce WildAIGC-Bench, built through recurring collection across news websites, fact-checking sources, and related social media content. The pipeline combines broad coverage, timely updates, and human-verified labels through scheduled discovery, automatic screening and preliminary annotation, and human review of image-specific evidence. The benchmark contains 407 verified AI-generated or edited images, each accompanied by a source webpage link and an evidence summary. Images retain the conditions encountered online, including compression, screenshots, and recapture. Evaluation of dedicated visual detectors, multimodal large language models, and a simple Perception Encoder baseline reveals substantial recall gaps relative to existing benchmarks. WildAIGC-Bench provides both a practical detection challenge and a basis for continued data expansion and source-based verification. The dataset is released at https://github.com/YuZijian/WildAIGC-Bench.

0.36

FACT: A Forensic Agent with Compiled Tool-Use Trajectories for AI-Generated Image Detection

cs.CV cs.AI ·2026-09-05 ·Jiaoyang Chen, Bin Hu, Jingyu Hu, Kun Zhou, Qin Zhang +1 more
similarity 0.64
Abstract

AI-generated image detection is increasingly open-world: new image generators produce highly realistic images that make visual artifacts harder to identify. Existing detectors usually rely on a fixed set of forensic cues, so a detector that works well for one generator family may fail on another. We introduce FACT (Forensic Agent with Compiled Tool-use Trajectories), which learns an image-conditioned tool-use policy for forensic analysis. Instead of applying a fixed detector, FACT decides which forensic tools to call, interprets the returned evidence, and stops when sufficient evidence has been collected. FACT follows an Evolve--Distill--Refine pipeline: it evolves an execution-verified forensic skill, compiles the skill into action--observation tool-use trajectories, distills them into a compact agent, and refines the policy with cost-aware GRPO. Across two internal and four public benchmarks, FACT achieves the best performance among all compared methods, including on recent unseen generators, deepfakes, and manipulated images.

0.27

Must You See Fakes to Catch Fakes? Language Says What to Look For, Real Images What to Ignore

cs.CV ·2026-11-17 ·Tianchi Xu, Yuxuan Zhou, Yuheng Zhang, Yanan Guan, Tao Dai +1 more
similarity 0.64
Abstract

Must a detector see generated images in order to catch them? Detection of AI-generated images has long faced two problems. The cues a detector learns from the few generators in its training set carry traces specific to those generators and transfer poorly to new ones, which limits generalization. The real and generated training images also differ in format and content, so the detector takes these differences as evidence of authenticity, and its decisions change with factors that have nothing to do with authenticity, which harms robustness as well as generalization. Both problems stem from training on generated images. A detector never trained on them can largely avoid both at the source. Vision foundation models have already learned the concept of "AI-generated" during pretraining, and text can point to it directly. Building on this, we propose SaySee, a training-free detector: Say builds a generation direction from sets of paired descriptions that differ only in authenticity and scores an image by the projection of its embedding onto it, and See uses real images to correct the score offsets caused by format, image type and content. On 24 evaluation sets that span standard benchmarks, in-the-wild data and the latest generators, SaySee reaches a mean AUC of 98.2% with MoE-ViE, 4.2 points above the best existing detector. When format is decoupled from authenticity, it remains the best of all methods. Code will be released.

0.27

ImageRSI: Verification-Driven Recursive Self-Improvement for Agentic Image Generation

cs.CV cs.AI ·2026-10-23 ·Shuo Han, Zhilin Wang, Peng Ling, Wenjia Gong, Yang Liu +3 more
benchmark similarity 0.64
Abstract

Complex text-to-image requests remain difficult to satisfy reliably. Verifying and repairing an image can fix an individual output, but a successful repair does not necessarily carry over to other requests. We present ImageRSI, a framework that keeps the image generator frozen and combines three components: five-dimensional verification and repair within each task, an experience library shared across tasks, and offline updating of the judge. Candidate experiences enter the library only after paired A/B validation, and the judge is updated with supervised fine-tuning and GRPO on accumulated image-judgment records and evaluated before reuse. We also introduce ImageReq-Bench, a benchmark of 961 requests built around hard cases in real-world image-generation requests, each paired with an evaluation checklist. ImageRSI achieves a Checklist Strict Accuracy (CSA) of 52.6% on MindBench (with Nano Banana Pro as the generator) and 65.66% on ImageReq-Bench, and we compare system outcomes across judge-training configurations.

0.35