CS Paper Radar

Benchmarking color judgment in multimodal LLMs with CIE-computable ground truth

Shuwei Yue, Kaida Xiao, Zhihua Wang, Qiao Chen

cs.CV ·submitted 2026-10-29 14:58 UTC · v1
benchmark

Abstract

Multimodal large language models (MLLMs) are now used as zero-shot judges of image quality, but their color judgments are rarely tested with colorimetric ground truth. We present a five-task benchmark based on the CIE colorimetric framework: AWB pair judgment, skin-tone bias direction, CCT estimation, CIE Delta E 2000 pair ordering, and color naming under changing illuminants. We test eight MLLM endpoints, including four Qwen3-VL variants, GLM-4.5V, GPT-5.4, Claude-Opus-4.7, and Gemini-3.1-Pro. The models often solve coarse binary AWB judgments, but they remain weak on numeric CCT estimation and on some fine color-difference judgments. On ColorChecker-24 stimuli, the three closed-weight frontier models reach CCT MAE of 715-1056 K, compared with 1358-2133 K for the five open-weight models. A non-ColorChecker control shows that this advantage is partly template-dependent. The results suggest that current MLLMs have useful color words and coarse visual priors, but their color judgments are not reliably calibrated to CIE-computable color differences.

Closest papers

Nearest neighbours by meaning across the whole index.

Where Do Vision Language Models Fail at 3D Reasoning?

cs.CV cs.AI ·2026-10-29 ·Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song +9 more
benchmarkcode/data similarity 0.70
Abstract · Preprint, 150 pages

Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twelve open-weight and four frontier models, per-color counting exposes a pronounced weakness: humans achieve 93.3% accuracy, compared with 7.2% for the best model. Targeted fine-tuning raises one model's accuracy from 3.0% to 68.6%, but reveals a further difficulty: its answers about the most common colors remain inconsistent with its own per-color counts. Supplying those counts as text improves its judgments, showing that recovering a property does not mean using it consistently. Reasoning, scaling, and added visual cues each improve individual judgments, but none makes exact answers reliable. To locate the difficulty, we probe Qwen3.5-27B: exact per-color counts remain recoverable at the language model's input, yet the model's answers rarely recover them (65.0% for the probe vs. 3.0% for the model). Image-removal controls further show that some apparent successes require no visual evidence. These findings motivate evaluating 3D reasoning by exact correctness, consistent use, and image dependence, rather than accuracy alone. Code and dataset are publicly available: https://github.com/leduckhai/BedBench

0.64

MCIQA-2K: A Multi-Dimensional Dataset and No-Reference Quality Assessment Benchmark for Colorized Images

cs.CV ·2026-09-13 ·Yunkai Zhuang, Qihang Yan, Zicheng Zhang, Guangtao Zhai
benchmarkdatasetcode/data similarity 0.65
Abstract · 15 pages, 5 figures, 2 tables

Image colorization is an inherently ill-posed task, since a single grayscale image may correspond to multiple plausible colorized results. Consequently, conventional full-reference image quality assessment (IQA) metrics fail to accurately reflect human perceptual preferences for colorized images. In this paper, we present MCIQA-2K, a large-scale multi-dimensional benchmark specifically designed for no-reference quality assessment of colorized images. We construct a dataset containing 2,000 colorized images generated by five representative colorization models, together with human annotations across three perceptual dimensions: color smearing, semantic color misalignment, and global naturalness. Building upon the proposed benchmark, we further introduce MCIQA, a dedicated multi-branch NR-IQA framework for colorized images. Extensive experiments demonstrate that MCIQA significantly outperforms existing full-reference and no-reference IQA methods on the proposed benchmark, while also exhibiting competitive generalization capability on several widely-used IQA datasets. The dataset and code are publicly available at https://github.com/ARBEZ-ZEBRA/MCIQA.

0.37

Mutual Colormap Dependence: How Recoloring Biomedical Images Changes Vision-Language Model Answers

eess.IV ·2026-11-11 ·Siddharth Vohra, Runmin Jiang, Min Xu
publishedcode/data similarity 0.63
Abstract · Accepted to the FABRIC workshop at WACV 2027. 10 pages, 1 figure, 5 tables. Project page: https://siddvoh.com/mutual-colormap-dependence/

Vision-language models (VLMs) are being built to answer questions about biomedical images, yet they never see the measured intensities, only the colors that a colormap has assigned to them. For questions about image content, the colormap should not change the answer. We call the extent to which it does Mutual Colormap Dependence (MCD), the mutual information between the colormap and a model's answer, which is zero exactly when every colormap yields the same answer distribution. Because VLMs sample their answers, we compare recoloring with a control that uses the same number of calls to ask again about the unchanged gray image, so that ordinary resampling is not mistaken for an effect of the colormap. We test GPT-5.4, Gemini 3.1 Pro, Claude Sonnet 4.6, and Qwen3-VL-235B under six luminance-monotonic colormaps (gray, viridis, plasma, inferno, magma, and cividis) on 50 medical visual-question-answering items from 16 SLAKE images, 64 ChestX-ray14 chest radiographs, and 200 BBBC039 fluorescence-microscopy images for exact nucleus counting. Every prompt tells the model to ignore the colormap. Even so, with three answers per colormap on the chest radiographs, the answers of all four models depend on the colormap (all Holm-adjusted $p \leq 0.002$). For Claude, some colored map overturns the majority answer on 29 of the 64 radiographs, where resampling gray alone overturns about 4, and Qwen's accuracy falls from 0.54 under gray to between 0.32 and 0.41 under the colored maps, mostly because it abstains. With one answer per colormap, recoloring adds disagreement beyond repeated sampling in 5 of 12 model-panel cells under per-cell 95% intervals, and repeating the instruction to ignore color does not reliably reduce it. The colormap is therefore part of what a VLM sees, and it can change the answer.

0.37

The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence

cs.CV cs.CLcs.LG ·2026-09-01 ·Genpei Zhang
similarity 0.63
Abstract · 17 pages (7-page main text plus technical appendix), 10 figures, 6 tables

Vision-language models are evaluated by aggregate accuracy on multimodal benchmarks, a practice that implicitly assumes the model uses its visual input. We show this assumption fails on 40%--97% of samples across six VLMs and three perceptual benchmarks: blurring the question-relevant visual region leaves the next-token distribution nearly unchanged. We name this phenomenon the Visual Insensitivity Gap and quantify it with a per-sample Visual Sensitivity Index (VSI). The gap is a property of samples, not of models: VSI ranks correlate across models (grand-mean Spearman rho=+0.40, permutation p<10^-3), so the same samples are flagged insensitive by VLMs sharing no architectural detail beyond a contrastively pretrained vision tower. The mechanism is concrete: on the insensitive samples, a linear probe on each model's own vision tower distinguishes perturbed from clean images at 0.72--0.79 accuracy, yet the model's argmax token changes on only 2%--11% of the same samples, an encoder--LLM gap above 0.65 on every model. Mapping VSI's diagnostic utility cell by cell surfaces a strong regime (multi-choice reasoning on capable VLMs: AUROC=0.85--0.87) and a weak regime (well-calibrated factuality, where softmax confidence already leads). VSI is not a universal best abstention signal; it is a sample-intrinsic indicator of vision-ignoring failure, best used as a conditional ensemble component.

0.52

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

cs.AI cs.CLcs.CV ·2026-11-02 ·Ahmad Jarjis Hasan, Jonathan Lwowski
publishedbenchmarkcode/data similarity 0.62
Abstract · Accepted at WACV 2027 (Evaluations and Datasets Track). Code and data: https://github.com/ahmadhasan2k8/grader-audit

Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left unexamined, yet it sets the published number. We reproduce the official scorers of eight peer-reviewed benchmarks on five open vision-language models (VLMs) that fit a single 16 GB GPU, and introduce a blinded human-plus-LLM-judge adjudication (Cohen's kappa=0.956) that recovers what the model actually answered. The scoring rule alone moves accuracy by up to 22 points on identical outputs, yet exactly 0 wherever a model emits clean option letters, so it hides on strong-model leaderboards. It is the fallback and not the rambling that does the damage: the benchmark with the highest non-compliance we measure has an artifact of exactly zero, because its scorer abstains instead of guessing; a reasoning model with non-compliance 0.000 still swings 9.5 points with which regex match of its trace the parser keeps. No standard rule is reliable: snapping a non-answer to its closest option credits answers never given yet buries correct ones, abstention discards answers stated in words, and likelihood scoring follows answer-token priors, so a single grade conflates capability with luck rather than measuring either. We release all code, labels, and the protocol, and argue benchmarks should report a non-compliance rate and at least two rules, not one unexamined grade.

0.39

What Looks Like a Capability Limit in Vision-Language Models Is a Readout Limit

cs.CV cs.AIcs.CL ·2026-09-23 ·Alfredo F. Frontera Del Valle
benchmark similarity 0.61
Abstract · 14 pages, 1 figure, 8 tables

Benchmarks for vision-language models offer their answer choices in some convention: a letter, a color name, a pixel coordinate. That convention is treated as neutral. We find it is not, and that the limits a benchmark reports can belong to the readout rather than to the model. On 200 COCO photographs, Qwen3-VL-4B picks the correct one of nine locations for a named object 68.5% of the time when the locations are given in English and 20.0% when the same locations are given as pixel coordinates. Chance is 11.1%. The cost arises when the answer options are coordinates; giving the model a coordinate in the question instead costs 3.5 points and is not significant. The gap holds on a 4x4 grid, under 8-bit rather than 4-bit quantization, and in every slice by object size, boundary distance and category. It also decides which model wins. Two models that tie under English names differ by 39 points in one coordinate system and by 54 in the other, in opposite directions. On the color task, three of the four open models capable of the task show the penalty; on photographs, two of three open models do, and so does Gemini, at 11.1 points on parseable answers (p = 1e-4). GPT-4o does not. To ask whether a model reads a coordinate at all, we attach the wrong name to each one and record which the model follows. Color options written as hue angles are followed below chance; a normalized pixel convention is followed at four times chance. This tells apart conventions a model can use from ones it cannot, though it did not predict accuracy on two untried conventions. Five models also name the same color wheel five different ways, so a fixed answer vocabulary is not neutral across models either. Five times during this work we measured a capable model as incapable because our scorer and the model disagreed about what an answer looks like. We report each case. They are the phenomenon in miniature.

0.34

HALDETECT at ImageEval 2026 Shared Tasks: Answer-First Contrastive Grounding with QLoRA

cs.CV cs.AI ·2026-09-10 ·Syed Mohaiminul Hoque, Md Sakhawat Hossain
published similarity 0.60
Abstract · 10 pages, 3 figures, 10 tables (including appendices). System description paper for Task 1b (English) of ImageEval 2026 Shared Tasks (Fourth…

Large multimodal models tend to hallucinate visual detail fluently, which limits their deployment for fine-grained interpretation. We present HALDETECT, our system for the English hallucination-detection track (Task 1b) of ImageEval 2026, in which a system must identify, from an image and three culturally plausible statements, the single visually grounded one. We frame the item as one contrastive decision, emit the answer before its explanation, and structure reasoning around colour/texture, shape/form, and context. Our best submitted adapter fine-tunes Qwen2.5-VL-7B-Instruct with 4-bit QLoRA while freezing the vision encoder and reaches Contrastive Instability (CI) 0.035 on the 1,000-item test set; we placed third of eight teams. Development experiments show that answer order can matter more than model scale and that adaptation beats prompting alone. Retrospective paired analysis of the released gold labels confirms the QLoRA gain over the best prompt but not the small gap between the devtest-selected and best-test adapters, and reseeding all four training sizes shows that the apparent data-scaling curve does not survive a seed change. The 35 residual errors are culturally plausible function, material, and recognition distinctions; naive adapter voting does not help.

0.29

Paint-Anything: Unified Any-Color Control for Image Generation and Editing

cs.CV cs.AIcs.LG ·2026-09-17 ·Ji Xie, Dewei Zhou, Xinyu Huang, Zhennan Chen, Xun Wang
benchmark similarity 0.60
Abstract · 29 pages, Seed Technical Report. HTML compatibility fixes; scientific content unchanged

Professional design requires any-color control: the ability to specify an object's target color with any 24-bit hex value for image generation and editing. Prior work has explored color generation, editing, and colorization, but often relies on dedicated color representations or specialized inference procedures. Advances in large language models offer a simpler starting point: even compact models can associate hex values with color semantics. We present Paint-Anything, which learns a shared hex-prompt interface for generation and editing through object-level color supervision. We develop a data pipeline that constructs Paint-500K from real images through object grounding, perceptual color labeling, and editing-pair synthesis. Since shadows make real-image labels only approximate colors, we complement this supervision with pure-color anchors whose pixels exactly match their paired hex values. These anchors are used only at high-noise timesteps, leaving low-noise training to natural images. We further introduce Any Color Benchmark (ACBench), comprising ACBench-T2I and ACBench-Edit, to measure object-level hex color fidelity across both tasks. On FLUX.2-4B, Paint-Anything improves ACBench-T2I and ACBench-Edit scores by 85.3% and 28.3%, respectively, relative to the base model, with ablations supporting the training recipe. It also achieves the highest average CompColor score among the compared methods.

0.15