CS Paper Radar

Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?

Ziyuan Wang, Genliang Zhu, Xiaobo Chen

cs.LG cs.CLphysics.optics ·submitted 2026-10-30 22:21 UTC · v1
code/dataunusual

Abstract

A spectrum can be given to a model as a table in nanometers or as a plot in photon energy, and the decision should not depend on which one it sees. When the data needed for an answer are missing, the model should say so. We test representation invariance and abstention on measured optical spectra of silicon metasurfaces (MOCLIP) and gold nanoparticles, with labels computed from the data. Each spectrum is shown as four tables (nm, $μ$m, scientific notation, eV) and two plots (nm, eV), and partial spectra remove data inside or outside the queried range. We evaluate the decision models d3-nano and Jev 1.13, three GPT-6 Luna configurations and the Qwen3.5-2B base model. Changing only the representation moves accuracy by up to 41 percentage points within one model, mostly on photon-energy tables and plots. With the same evidence, asking in photon energy instead of wavelength helps some model-view pairs by 16-50 points and hurts others by up to 25. When the relevant extremum is removed, unadapted models rarely answer "undetermined". LoRA fine-tuning of d3-nano improves comparisons on the trained table formats, but much of its gain on unanswerable questions comes from a shortcut: it abstains whenever the input is truncated. Training with answerable truncations avoids the shortcut in all six seeds, with a 90-100 point gap between abstention after inside and outside cuts on held-out structures; with the same amount of extra original-task data, five of six seeds keep the shortcut. On gold spectra the two training sets perform similarly. Multimodal language models used for scientific decisions should be tested on equivalent representations and on missing evidence, not only on average accuracy. We release the tasks, per-response results and analysis code.

Comment
20 pages, 4 figures, 11 tables. Code and data: https://github.com/zwang-devv/spectra-decision-audit

Closest papers

Nearest neighbours by meaning across the whole index.

Benchmarking color judgment in multimodal LLMs with CIE-computable ground truth

cs.CV ·2026-10-29 ·Shuwei Yue, Kaida Xiao, Zhihua Wang, Qiao Chen
benchmark similarity 0.54
Abstract

Multimodal large language models (MLLMs) are now used as zero-shot judges of image quality, but their color judgments are rarely tested with colorimetric ground truth. We present a five-task benchmark based on the CIE colorimetric framework: AWB pair judgment, skin-tone bias direction, CCT estimation, CIE Delta E 2000 pair ordering, and color naming under changing illuminants. We test eight MLLM endpoints, including four Qwen3-VL variants, GLM-4.5V, GPT-5.4, Claude-Opus-4.7, and Gemini-3.1-Pro. The models often solve coarse binary AWB judgments, but they remain weak on numeric CCT estimation and on some fine color-difference judgments. On ColorChecker-24 stimuli, the three closed-weight frontier models reach CCT MAE of 715-1056 K, compared with 1358-2133 K for the five open-weight models. A non-ColorChecker control shows that this advantage is partly template-dependent. The results suggest that current MLLMs have useful color words and coarse visual priors, but their color judgments are not reliably calibrated to CIE-computable color differences.

0.21

Robust small-molecule identification from incomplete, degraded, and inconsistent spectra using multimodal mixed-condition training

cs.LG ·2026-09-13 ·Bowen Gao, Lei Zhu, Yiying Wang, Wenjie Yu
similarity 0.51
Abstract · 17 pages, 7 figures, 2 tables. Supplementary information: 12 pages, 3 figures and 11 tables, provided as an ancillary PDF. Revised content

Reliable small-molecule identification often requires complementary evidence from multiple spectroscopic measurements. In practice, however, spectra may be unavailable, degraded by measurement-related variations, or even incorrectly associated with a sample, thereby hindering accurate molecular identification. Herein, we propose a multimodal mixed-condition training strategy that accommodates missing, degraded, and mismatched measurements for small-molecule structure identification. The strategy incorporates chemical and spectroscopic knowledge through predefined missing-input configurations, modality-specific spectral perturbations, and chemically informed spectrum replacements. Models were trained on 635,441 samples comprising mass spectrometry (MS), infrared (IR), and nuclear magnetic resonance (NMR) simulated spectra from the Multimodal Spectroscopic Dataset (MSSD). They were then systematically evaluated on 79,462 held-out samples across 30 views designed to represent variations in spectra. A controlled comparison of complete-input and mixed-condition training under concatenation and mixture-of-experts (MoE) fusion showed that the training strategy was the principal source of improvement. For MoE, mixed-condition training increased the mean reciprocal rank (MRR) by 6.08% (from 0.9203 to 0.9763) and the top-1 molecular identification rate by 7.67% (from 89.50% to 96.36%). Notably, under single-modality inputs, IR MRR increased 2.15-fold (from 0.4337 to 0.9307), while MS MRR increased 2.31-fold (from 0.3711 to 0.8575). With the proposed strategy, complete-input performance remained high, while sample-level mismatch detection also improved. Together, these results highlight the potential of multimodal mixed-condition training for practical molecular identification by explicitly addressing incomplete, degraded, and mismatched measurements encountered in real-world analysis.

0.40

Every Measurement, Every Direction, All at Once: Multimodal Flow Matching for Molecules and Spectra

cs.LG physics.chem-ph ·2026-11-17 ·Thorben Prein, Elton Pan, Rafael Gómez-Bombarelli, Santiago Miret
top venueposition similarity 0.51
Abstract · 30 pages, 12 figures. Accepted at NeurIPS 2026

Generative modeling in science is often data-limited, where experiments are expensive. However, each experiment may yield several coupled measurements of the same physical entity. Standard generative models often train only on a single direction, discarding supervision from the rest. We argue that each experiment contains more training signal than this asymmetry allows the model to exploit: random masking enables measurements to serve as either conditioning inputs or prediction targets, yielding multiple supervised denoising tasks from a single experiment. We investigate this premise with MOSAIC, a multimodal flow matching transformer over molecular 3D structure and UV, IR, and Raman spectra. MOSAIC is a single model that is capable of any-to-any generation: generating any subset of modalities from any disjoint subset of the others. On QM9S, MOSAIC achieves 88.8% molecular graph recovery, outperforming the strongest spectra-to-structure baseline by 23.2% in structure generation, while simultaneously outperforming a structure-to-spectra specialist model in spectral generation. Ablations show that the any-to-any objective improves structure generation over structure-only training across model sizes. Furthermore, MOSAIC supports arbitrary subsets of available measurements and enables molecular identification by matching generated spectra to experimental references.

0.44

Beyond CCT: Multi-channel SPD recovery for per-shot camera color correction

cs.CV ·2026-10-29 ·Shuwei Yue, Hongke Wei, Qiao Chen
benchmark similarity 0.51
Abstract

Modern cameras correct color via two or three pre-calibrated 3x3 matrices interpolated by Correlated Color Temperature (CCT). This collapses the illuminant SPD to a scalar. LEDs, fluorescents, and mixed sources sit off the Planckian locus, and metameric illuminants with the same chromaticity demand different optimal CCMs that a CCT pipeline cannot distinguish. We recover the illuminant SPD instead using non-negative least squares against a basis dictionary, conditioned on the camera white point and a multi-channel ambient-light sensor (ALS). A closed-form 3x3 CCM is then fit per shot. On a synthetic benchmark of 300 illuminants and 3,100 reflectances, mean angular error drops from 1.89 to 1.20 degrees, with a 1.7x tail reduction on the LED-like off-Planckian group (n=45) where metameric ambiguity is tangible. The gain holds under realistic ALS imperfections and observation noise, and the same off-Planckian advantage carries over to 1,684 real measured SPDs.

0.25

Broadband Mid-Wave Infrared Imaging Enabled by All-Silicon Hyperboloid Metalens

cs.AI physics.optics ·2026-10-21 ·Rose Johnson, Debabrata Mandal, Zihao Zou, Jessie Wei, Anna Wirth-Singh +4 more
unusual similarity 0.49
Abstract

The mid-wave infrared (MWIR, 3-5 microns) is indispensable for thermal sensing, gas detection, long-range imaging, and free-space optical communications. Meta-optics offer a compact, lightweight, and potentially low-cost alternative to conventional refractive optics, making them attractive for size, weight, and power (SWaP)-constrained systems. However, their strong chromatic dispersion has largely restricted imaging to narrow spectral bands. Here, we demonstrate broadband imaging across the full 3-5 micron MWIR band using a single-layer silicon metasurface coupled with computational reconstruction. Remarkably, the reconstruction model is adapted using only four image captures, overcoming large training data requirements and enabling broadband image recovery despite the strongly wavelength-dependent optical response. In contrast to approaches that engineer complex, dispersion-optimized metalens profiles for broadband operation, our method leverages the wavelength-dependent information encoded by a simple 2cm diameter hyperboloid metalens. We show that this metasurface captures sufficient broadband intensity information for computational reconstruction across the entire MWIR band. These results establish a simple route toward broadband computational meta-optics without requiring broadband achromatic metalens designs, offering a promising platform for compact MWIR imaging systems.

0.25

When is a closed-form RGB->S/P ratio adequate? A hyperspectral characterization on natural scenes for mesopic display

eess.IV cs.CVcs.GR ·2026-09-21 ·Naoyuki Uchida
unusual similarity 0.49
Abstract · 14 pages, 5 figures, 2 tables

Mesopic and low-light display transforms require, as their driving signal, a per-pixel scotopic-to-photopic luminance ratio (S/P); the exact spectral S/P is unavailable for ordinary RGB content, so a low-cost closed form that estimates S/P from a linear-RGB triplet is used in its place. Such closed forms exist but have been characterized only on narrowband / LED sources, i.e. spectrally sparse spectra, where a relative error of ~41% has been reported for a three-channel projection. Display content, however, is natural and broadband. We ask whether the same closed form is adequate there, using per-pixel spectral S/P from hyperspectral imagery as ground truth. On a daylight radiance time-series, a six-scalar closed form (three photopic and three scotopic channel weights) reproduces spectral S/P with a median error of ~0.07 that is time-invariant once the RGB input is chromatically adapted to D65; evaluated in un-adapted sRGB the error instead carries a color-temperature tilt across illuminants (~0.19), so adaptation is the enabling step for this use case. The result generalizes to an independent fifty-scene set (pooled median 0.024; 45/50 scenes within a pre-registered 0.10 band), with the few exceedances concentrated in saturated, spectrally-peaky surfaces that approach the narrowband regime (floral close-ups in this set). The scotopic weight vector is shown to be primary-model dependent, but the value used here is corroborated by a primary-free XYZ projection, and the median error stays within the band across all principled coefficient choices. We do not claim observer-validated appearance fidelity or adequacy on narrowband sources; both are out of scope. Both outcomes follow from the same three-channel projection: it is overwhelmed by spectrally sparse inputs and adequate on spectrally smooth ones.

0.30

Searching the Pareto Front: Multi-Objective Retrieval for Heterogeneous Scientific Modalities

cs.AI astro-ph.IM ·2026-11-18 ·Song Young Oh, Nesar Ramachandra, Bin Xia, Ian Foster, Kyle Chard
similarity 0.48
Abstract

In cosmology, surveys span different sky regions and wavelength ranges, resulting in uneven modality coverage across observational datasets. Such heterogeneity can be addressed through either multimodal foundation models (FMs), which learn shared embedding spaces, or independently learned modality embeddings combined via late fusion. We investigate whether the latter provides an adaptive alternative to FM-based scientific retrieval. We evaluate this approach on a galaxy retrieval task spanning diverse modalities--history of star formation, spectra of light emitted, and multi-wavelength observations from two telescopes--comparing an FM baseline with separately trained embeddings fused through multi-objective optimization based on Pareto optimality. We find that the FM baseline achieves strong embedding-space proximity, but our method preserves modality-specific distance structure that allows scientists to inspect which physical signals drive similarity. Our method also achieves improved fidelity in raw-data space: across six measured properties, its nearest neighbor is 8% closer to the query on average than that retrieved by kNN in the FM embedding space (spread 0.128 vs. 0.139). These results suggest that multi-objective retrieval can surface astrophysically coherent candidates that single-objective search alone may overlook.

0.28

Cross-Spectral Dense Correspondence for Multimodal Spectral Medical Imaging

cs.CV cs.AI ·2026-08-28 ·Eric L. Wisotzky, Jost Triller, Simon W. Härtl, Oliver T. Bruns, Peter Eisert +1 more
publishedbenchmark similarity 0.48
Abstract · Accepted at 2nd Data Curation & Augmentation in Medical Imaging Workshop at ECCV 2026

Precise dense correspondence is a fundamental prerequisite for multimodal spectral imaging systems that fuse disparate wavelength ranges for subsequent analysis in medical and scientific imaging. Corresponding image points are often observed with non-overlapping spectral sensitivities, leading to wavelength-dependent contrast changes, intensity inversions, and appearance shifts for which dense ground truth is difficult to obtain and conventional RGB-based training data provides only limited supervision. We address this data gap by introducing a sensor-agnostic cross-spectral modulation protocol on established correspondence benchmarks with intensity input projection, and by proposing a synthetic cross-spectral correspondence benchmark simulating physically plausible radiometric differences. Evaluation on several modern dense correspondence backbones trained with our unified cross-spectral protocol showed substantial improvements under severe spectral mismatch while maintaining performance on standard RGB benchmarks. Ablation experiments show that view-dependent channel selection and nonlinear radiometric transformations provide complementary robustness, indicating that the primary limitation of existing models is not their structural matching capacity but the mismatch between training distribution and spectral characteristics of the target image pair. Qualitative evaluations on heterogeneous medical spectral acquisition systems demonstrate the practical relevance of the proposed training data augmentation protocol as an enabler for spatially coherent spectral fusion in HSI workflows.

0.34