CS Paper Radar

Can a Neural Encoding Model Replicate an fMRI Visualization Study?

Erfan Nasirzadeh Orang, Zack While

cs.HC cs.CV ·submitted 2026-09-14 14:55 UTC · v1
code/data

Abstract

Most knowledge of graphical perception comes from behavioral studies. Understanding from a neural perspective is much more limited due in part to neuroimaging studies' expensiveness and difficulty to conduct. In this paper, we evaluate whether Meta's Tribe V2 neural encoding model can recover neural contrasts from a visualization fMRI study. Specifically, we evaluate Tribe V2 through a conceptual replication of the visualization-viewing component of a prior comparison of Bubble charts and three-dimensional Surface charts in color and grayscale. We generate TRIBE-predicted cortical responses for the original stimuli and compare the resulting contrasts with those reported in the human study. The model reproduced the direction of 11 of 14 reported cortical effects, with agreement concentrated in visual-processing regions. This agreement characterizes the model's alignment with the prior human-generated fMRI results rather than independently confirming them. We discuss the limitations encountered when working with this model for in-silico replication and hope to encourage future work exploring this new avenue for neuroimaging studies in visualization. Supplemental materials are available at https://osf.io/8a96x/.

Comment
5 pages, 1 figure, accepted to the VISxVISION Workshop at IEEE VIS '26

Closest papers

Nearest neighbours by meaning across the whole index.

An In-Silico Audit of Topographic Organization in a Multimodal Brain-Response Model

cs.AI q-bio.NC ·2026-11-13 ·Jakub Strawa
code/data similarity 0.65
Abstract · 12 pages, 5 figures, 4 tables. Code and stimuli: https://github.com/jakubstrawa1/audit-multimodal-brain-response

TRIBE v2 (d'Ascoli et al., 2026) predicts fMRI responses across the cortical surface from video, audio and text. Its authors validated it with category-selective localizers (faces, places, bodies, words) and language contrasts, but not with the classic topographic organization of sensory cortex: retinotopy, tonotopy, and motion selectivity in area MT+. There is a clear architectural reason to expect retinotopy to fail: TRIBE's video features are averaged over space before the response head, which should discard visual-field position. I present mapping stimuli, a fitting pipeline adapted from standard fMRI mapping analyses, and a stand-in model with known ground truth that serves as a positive and negative control, and apply them to TRIBE v2's predictions, comparing the fitted maps with human atlases using spin tests. TRIBE v2 shows a sharp split. There is no contralateral hemifield response and no polar-angle map, robustly across stimulus controls. There is no tonotopic gradient in primary auditory cortex: best frequencies span an interquartile range of only 0.14 octaves across core and belt, and the null persists with loudness-equalized tones. An eccentricity-like gradient appears only when the ring geometry changes, consistent with global image statistics rather than a positional code. In contrast, motion selectivity is strong and precisely localized: 22% of the 1% most motion-selective vertices fall in MT+, which covers 0.42% of cortex. In-silico experiments that depend on feature identity appear viable with TRIBE v2, while those that depend on visual-field or frequency position are not supported by the current model.

0.33

Prototypes of the Mind: A Unified Framework for Probing the Visual Brain

cs.AI q-bio.NC ·2026-11-11 ·Shi Chen, Seoyoung Ahn, Doris Tsao
published similarity 0.58
Abstract · NeurIPS 2026

Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective---such as neural encoding, stimulus decoding, or post hoc interpretation---making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives. Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout. This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel's selectivity. Using this interface, we identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.

0.27

Analysis choices decide whether untrained networks beat trained ones at V1: a multiverse analysis of model-brain RSA

cs.LG q-bio.NC ·2026-11-11 ·Nils Leutenegger
code/data similarity 0.57
Abstract · 31 pages, 6 figures; code and data pipeline: https://github.com/nilsleut/ridge-readout

Untrained convolutional networks have been reported to rival trained networks as models of visual cortex, and we had reported an untrained-network advantage over backpropagation at V1 in THINGS fMRI, together with a dependence on evaluation resolution. Under a pre-specified ridge readout, the untrained network scored below the trained one at every pooling grid (median r difference -0.043 at g = 4, 0 of 5 seeds positive), and also in MSE skill and per-fold r. In exploratory analyses, across 36 combinations of unit normalisation, distance measure and voxel normalisation (28 distinct analyses), the sign of the V1 contrast depended on the analysis (intervals above 0 in 5 and below 0 in 9 distinct analyses), and in our networks the resolution dependence of the original convention appeared in only one cell. Raw correlation-distance representational similarity analysis (RSA) covaried with the alignment of model and voxel patterns with the unit-mean profile (Spearman 0.97 across our networks, 0.79 across six ImageNet architectures); the advantage was absent from the component orthogonal to the mean profile, and the parallel component alone did not resolve it. On the unweighted angular component of centred patterns, trained networks were ahead at V1 (our network and 5 of 6 ImageNet models) and at LOC (all seven); the norm-weighted angular term was not resolved. A luminance-and-contrast bank scored above every network under ridge (point estimates), and adding network features under a shared penalty did not improve prediction. Intervals are stimulus-bootstrap intervals for three subjects, and 6 pre-stated predictions were not met. We recommend reporting sensitivity to centring over the evaluated stimuli on both sides, reporting components separately, and including a small multiverse and simple baselines.

0.36

AI-Driven Neural Surrogates for In Silico Design of Cognitive-Affective Neuromodulation Targets

cs.AI q-bio.NCeess.SY ·2026-09-23 ·Marco Rothermel, Madleen Stenger, Soroush Daftarian, Svenja Jule Francke, Bita Shariatpanahi +5 more
similarity 0.57
Abstract

In neuropsychiatry, the primary goal is often not only to decode brain activity but to change it, for example to lessen a negative affective bias or an overly salient memory. Motivated by control theory, we develop an AI-driven neural-surrogate framework that proposes candidate representational changes and tests their predicted perceptual effects from snapshots of stimulus-evoked fMRI activity, without physical stimulation. The framework combines fMRI decoding, deep generative modeling, and constrained latent-space steering. Valence and memorability are used only as worked examples. Using more than 36,000 image-fMRI observations from four deeply sampled Natural Scenes Dataset participants, subject-specific models recovered coarse generative structure from visually responsive cortex (two-way identification, 0.79-0.88; chance, 0.5). Graded perturbations were reconstructed as images and evaluated with automated scorers and human ratings from 7,200 trials by 18 participants. In the primary VDVAE model, valence shifted from -0.61 to +1.03 SD and memorability from -1.34 to +1.45 SD; a later Versatile Diffusion refinement reduced or altered these effects. Across five perturbation levels, human valence ratings moved in the predicted direction under the linear time-correction model (mean slope, 0.038 SD per unit of alpha; 95 percent CI, 0.003-0.074; positive in 16 of 18 participants). Perceived memorability did not change reliably. Baseline agreement with the automated assessor was suggestive for valence (r = 0.30) and weak for memorability (r = 0.10). Extreme perturbations drifted from the original stimulus, so intended change must be weighed against loss of fidelity. These findings provide a falsifiable upstream method for designing and behaviorally testing candidate representational targets for future neuromodulation in psychiatry, while marking the limits of the present static approximation.

0.32

CoFlu-Brain: An fMRI Connectivity Foundation Model with Cofluctuation-Based Pretraining

cs.NE ·2026-10-11 ·Ziteng Sui, Junfeng Xia, Wenhao Ye, Mo Wang, Quanying Liu
code/data similarity 0.54
Abstract

Functional magnetic resonance imaging (fMRI) commonly characterizes large-scale brain organization through functional connectivity (FC). However, an observed FC matrix is not a fixed brain graph but a statistical estimate determined by the finite BOLD frames used to compute it. Conventional FC-based approaches typically represent each scan with a single Pearson-correlation matrix, leaving frame-resolved variations in regional interactions unexploited. To address this limitation, we introduce CoFlu-Brain, which organizes framewise cofluctuation observations into multiple retained and dropped views, each summarized as an FC matrix. The model learns connectivity structure reproducible across retained views and predicts the representations of the corresponding dropped views, while adopting SIGReg from LeJEPA to prevent representation collapse. We pretrained CoFlu-Brain on 13,444 subjects from eight dataset groups and evaluated its representations via frozen-encoder linear probing on seven classification and two age-prediction tasks across seven datasets. CoFlu-Brain showed strong transfer across demographic, developmental, and clinical prediction tasks, outperforming the evaluated released models on most tasks. Random frame partitioning improved aggregate transfer, while the complete objective achieved the strongest overall performance. Code is available.

0.20

Seeing Without Seeing: A Methodological Review of Visual Mental Imagery Reconstruction from Brain Activity

cs.HC ·2026-11-10 ·Shiyi Zhang, Dong Liang, Yihang Zhou
benchmark similarity 0.54
Abstract

Reconstructing visual mental imagery from brain activity differs from reconstructing perceived images. Imagery is generated top-down, its signal is weak, and the imagined content is never observed, so decoders are trained and scored against a proxy such as a cue or a seen exemplar. This criterion-guided methodological review of mainly non-invasive studies (fMRI, EEG, MEG) is narrative rather than systematic. We organise the field into a two-stage pipeline, two design axes and four method families, and formalise target-free evaluation, in which the proxy gap is a systematic bias rather than noise. A two-dimensional admissibility criterion crosses trial-level separability from working-memory maintenance with the memory source of the imagined content. Among the studies reviewed, every image-scored reconstruction concerns a recalled exemplar, none builds a scorable target for concept-generated content, and all come from fMRI, consistent with missing image-level targets in electrophysiological data. Published values for five decoders, compared under a common human forced-choice framing, show seen-to-imagined drops of 5.9 to 26.1 percentage points and a change in method ordering. We treat this as a diagnostic signal, not a controlled benchmark result, and specify a within-subject test for an altered neural code. Beyond this synthesis, we make two proposals. First, because frozen generative priors can produce photorealistic images weakly tied to neural signals, we combine existing null-signal controls into a normalised neural dependence with a within-category null, an unvalidated diagnostic quantity. Second, we propose a candidate ten-item reporting checklist with illustrative pass criteria. In an exploratory application to ten studies, none reports all items in a fully checkable form; this describes reporting practice, not study quality.

0.38

Alignment of visual brain encoding models with individual perceptual similarity judgments

cs.CV q-bio.NC ·2026-10-15 ·Daniel Chong, Haomiao Chen, Keith W Jamison, Raja Marjieh, Ilia Sucholutsky +2 more
similarity 0.54
Abstract

Encoding models that predict brain activation responses to images have traditionally been trained to maximize accuracy when predicting observed brain data. However, their internal representations may not reflect individual human perception. In this work, we investigate whether we can fine-tune brain encoding models to better align their representations with individuals' similarity judgments. Using similarity judgments of 100 images from 8 participants in the Natural Scenes Dataset (NSD), we adapt a generic human similarity judgment model (DreamSim) separately for each participant to obtain personalized perceptual similarity models. These models are then used to obtain subject-specific similarity structure across images in their encoding model training sets. We fine-tune each participant's baseline encoder in two conditions, aligning its representations with either generic or personalized similarity judgments, with the same initialization and objective. We find that baseline brain encoders reflect within-subject perceptual similarity more strongly than measured fMRI responses and preserve intersubject correlations present in measured fMRI, but fail to reflect those of human perceptual similarity judgments. Fine-tuning with generic, population-level perceptual similarity information results in better alignment with human judgments than baseline encoders, but these models still fail to capture intersubject correlations of human judgments. However, fine-tuning with personalized perceptual similarity information results in models that are the most aligned with human perceptual similarity judgments and also reflect intersubject correlations of human judgments. Our framework provides an approach for bootstrapping human similarity judgment data into more individualized, human-aligned brain encoding models of vision.

0.19

Many Brains, One Geometry: A Shared Visual-Semantic Space for Cross-Dataset fMRI Decoding

cs.HC q-bio.NCq-bio.QM ·2026-10-07 ·Moein Khajehnejad, Michelangelo Tronti, Forough Habibollahi, Tommaso Boccato, Matteo Ferrante +1 more
similarity 0.54
Abstract · 21 pages, 5 figures, 1 table, 3 supplementary figures, 2 supplementary tables

Visual decoding from fMRI is typically siloed by participant and experiment, obscuring whether heterogeneous neural measurements can be organized within a common computational geometry. Here we introduce BRAID-fMRI (Brain Representation Alignment across Individuals and Datasets), a shared CLIP-supervised decoding framework. BRAID-fMRI uses a single ROI-wise Transformer with optional participant conditioning across eight visual-fMRI datasets comprising 93 dataset-specific participant entries, 430,007 single-trial responses and 162,839 unique stimuli. Regional brain activity is aligned with 512-dimensional CLIP ViT-B/32 representations using a multi-positive contrastive objective that treats repeated stimuli across participants and datasets as positives. BRAID-fMRI supports retrieval across seven evaluation datasets. On eight matched participant entries, it achieves 35.0 +/- 11.1% Top-10 accuracy, exceeding the observed mean accuracy of the two evaluated baselines - the MindEye-style pooled-CLIP decoder (27.1 +/- 5.1%) and ridge regression (21.1 +/- 9.1%) - and attaining the highest observed accuracy for seven of eight entries. In separately trained participant-agnostic models, expanding the source pool increased target-dataset holdout accuracy by up to 92.1% relative to the initial source-training condition. The learned space preserves graded semantic structure, while ablations and saliency highlight ventral and early visual cortex and category-specific motion and attentional systems. These results support scalable cross-dataset decoding into a common CLIP-aligned space, with model sensitivity concentrated in ventral and early visual inputs.

0.32