CS Paper Radar

Brain-to-Image Generation: Reconstructing Visual Stimuli from EEG using Generative Adversarial Networks

Harshit Goyal

cs.CV cs.AI ·submitted 2026-09-13 03:26 UTC · v1

Abstract

Reconstructing visual stimuli from electroencephalography (EEG) is difficult because scalp measurements have high temporal but limited spatial resolution, and paired EEG-image datasets remain small relative to modern generative-model training corpora. We present a reproducible single-subject baseline on THINGS-EEG2 that first tests the more defensible question of whether EEG can retrieve the viewed stimulus in a visual embedding space. A compact temporal-spatial convolutional encoder maps repetition-averaged EEG (63 by 250) to provided 512-dimensional ViT-B/32 image features. Model selection uses a concept-disjoint validation split, and final evaluation uses the official 200-image, 200-concept test gallery. Across three training seeds, the model obtains 12.83 +/- 0.58%, 39.17 +/- 1.76%, and 58.00 +/- 1.73% image recall at 1, 5, and 10 (mean +/- sample standard deviation), compared with analytical chance levels of 0.5%, 2.5%, and 5.0%. A session-balanced ablation shows that averaging more test repetitions generally improves ranking. Applying the Subject 01 model to the other nine subjects without adaptation causes a sharp performance drop, exposing subject specificity. We further report exploratory stress tests of direct conditional generators trained without external visual weights: single-subject and ten-subject variants produce noise-dominated outputs, with early validation improvements reversing after one to four epochs. Finally, we distinguish direct reconstruction from semantic rendering with a pretrained diffusion prior. The results support above-chance coarse semantic decoding under a closed-set, repetition-averaged protocol, but do not support faithful recovery of stimulus pixels.

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond Alignment: Supervising the Generator for EEG-to-Image Reconstruction

cs.CV cs.AI ·2026-11-14 ·Sangmin Jo, Wootaek Jeong, Heung-Il Suk
similarity 0.70
Abstract

Visual brain decoding translates neural activity into human-interpretable images. While recent EEG-to-image reconstruction methods rely on aligning EEG signals with pretrained visual embeddings to condition an image generator, we empirically demonstrate that stronger cross-modal alignment do not translate into comparable gains in reconstruction quality. This indicates that improving the input condition alone is insufficient; the generative process itself requires explicit supervision. Although Representation Alignment (REPA) offers a solution by guiding diffusion transformers with pretrained vision encoder features, we find that EEG-conditioned generators struggle to follow these fixed targets. To address this mismatch, we introduce generation-derived supervision (GDS): an image-conditioned teacher that translates static image-encoder features into the internal representation space of a diffusion transformer. Our EEG-conditioned student generator closely tracks these dynamic targets during denoising, discarding the teacher at inference. Evaluated on THINGS-EEG and THINGS-MEG, our approach consistently outperforms conditioning-centric baselines, achieving state-of-the-art reconstruction fidelity.

0.09

EEGFaceSem: An EEG Benchmark with Paired Generative Latents for Semantic Visual Modeling

cs.LG ·2026-11-07 ·Jun Ma, Michiel M. Spapé, Keith M. Davis, Tuukka Ruotsalo
publishedbenchmarkdatasetcode/data similarity 0.70
Abstract · 18 pages, 5 figures. Accepted at NeurIPS 2026, Evaluations and Datasets Track. Conference page: https://neurips.cc/virtual/2026/loc/paris/po…

Electroencephalography (EEG) is widely used to measure brain activity for brain-computer interface applications for its non-invasive nature and high temporal resolution. However, progress in modeling the neural responses to visual semantics has been limited by a reliance on fixed image collections that lack the fine-grained control needed to isolate fine-grained changes in visual features and their relation to perception. In this work, we present EEGFaceSem, a large-scale dataset pairing high-quality EEG with photorealistic face images synthesized from a generative model. We provide the generative latent vector for each stimulus, enabling research into the continuous mapping between a visual latent space and neural representations. Each participant's brain responses were recorded while they attended to specific semantic features (e.g., smiling, hair color) in a rapid serial visual presentation paradigm. The dataset contains EEG signals from 30 participants, totaling 67,200 time-locked EEG recordings. We present benchmarks for single-trial semantic classification under single-subject, cross-subject, and subject-adapted settings, as well as for cross-subject voting that aggregates per-stimulus predictions across participants. We further demonstrate the dataset's unique utility with two complementary applications: deriving semantic directions in the latent space for brain-supervised image editing, and conditioning a generative model on brain-decoded labels for brain-supervised image generation. Our dataset paves the way for advancing brain-computer interfaces (BCI) for semantic-level visual saliency detection and brain-supervised generative modeling. Our dataset and code are openly released at OSF (https://osf.io/2guk3/) and at HuggingFace https://huggingface.co/datasets/yefllower/EEGFaceSem.

0.48

ProtoEEG: Structured Prototype Alignment for EEG Visual Decoding

cs.CV cs.AIcs.LG ·2026-10-20 ·Kieren Yu, Chang Huang, Ziyang Liu, Kaishun Wu
similarity 0.68
Abstract

EEG visual decoding remains difficult because scalp recordings are noisy, spatially coarse, and variable across trials and subjects. Most systems align each trial with a dense image embedding, although that embedding mixes semantic content with instance-specific texture, pose, and background. This target can demand details absent from EEG and provides no explicit account of the visual factors supporting a prediction. We propose \textbf{ProtoEEG}, a prototype-mediated framework that reformulates EEG--image alignment as soft composition over a shared learnable visual codebook. A trainable EEG encoder and a frozen visual teacher produce assignment distributions over the same $K=64$ prototypes. Their weighted reconstructions form a structured representation for retrieval and image generation. Prototype contrast supplies the main cross-modal training signal, distribution alignment coordinates codebook use across modalities, an instance anchor preserves fine-grained discrimination, and diversity regularization prevents redundant prototypes. The shared assignment profile also makes the learned bottleneck directly testable through prototype retention, random selection, and identity shuffling. Across five seeds, ProtoEEG reaches 64.0\% Top-1 and 91.5\% Top-5 on THINGS-EEG2 200-way retrieval. After retraining on EEG-ImageNet, it obtains 44.8\% Top-1 and 73.8\% Top-5 in 80-way retrieval. ProtoEEG also ranks first on all seven reconstruction metrics. Removing prototype contrast lowers Top-1 by 56.9 percentage points, while shuffling prototype identities reduces it to 0.5\%. Prototype analyses further show distributed assignments and broad codebook use rather than one-prototype routing. These results support structured alignment targets as a practical alternative to increasing encoder or teacher capacity alone.

0.14

Look Closer, Align Better: Explicit Visual Attributes for Robust EEG-to-Image Decoding

cs.CV ·2026-11-06 ·Ke Xiao, Banghua Yang, Wenbin Guo, Fenqi Rong
similarity 0.67
Abstract

EEG-to-image retrieval commonly aligns neural responses with representations from pretrained vision encoders. However, these representations may underrepresent fine-grained visual details needed to distinguish semantically similar images. To address this limitation, we first investigate the EEG decodability and complementary predictive value of explicit visual attributes, identifying visual cues that capture EEG-relevant information beyond pretrained representations. Building on these findings, we propose AttrEcho, a framework that enriches pretrained visual representations with complementary, spatially aligned visual attributes. Specifically, Region-Aligned Visual Attribute Fusion (RAVF) integrates EEG-relevant local visual attributes with spatially corresponding pretrained features through a learnable signed residual gate. We further employ Feature Token Aggregation (FTA), which learns dimension-specific weights across regions to aggregate refined visual features for contrastive alignment with EEG representations. Experiments on THINGS-EEG demonstrate that AttrEcho achieves 94.8% Top-1 and 99.7% Top-5 accuracy in intra-subject 200-way retrieval, while maintaining 82.1% Top-1 accuracy as the gallery expands to 1,600 images with same-concept distractors. AttrEcho also achieves 25.3% Top-1 and 53.0% Top-5 accuracy in inter-subject retrieval. These results highlight the importance of EEG-relevant visual attributes and spatially aligned fusion for fine-grained EEG-to-image retrieval.

0.21

Attention Guided Conditional Adversarial Learning for EMG Artifact Suppression in Single Channel EEG

eess.SP ·2026-09-22 ·Haoyi Wang, Haowei Wang, Yihang Li, Wenjie Zhang, Yibo Wang
similarity 0.66
Abstract · 4 pages, 3 figures, 2 tables

Electromyographic activity can dominate scalp electroencephalography across a broad frequency range and obscure neural structure that is relevant to analysis and brain computer interfaces. This paper evaluates P2P Trans, a conditional adversarial denoiser that combines a U shaped encoder decoder, multi head self attention at the bottleneck, cross attention guided skip fusion, and a PatchGAN discriminator. The reported experiment used EEGdenoiseNet segments represented as 1024 sample sequences, synthesized paired mixtures at ten signal to noise ratios from minus 7 to 2 dB, and evaluated 50000 training examples and 5980 test examples. Against five convolutional encoder decoder baselines, P2P Trans achieved the best mean correlation coefficient of 0.8645 and the lowest mean spectral RRMSE of 0.3596. Its mean temporal RRMSE was 0.5797 and did not lead the comparison, which indicates a tradeoff between spectral suppression and pointwise waveform fidelity. The advantage was strongest under severe contamination. The results support attention guided adversarial restoration as a useful low SNR approach, while also identifying the need for subject independent evaluation, direct one dimensional modeling, and downstream task validation.

0.30

Structured Visual Target Learning For Cross-Subject eeg-to-image retrieval

cs.CV ·2026-09-29 ·Salini Yadav, Taveena Lotey, Mickaël Coustaty, Pravendra Singh, Partha Pratim Roy
similarity 0.65
Abstract

Cross-subject EEG-to-image retrieval requires a neural represen- tation trained on source subjects to remain aligned with a visual embedding space for an unseen subject. Whereas existing methods primarily focus on the EEG side, we address this problem from the perspective of the visual target. Our approach preserves the spatial information of the Perception Encoder, converts its patch grid into a compact set of learned visual views, and aggregates them for each image with a block-structured, content-dependent router. The target is learned jointly with the EEG encoder through contrastive learning with MMD regularization across source subjects. For deployment, we propose a training-free representation refinement that aligns frozen embeddings without updating either encoder. Under leave- one-subject-out evaluation on THINGS-EEG2, the structured target achieves 35.3%/65.6% Top-1/Top-5 accuracy, the best among com- pared methods. Refinement raises this to 48.1%/77.1%, an 18.5% Top-1 gain over the strongest compared method, improving all ten held-out subjects.

0.09

Replaying the Movie in Your Mind Decoding Dynamic Visual Perception and Recall from EEG

cs.CV ·2026-10-09 ·Zikai Fang, Tian-Yi Zhou, Xuan-Hao Liu, Yixi Li, Xiaoyu Xing +3 more
benchmark similarity 0.65
Abstract · ICONIPS 2026 15 pages, 5 figures

Decoding dynamic visual content from EEG has advanced rapidly for visual perception, yet it remains unclear which aspects of neural representations are preserved when people internally replay dynamic scenes. We introduce PAIR, an EEG benchmark collected from 20 participants using an interleaved, clip-wise paradigm that pairs naturalistic video viewing with immediate visual recall. This design enables a comparison between externally driven perception and internally generated recall for the same stimulus content within a fixed trial sequence. We benchmark multiple decoding models on hierarchical semantic classification at both fine-grained and coarse-grained levels, meta-feature decoding tasks covering motion, color, object number, face, and human presence, and a state-decoding task that discriminates perception from recall. Within-subject validation results show numerically similar performance for several model--task combinations, alongside higher perception accuracy in two coarse-grained comparisons after multiple-comparison correction. In contrast, perception and recall are highly separable at the state level under the present validation protocol. State accuracy remains high after alpha-band ablation, although this does not exclude ocular or other task-related contributions.

0.32

Vision Models Predict Urban Scene Appraisal with Limited Neural Alignment

cs.CV ·2026-08-31 ·Kaizhen Tan, Yuantao Deng
similarity 0.65
Abstract

Pretrained vision embeddings are widely used to model how people appraise urban scenes, and they are validated almost entirely by how well they predict human ratings. Accurate prediction shows that an embedding contains the information needed to recover the ratings. It does not show that the embedding arranges scenes as the human visual system does, which is assumed when distances or dimensions in the embedding are read as perceptual. Using openly released EEG recorded while 63 adults viewed and rated street scenes of Berlin, we compared the neural representational geometry of the scenes with the geometry of pretrained vision models that differ in training objective and size, of simple image descriptors and of the ratings themselves, each relative to a noise ceiling given by the agreement between participants. No feature space reached more than about half of the noise ceiling, which corresponds to about a fifth of the reliable variance in the neural geometry. A descriptor of oriented edge energy reached the level of most pretrained models while capturing a different part of the neural geometry, and deeper layers corresponded to later neural responses. The same embeddings predicted held-out ratings well, up to r = 0.87, yet prediction accuracy and neural correspondence were not reliably related across models, and the best predictor was among the least aligned. Predicting how a street is appraised is therefore weak evidence that a model represents the street as the brain does.

0.25