CS Paper Radar

EEGFaceSem: An EEG Benchmark with Paired Generative Latents for Semantic Visual Modeling

Jun Ma, Michiel M. Spapé, Keith M. Davis, Tuukka Ruotsalo

cs.LG ·submitted 2026-11-07 15:07 UTC · v1
publishedbenchmarkdatasetcode/data

Abstract

Electroencephalography (EEG) is widely used to measure brain activity for brain-computer interface applications for its non-invasive nature and high temporal resolution. However, progress in modeling the neural responses to visual semantics has been limited by a reliance on fixed image collections that lack the fine-grained control needed to isolate fine-grained changes in visual features and their relation to perception. In this work, we present EEGFaceSem, a large-scale dataset pairing high-quality EEG with photorealistic face images synthesized from a generative model. We provide the generative latent vector for each stimulus, enabling research into the continuous mapping between a visual latent space and neural representations. Each participant's brain responses were recorded while they attended to specific semantic features (e.g., smiling, hair color) in a rapid serial visual presentation paradigm. The dataset contains EEG signals from 30 participants, totaling 67,200 time-locked EEG recordings. We present benchmarks for single-trial semantic classification under single-subject, cross-subject, and subject-adapted settings, as well as for cross-subject voting that aggregates per-stimulus predictions across participants. We further demonstrate the dataset's unique utility with two complementary applications: deriving semantic directions in the latent space for brain-supervised image editing, and conditioning a generative model on brain-decoded labels for brain-supervised image generation. Our dataset paves the way for advancing brain-computer interfaces (BCI) for semantic-level visual saliency detection and brain-supervised generative modeling. Our dataset and code are openly released at OSF (https://osf.io/2guk3/) and at HuggingFace https://huggingface.co/datasets/yefllower/EEGFaceSem.

Comment
18 pages, 5 figures. Accepted at NeurIPS 2026, Evaluations and Datasets Track. Conference page: https://neurips.cc/virtual/2026/loc/paris/poster/138993 ; Project page: https://EEGFaceSem.github.io/

Closest papers

Nearest neighbours by meaning across the whole index.

Brain-to-Image Generation: Reconstructing Visual Stimuli from EEG using Generative Adversarial Networks

cs.CV cs.AI ·2026-09-13 ·Harshit Goyal
similarity 0.70
Abstract

Reconstructing visual stimuli from electroencephalography (EEG) is difficult because scalp measurements have high temporal but limited spatial resolution, and paired EEG-image datasets remain small relative to modern generative-model training corpora. We present a reproducible single-subject baseline on THINGS-EEG2 that first tests the more defensible question of whether EEG can retrieve the viewed stimulus in a visual embedding space. A compact temporal-spatial convolutional encoder maps repetition-averaged EEG (63 by 250) to provided 512-dimensional ViT-B/32 image features. Model selection uses a concept-disjoint validation split, and final evaluation uses the official 200-image, 200-concept test gallery. Across three training seeds, the model obtains 12.83 +/- 0.58%, 39.17 +/- 1.76%, and 58.00 +/- 1.73% image recall at 1, 5, and 10 (mean +/- sample standard deviation), compared with analytical chance levels of 0.5%, 2.5%, and 5.0%. A session-balanced ablation shows that averaging more test repetitions generally improves ranking. Applying the Subject 01 model to the other nine subjects without adaptation causes a sharp performance drop, exposing subject specificity. We further report exploratory stress tests of direct conditional generators trained without external visual weights: single-subject and ten-subject variants produce noise-dominated outputs, with early validation improvements reversing after one to four epochs. Finally, we distinguish direct reconstruction from semantic rendering with a pretrained diffusion prior. The results support above-chance coarse semantic decoding under a closed-set, repetition-averaged protocol, but do not support faithful recovery of stimulus pixels.

0.27

Look Closer, Align Better: Explicit Visual Attributes for Robust EEG-to-Image Decoding

cs.CV ·2026-11-06 ·Ke Xiao, Banghua Yang, Wenbin Guo, Fenqi Rong
similarity 0.69
Abstract

EEG-to-image retrieval commonly aligns neural responses with representations from pretrained vision encoders. However, these representations may underrepresent fine-grained visual details needed to distinguish semantically similar images. To address this limitation, we first investigate the EEG decodability and complementary predictive value of explicit visual attributes, identifying visual cues that capture EEG-relevant information beyond pretrained representations. Building on these findings, we propose AttrEcho, a framework that enriches pretrained visual representations with complementary, spatially aligned visual attributes. Specifically, Region-Aligned Visual Attribute Fusion (RAVF) integrates EEG-relevant local visual attributes with spatially corresponding pretrained features through a learnable signed residual gate. We further employ Feature Token Aggregation (FTA), which learns dimension-specific weights across regions to aggregate refined visual features for contrastive alignment with EEG representations. Experiments on THINGS-EEG demonstrate that AttrEcho achieves 94.8% Top-1 and 99.7% Top-5 accuracy in intra-subject 200-way retrieval, while maintaining 82.1% Top-1 accuracy as the gallery expands to 1,600 images with same-concept distractors. AttrEcho also achieves 25.3% Top-1 and 53.0% Top-5 accuracy in inter-subject retrieval. These results highlight the importance of EEG-relevant visual attributes and spatially aligned fusion for fine-grained EEG-to-image retrieval.

0.21

EEG2MOTION: Towards Open-Vocabulary Human Motion Synthesis from Non-invasive Brain Signals

eess.SP ·2026-08-14 ·Yulong Peng, Yijian Pan, Yuqi Yang, Nenggan Zheng, Weidong Chen +2 more
datasetcode/data similarity 0.68
Abstract

Human motion is governed by a hierarchical motor system where the brain provides high-level intentions and lower-level structures coordinate detailed dynamics. Existing brain-computer interfaces (BCIs) typically oversimplify this into constrained classification or low-dimensional control, failing to capture the richness of natural movement. Bridging this gap to achieve open-vocabulary, full-body motion synthesis remains challenging due to the substantial cross-modal divergence between sparse neural signals and high-dimensional kinematics, as well as the lack of large-scale paired EEG-motion datasets. To address this, we introduce EEG2MOTION, the first EEG-motion-text dataset for human motion synthesis, comprising nearly 20,000 paired samples across thousands of motions. Using this dataset, we first demonstrate via multimodal contrastive learning that non-invasive EEG embeddings can be effectively aligned with text, video, and motion representations to decode high-level semantics. We then propose EEG-conditioned Masked Motion Model (EMMM), a generative framework that unites an EEG encoder with a motion decoder to synthesize continuous, full-body human motions directly from brain activity. Experimental results show that EMMM generates coherent and realistic motion sequences from non-invasive brain signals. To the best of our knowledge, this is the first work to generate diverse full-body human motions from non-invasive brain signals, opening a new direction toward generative and open-vocabulary motor BCIs. See our project page: https://yulom.github.io/EEG2MOTIONdemopage/.

0.20

Brain2Face: A Study on the Relation of EEG Signals and Spatio-temporal Head Avatars

cs.CV cs.HC ·2026-10-12 ·Camilo Andrés Martínez Martínez, Olaf Dünkel, Kartik Teotia, Maurice Rekrut, Christian Theobalt +1 more
benchmarkcode/data similarity 0.68
Abstract · Project page: https://vcai.mpi-inf.mpg.de/projects/Brain2Face

Driving digital head avatars is classically done by hand-crafted artist design or by image-based facial performance capture. In this work, we ask whether spatio-temporal tracking of the humans's head geometry and appearance can be performed solely from brain signals, i.e., Electroencephalography (EEG). Since prior work mostly focused on activity and emotion recognition, this task is relatively unexplored without even having publicly available high-quality datasets and benchmarks. Thus, we contribute a dataset featuring paired EEG and human head geometry as well as appearance annotations. We synchronize 64-channel EEG with per-frame expression latent codes of a personalized Gaussian head avatar. Second, we introduce a baseline to the task of expression recovery of a head avatar from EEG signals. To this end, we propose a dedicated EEG-to-expression model that takes a temporal sequence of EEG signals as input and regresses the latent expression parameters of the 3D Gaussian head avatar. Lastly, we validate our capture and algorithmic design choices through a benchmark protocol that we introduce to this task. We believe this work paves the way towards better understanding of the relation between EEG signals and facial surface as well as appearance changes.

0.21

Multimodal LLMs Can Learn to Read Brain Signals: A Vision--Language Model for Unified Multi-Task EEG Decoding

cs.LG cs.AIcs.CV ·2026-10-07 ·Parastoo Azizeddin, Omid Sharafi, Maryam M. Shanechi
highly cited similarity 0.67
Abstract

Learning EEG representations that generalize across cognitive tasks, subjects, and recording conditions remains a key challenge in electroencephalography (EEG) decoding. Recent advances in foundation models have improved EEG decoding performance, yet a fundamental open question remains: how to effectively interface neural signals with these models to enable multi-task learning across datasets. To investigate this question, we introduce BraVista, a visual-language framework that encodes multichannel EEG signals as structured images and enables multi-task learning through instruction-conditioned vision-language models (VLMs). Our approach relies on continued post-training of a general-domain VLM, leveraging its visual and linguistic priors to adapt to neural signals without a separate large-scale EEG-specific pretraining stage. We evaluate BraVista on four datasets spanning sleep staging, emotion recognition, cognitive workload classification, and abnormal EEG detection, showing strong performance across these tasks. Further analyses show that the choice of EEG-to-image representation is critical to performance. Moreover, through controlled perturbations of the EEG signal, we observe a gradual performance degradation under increasing noise, suggesting that the model relies on EEG-relevant information rather than superficial visual patterns. Together, these findings establish structured visual representations as an effective and scalable interface between neural signals and general-domain foundation models for unified multi-task EEG decoding.

0.33

From Neural Tokens to Language: Dual-View EEG Semantic Bridging for Brain-to-Text Generation

cs.CV ·2026-11-18 ·YaoYang Liu, Jun Dai, ZhuoShi Tang, Weijun Sun, Xiaozhao Fang +1 more
similarity 0.67
Abstract · 30 pages, 15 pages of supplementary material

Visually evoked EEG-to-text generation aims to produce natural-language descriptions from neural responses elicited by visual stimuli. Existing approaches typically compress EEG signals into a single global representation and directly project it into the hidden space of a large language model (LLM), which may limit the preservation of complementary visual semantics under noisy and subject-dependent EEG signals. We propose a dual-view semantic token bridging framework that exploits raw images and sketch images as complementary visual supervision to learn raw-oriented and sketch-oriented EEG semantic token streams. To interface these structured neural representations with an LLM, we introduce a branch-specific semantic query bridge that aggregates each token stream into compact semantic slots and converts them into LLM-compatible soft prompts. A progressive adaptation strategy further transitions the learned visual-semantic interface from image-supported training to image-free EEG-conditioned generation. Experiments demonstrate strong subject-wise EEG classification performance and competitive EEG-to-text generation across multiple LLM backbones. Ablation, branch-level representation, and EEG-label disentanglement analyses further show that the two EEG streams learn complementary view-oriented representations and that EEG-derived prompts provide sample-specific information beyond category labels.

0.28

OneEEG: A Unified Signal-Visual Multimodal Language Model with Task-Aware Mixture of Projectors for Multi-Task EEG Understanding

cs.LG ·2026-10-26 ·Jiacheng Zhang, Xueyi Zhang, Xin Lu, Siqi Cai, Haizhou Li
benchmark similarity 0.65
Abstract

Electroencephalography (EEG) is widely used in clinical assessment, brain-computer interfaces, and cognitive-state analysis, yet EEG modeling remains fragmented across task paradigms. Recent multimodal large language models (MLLMs) provide a promising interface for flexible multimodal reasoning, but their ability to understand EEG inputs remains limited. In this work, we propose OneEEG, a unified EEG-MLLM framework for multi-task EEG understanding. By training on our curated OneEEG-Instruct, a multi-task EEG instruction-tuning dataset, OneEEG jointly models numerical EEG representations and visual EEG representations, and introduces a task-aware Mixture of Projectors (MoP) to align heterogeneous EEG features across different task families with the LLM embedding space. We further establish OneEEG-Bench, a benchmark for evaluating current MLLMs and our method on multi-task EEG understanding. Experiments show that existing MLLMs have limited ability to interpret EEG inputs, while OneEEG achieves stronger performance by integrating numerical and visual EEG representations with task-aware EEG-to-LLM projection. Extensive ablations and analyses further reveal the effects of visual EEG renderings, MoP variants, router behavior, and feature dispersion across EEG tasks.

0.18

Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI

cs.LG cs.AIq-bio.NC ·2026-10-29 ·Serwan Jassim, Victoria Bosch, Daniel Anthes, Elia Bruni, Tim Kietzmann
similarity 0.65
Abstract

Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.

0.12