CS Paper Radar

CoupleFlow: Null-Calibrated Auditory Attention Decoding With Time-Locked Brain-Speech Coupling and Flow-Matching Likelihood Ratios

Ali Alavi, Donald S. Williamson

eess.AS ·submitted 2026-10-23 17:44 UTC · v1

Abstract

Auditory attention decoding (AAD) seeks to identify the attended talker from brief noninvasive brain recordings by comparing neural responses with competing speech streams. Recent electroencephalography (EEG) decoders report nearly perfect accuracy from one second of data, yet we show that this performance fails to generalize to unseen speech because models exploit trial leakage, eye gaze, and acoustic cues instead of speech evoked neural activity. We introduce CoupleFlow, which decodes attention through temporal coupling between neural and acoustic embeddings. Its predictions remain at chance when the brain signal contains no informative coupling, preventing shortcuts based on speech content or talker identity. A conditional flow matching head implements a likelihood ratio decision rule based on pointwise mutual information between brain and speech. This rule is Bayes optimal when the recording depends only on the attended talker and removes preferences for speech that is likely a priori. We evaluate CoupleFlow with content disjoint splits recovered from the audio and null tests that distinguish genuine temporal coupling from trial recognition. Across nine EEG and magnetoencephalography (MEG) datasets processed through a common pipeline, spanning four languages, high density EEG recordings, listeners with hearing impairment, and a room with multiple talkers, CoupleFlow outperforms linear and deep baselines while all null tests remain at chance. Finally, we derive an evidence accumulation law that predicts accuracy across window lengths from a single parameter per dataset and predicts that parameter from the strength of neural speech tracking. This law identifies recording signal to noise ratio as a limit on decoding speed and reliability and estimates the window length required to reach a target accuracy.

Closest papers

Nearest neighbours by meaning across the whole index.

A multi-scenario EEG dataset for auditory attention decoding in naturalistic multi-talker environments

cs.SD ·2026-10-07 ·Shu Peng, Rui Liu, Yufei Zhang, Wenlong You, Zhige Chen +5 more
benchmarkdatasetcode/data similarity 0.70
Abstract

Understanding how the brain selectively follows relevant speech amid competing voices is a central challenge in auditory neuroscience and a key step toward neuro-steered hearing technologies. However, most open-source Electroencephalography (EEG) datasets for Auditory Attention Decoding (AAD) use idealized single-competing-talker paradigms that oversimplify the acoustic, spatial, and semantic structure of everyday communication. To capture this ecological complexity, we introduce the SoundBubble-EEG dataset: a high-density 128-channel EEG resource comprising more than 25 hours of recordings from 30 participants. The paradigm requires listeners to selectively attend to a dynamic target speaker group, a designated "sound bubble", amid competing multi-speaker distractor bubbles across three realistic scenarios: a restaurant, a home TV viewing, and a meeting discussion. By bridging the gap between constrained laboratory protocols and real-world auditory scenes, this dataset enables investigations of multi-talker speech comprehension, neural speech tracking, and cross-scenario generalization. It also provides a benchmark for AAD algorithms under realistic acoustic and semantic variability and may support auditory neuroscience and the development of neuro-steered hearing technologies.

0.13

MOV-AAD: A Large-Scale Multimodal Dataset for Auditory Attention Decoding During Moving Conversations

eess.AS ·2026-10-03 ·Xiaomin He, Vishal Choudhari, Tristan J. Spratt, Aarya Raghavan, Richard T. Lee +1 more
top venuedataset similarity 0.69
Abstract · Xiaomin He and Vishal Choudhari contributed equally to this work

Auditory attention decoding (AAD) is often evaluated on static, simplified speech scenes that poorly match everyday listening. We introduce MOV-AAD, a large-scale dataset for studying auditory attention under moving, naturalistic conversations. MOV-AAD combines 64-channel EEG with synchronized physiological recordings, including eye tracking, respiration, galvanic skin response, heart rate, peripheral oxygen saturation, body temperature, body motion, and photoplethysmography, enabling analysis of cross-modal neural and physiological markers of attention and listening effort. This dataset uses a more ecologically valid listening paradigm with dynamically moving conversational speech sources and behavioral measures of attentional engagement. MOV-AAD supports research on robust AAD in realistic spatial dynamics, multimodal attention modeling, listening effort, and intersubject neural responses, providing a resource for benchmarking selective auditory attention in naturalistic listening.

0.29

Multiscale Gaussian-Mixture Modeling for HMM Post-Processing in Selective Auditory Attention Decoding

eess.SP ·2026-09-14 ·Tianyi Li, Simon Geirnaert, Bert De Smedt, Alexander Bertrand
similarity 0.68
Abstract · 5 pages, 3 figures, conference paper (submitted to ICASSP)

Selective auditory attention decoding (sAAD) infers from electroencephalography (EEG) which speaker a listener attends to in multi-speaker scenarios. A widely used approach reconstructs the attended speech envelope from EEG, correlates it with candidate envelopes, and selects the speaker with the highest Pearson correlation. However, correlations in each decision window have high intrinsic variance, making per-window decisions unreliable, especially for short windows. Recent work introduced a hidden Markov model (HMM) that integrates correlation evidence over time and requires state-dependent emission distributions. Since labeled attention states are typically unavailable in practice, these distributions can be inferred from unlabeled observations by fitting a Gaussian mixture model (GMM). However, when the underlying Gaussians are close, unsupervised fitting becomes less reliable with limited samples. We therefore propose a multiscale GMM whose parameters are jointly estimated across multiple window lengths. The model exploits the window-length dependence of Fisher-transformed correlation estimates, whose means are approximately stable while their variances decrease with the number of samples per window. Experiments show that the multiscale GMM estimates emission parameters more accurately than single-scale fitting, particularly at short windows. Its HMM-post-processed accuracy advantage is largest for short recordings and narrows as more data become available, motivating multiscale fitting in data-constrained settings.

0.44

Merit: Making every modality earn its contribution in multimodal attention decoding

cs.LG ·2026-10-13 ·Ali Alavi, Donald S. Williamson
benchmark similarity 0.68
Abstract

More than 1.5 billion people have hearing loss. When several people talk at once, hearing aids amplify every voice alike. Auditory attention decoding (AAD) aims to solve this by reading a listener's brain signals to find the attended talker, letting the device amplify only them. A real device will not see the brain alone; smart glasses and earbuds also track eyes, head, and scene. These behavioral signals help, as people often look at who they listen to. However, they show where a listener looks, not whom they hear. A decoder relying heavily on them fails when a listener cannot move their head or listens without looking - exactly when a brain-based device is needed. A multimodal decoder must use these signals without letting them replace the electroencephalogram (EEG). Accuracy alone cannot show whether it does. We present MERIT, a decoder and evaluation method asking: how much accuracy does each input earn? MERIT shuffles each input stream across test examples, measures the accuracy drop, and splits the result across streams using Shapley values. It also blocks two shortcuts: candidate audio is checked before training so it cannot reveal the answer, and the scoring function cannot beat chance if it ignores the brain. On a public dataset with EEG, eye tracking, head motion, and scene video, MERIT improves on the published benchmark, reaching 0.82 four-way accuracy within listeners and 0.74 on unseen listeners (chance is 0.25). The measurement reveals what accuracy hides: adding behavioral streams raises overall accuracy but almost halves the brain's share of it. On two other datasets, the same tests find a label imbalance in a widely used benchmark and flag results that accuracy alone would report as brain decoding.

0.27

A hidden Markov model for unsupervised EEG-based tracking of auditory attention to a single voice

eess.SP ·2026-10-15 ·Tianyi Li, Simon Geirnaert, Davina Van den Broek, Elien Bellon, Bert De Smedt +1 more
similarity 0.68
Abstract

Objective: Absolute auditory attention decoding (aAAD) aims to track how strongly a listener attends to a single speech source using electroencephalography (EEG). Unlike selective auditory attention decoding, aAAD lacks a competing speech reference, making reliable attention tracking challenging. Approach: We propose a fully unsupervised aAAD pipeline combining neural envelope tracking (NET) with hidden Markov model (HMM)-based temporal inference. Using EEG signals and the target speech envelope, the framework estimates a continuous attention trajectory without explicit attention-state labels. We evaluate the approach using EEG recordings from 30 adults, investigate the impact of analysis window length, and compare HMM post-processing with moving-average (MA) smoothing and alternative EEG features. Main results: With 1-s analysis windows, non-causal HMM post-processing improves the area under the receiver operating characteristic curve (AUC) for all 30 participants, increasing mean AUC from 0.547 for raw NET to 0.832. A 5-s window achieves the highest mean AUC of 0.858. Compared with non-causal MA smoothing, the HMM achieves higher AUC across the examined smoothing scales. Individual switch detection times are comparable, while the HMM generally achieves lower group-level detection times. NET also outperforms alpha power and spectral entropy as an observation feature in the unsupervised HMM framework. Significance: The proposed framework provides a label-free approach for estimating temporal fluctuations in auditory attention, demonstrating the utility of probabilistic temporal inference for continuous attention tracking in single-speaker listening scenarios.

0.11

NEUROTOKEN: Joint Source and Directional AAD with Envelope Decoding via Conditional Flow Matching

cs.LG ·2026-09-30 ·Ali Alavi, Donald S. Williamson
position similarity 0.68
Abstract

Identifying which speaker a listener is attending to in a noisy room -- the cocktail-party problem -- is the missing ingredient for next-generation hearing aids and brain-computer interfaces: it tells the device whose voice to amplify. Auditory attention decoding (AAD) reads this answer from EEG, but the literature splits into disconnected pieces: directional-AAD classifies side but does not map side to stream; regression-based source-AAD ranks candidate streams by a single Pearson correlation that is intrinsically noisy at the 1-5 s windows real devices need; and envelope reconstruction has no native AAD rule. We argue the right object is not any single statistic but the conditional likelihood of the attended envelope given EEG, and we make this practical with NEUROTOKEN: a single network whose three heads share one EEG front-end, with a conditional flow-matching head (ATTUNEFLOW) that scores candidates by an integrated velocity-residual likelihood ratio. Two inference-time ensembles -- QUADTRACK (four complementary statistics) and ENV-FLOW (z-normalised QUADTRACK+ATTUNEFLOW) -- absorb per-statistic failure modes for free. On KU Leuven, DTU, and NJU at 5 s, ATTUNEFLOW lifts per-segment source-AAD by 9%-16% over the strongest non-generative baseline and shrinks across-subject variance by ~3x; trial-level fusion exceeds 93% on two of three datasets. In parallel reproductions we show that canonical 95-97% direction-AAD numbers collapse by 17%-45% under a strict trial-disjoint protocol, clarifying both the true ceiling and why a likelihood-based formulation is needed.

0.46

Cross-subject decoding of imagined speech from scalp EEG: a leakage-controlled evaluation on three public corpora

eess.SP cs.LGq-bio.NC ·2026-10-14 ·Blessing Isoyiza Adeika, Onyema Osuagwu
similarity 0.68
Abstract · 23 pages, 5 figures, 10 tables plus supplementary tables. To be submitted to Journal of Neural Engineering

Objective. A decoder of imagined speech that works for a new user without calibration would make a non-invasive speech interface practical. We tested whether imagined-speech content recorded with scalp electroencephalography (EEG) transfers across people, and examined why accuracy within a single person remains low. Approach. We analyzed three public corpora, ds003626, KaraOne and Track 3 of the 2020 International Brain-Computer Interface Competition, under a protocol that enforces split integrity by assertion and reports exact binomial thresholds, group tests and bootstrap confidence intervals. We trained decoders on each participant and tested them on every other, compared representational geometry across participants without a classifier, applied subject-level alignment, and tested five explanations for the within-subject ceiling. Main results. Across 482 ordered pairs of participants, no pair survived correction for multiple comparisons; mean pair-transfer accuracy lay within 0.006 of chance on every corpus, while every within-subject reference lay above it. Imagery against pre-stimulus rest generalized at about 0.61 after artifact and elapsed-time controls, but its decoder labelled the interval preceding imagery as imagery as often as imagery itself, so the contrast reflects a post-stimulus state, not imagery. Within subject, only the competition corpus decoded above threshold for every participant (0.3147; chance 0.200); syllable count explained little of this, but its frequency profile failed an artifact audit. Significance. In these corpora, imagined-speech content did not transfer between people through any representation we examined. Trial count per class was the only variable that ordered the corpora, which favors larger single-condition datasets over further model development.

0.34

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

eess.AS cs.HCcs.LGcs.SD+1 ·2026-09-25 ·K M Naimul Hassan, Ali Alavi, Donald S. Williamson
benchmarkcode/datahot direction similarity 0.66
Abstract · 11 pages, 8 figures. Submitted to IEEE Transactions on Audio, Speech, and Language Processing

Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .

0.73