CS Paper Radar

Beyond the Raw Waveform: Fusing Visual Representations of EDA for Stress Detection

Stefanos Gkikas, Thomas Kassiotis, Yang Guo, Guangliang Li, Eric Nichols, Houshyar Asadi, Nikolaos Smyrnis, Giorgos Giannakakis

eess.SP cs.LGcs.SDeess.AS ·submitted 2026-08-10 03:44 UTC · v1
top venue

Abstract

Electrodermal activity (EDA) is widely used in automatic stress detection, yet most pipelines treat it only as a raw one-dimensional waveform. This study examines whether complementary visual representations of EDA provide useful information for stress classification and whether their fusion im- proves recognition performance. Six image-based representations are derived from each EDA recording: an unwrapped short-time Fourier transform (STFT) phase spectrogram, an instantaneous-frequency map computed from that phase, a power spectral density (PSD) spectrogram, a continuous wavelet transform scalogram, a recurrence plot, and a rendered waveform trace. The selected representations are stacked as channels of a single multichannel input, together with the raw waveform, and processed by a shared asymmetric-attention architecture. Experiments on a 58-subject stress dataset show that representation fusion improves over the raw waveform. The best configuration, which combines five representations while excluding the unwrapped phase spectrogram, reaches 70.97% test accuracy, compared with 67.36% for the raw waveform. The single PSD spectrogram achieves 69.44%, remaining close to the best-fused configuration at a lower computational cost. The results show that alternative visual forms of the same EDA signal can provide useful inductive biases for stress detection, and that a compact selection of complementary representations can be more effective than the raw waveform alone.

Comment
The paper has been accepted at: IEEE | 2026 9th International Conference on Pattern Recognition and Artificial Intelligence (PRAI 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

MUPA${}^2$S: Multimodal Unified Perception with Asymmetric Attention for Stress Assessment

cs.CV cs.AI ·2026-11-01 ·Stefanos Gkikas, Thomas Kassiotis, Yu Fang, Christian Arzate Cruz, Miao Cheng +6 more
similarity 0.64
Abstract

Stress is a leading contributor to long-term physical and mental health decline. The automatic assessment of stress is a promising approach for continuous monitoring and mitigating human biases. However, the development of automated systems remains challenging due to the heterogeneity of physiological and behavioral signals. This paper presents MUPA${}^2$S, a multimodal framework for automatic stress assessment that combines facial video and ECG signals within a unified perception architecture. Both modalities are mapped into a shared token representation without modality-specific components, and processed through asymmetric attention comprising segment-local cross-attention and global self-attention across segment states. Experiments on a newly collected 58-subject stress dataset, evaluated under a stratified subject-level protocol, show that facial video achieves 69.44% test accuracy, while ECG-only performance remains close to chance under the proposed representation. Unified Video-ECG fusion achieves a test accuracy of 69.58% at approximately half the number of parameters of feature-level fusion, with negligible additional computational cost. Feature-level fusion does not exceed the video-only baseline, while unified input-level integration provides a parameter-efficient mechanism for jointly processing the two modalities.

0.14

FUSE: Frame-Unified Stress Estimation from Facial Video

cs.CV cs.AI ·2026-08-11 ·Stefanos Gkikas, Thomas Kassiotis, Yang Guo, Guangliang Li, Giorgos Giannakakis
similarity 0.61
Abstract

Automatic stress detection from facial video offers a practical path to non-intrusive affect monitoring, yet existing video-based approaches commonly decompose full recordings into short temporal windows before classification. This design introduces additional choices regarding window length, overlap, and aggregation, while limiting direct analysis of temporal information across the entire recording. In this study, we present FUSE (Frame-Unified Stress Estimation), a facial-video stress detection framework that processes complete recordings as a single input without temporal windowing or external segmentation. The name reflects the defining operation of the method: rather than dividing a recording into short clips, all frames are fused into one unified two-dimensional representation from which the stress state is estimated. This unification is realized by folding the temporal dimension into the channel dimension of the spatial representation, and the resulting high-dimensional input is processed using a unified asymmetric-attention architecture. At a temporal stride of t = 1, FUSE retains the full 120-second recording as one input, corresponding to 3,600 frames at 30 fps. Experiments on a 58-subject stress dataset using a stratified subject-level protocol evaluate seven temporal-stride configurations, ranging from full-frame input to sparse subsampling. FUSE achieves the highest test accuracy of 69.44% at t = 15, while the full-frame configuration remains competitive at 69.03%. Across the stride range, computational cost varies from 12.48 to 348.78 GFLOPs, showing the trade-off between temporal density and efficiency. These results demonstrate that temporal windowing is not required for effective facial-video stress detection in this setting, and that complete-recording inference can be achieved within a single unified architecture.

0.34

Electrodermal Activity (EDA) for Stress Detection: A Comprehensive Survey and Benchmark

eess.SP ·2026-09-18 ·Gang Liu, Zixiong Ning
surveybenchmark similarity 0.61
Abstract · 19 pages, 5 figures, 7 tables

Electrodermal activity (EDA) is widely used for pervasive stress monitoring because of its sensitivity to sympathetic arousal and its computational efficiency. However, research remains fragmented: studies concentrate on a small subset of available datasets, preprocessing and evaluation protocols are inconsistent, and widely adopted methodological conventions remain unvalidated. Existing reviews summarize these practices without evaluating them under a common protocol. To bridge these gaps, this paper presents a comprehensive survey and benchmark for EDA-based stress detection. We review the technical pipeline from preprocessing to modeling and compile 26 public datasets and 22 representative unimodal studies to inform future dataset selection and methodological design. Under a subject-independent protocol spanning five diverse datasets, we benchmark decomposition, filtering, and channel-input choices alongside a broad range of machine learning (ML) and deep learning (DL) models in both in-domain and cross-domain transfer settings. Our results provide the first multi-dataset validation of cvxEDA, the default decomposition method, show that feature-based ML outperforms end-to-end (E2E) DL, and identify ecological validity as a key factor in cross-domain transfer. We conclude with directions toward robust, data-efficient, and trustworthy deployment.

0.22

From ECG Signals to Representative-Morphology Heatmaps for Biometric Recognition

cs.CV eess.SP ·2026-09-23 ·Athanasios Angelakis, Marta Gomez-Barrero
similarity 0.57
Abstract · 31 pages, 7 figures, 14 tables

Electrocardiography (ECG) contains subject-specific morphology that supports biometric recognition, yet image-based performance depends on how the waveform is rendered. We introduce representative-morphology heatmaps, a deterministic ECG-to-image representation adapted from ECGXtractor. Within each block of ten aligned beats, the five beats closest to the block mean are averaged into a 400 by L matrix and rendered either as a conventional trace or as a dense cardiac-time-by-lead heatmap. Since both representations contain identical physiological samples, their comparison isolates the effect of rendering. We evaluate verification and closed-set identification on PTB, ECG-ID, and MIMIC-IV-ECG-DEMO. Five compact models, including ZACH-ViT, are trained from scratch, while six ImageNet-pretrained CNN and transformer backbones assess model scale and visual transfer. Heatmaps improve both FNMR operating points and both identification ranks in all 15 compact model-dataset comparisons, while EER improves in 14. Across the matched experiments, EER decreases by 9.59 percentage points and Rank-1 increases by 24.69 points on average. ConvNeXt-Tiny reaches 2.43% EER on PTB and 5.79% on ECG-ID, whereas DeiT-Base reaches 14.92% on MIMIC-DEMO. ImageNet initialization clearly benefits the two multilead datasets but has a mixed effect on ECG-ID, and performance does not increase monotonically with model size. The best heatmap systems approach the strongest signal-domain EER on PTB and ECG-ID, while DeiT-Base provides the strongest evaluated performance on MIMIC-DEMO. Lead-channel ablation further shows that useful channel combinations depend on the cohort and biometric task. Overall, representative-morphology heatmaps provide an effective image representation for ECG verification and identification.

0.42

MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment

cs.AI ·2026-08-17 ·Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz, Randy Gomez
similarity 0.55
Abstract

Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~$30$ achieves the highest validation performance and a test accuracy of $70.07\%$. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of $20$ seconds yielded a test accuracy of $62.71\%$, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.

0.24

Fusion techniques of time frequency-based images to predict the outcome of rTMS depression therapy

cs.LG ·2026-09-30 ·Wael Korani, Md Fahimul Kabir Chowdhury, Mohammed Aledhari, Reza Rostami, Reza Kazemi
published similarity 0.54
Abstract · Published in the Biomedical Signal Processing and Control

Depression is a mental condition that can lead to suicide and self-harm. Predicting the outcome of depression treatment is one of the most difficult tasks for clinicians. Among various treatment options, repetitive Transcranial Magnetic Stimulation (rTMS) is a widely used non-invasive method. Predicting rTMS response using Electroencephalogram (EEG) data is difficult because of high inter-subject variability and limited features from single-domain analysis. We introduce two fusion techniques, montage and blending, to overcome these limitations and extract richer features from EEG-derived Time-Frequency (TF) images. We then propose a lightweight custom Convolutional Neural Network (CNN) trained on fused TF representations. \textcolor{black}{We use a primary dataset of 15 patients and a secondary dataset of 46 patients. We run two sets of experiments. The first set uses segment-level 10-fold cross-validation. In this setup segments from the same patient can appear in both training and testing. The Montage CWT\_ST fusion reaches 99.90\% accuracy on the primary dataset and 91.90\% on the secondary dataset. The second set uses strict subject-disjoint cross-validation. All segments of a patient stay in one fold and no patient appears in both training and testing. Performance collapses. We test four time-frequency methods, six fusion mechanisms, and fourteen model architectures. With one exception, every configuration on both cohorts falls between AUC 0.31 and 0.54 and every 95\% confidence interval contains 0.5. A patient-level permutation test on the best standalone method returns $p = 0.703$. The best subject-level result is Montage CWT\_ST on the primary cohort, which reaches AUC $0.874 \pm 0.183$ and 82.7\% accuracy.

0.30

E2I-Flow: EEG-to-Image Reconstruction via Wavelet-Temporal Cross-Attention and Conditional Flow Matching

cs.CV cs.LG ·2026-11-03 ·Haowei Pan, Hwa Hui Tew, Junn Yong Loo, Fuad Noman, Hernando Ombao +1 more
similarity 0.54
Abstract

Electroencephalography (EEG)-based visual decoding and reconstruction aims to recover visual information from non-invasive brain signals. But there are challenges due to the complex temporal-frequency characteristics of EEG signals and the difficulty of discrete denoising framework in mapping neural activity to meaningful visual representations. Here, we propose E2I-Flow, an EEG-to-image decoding and reconstruction framework that combines a novel EEG encoder for time-frequency representation learning from EEG features, and conditional flow matching for visual decoding. A stationary-wavelet frequency branch and a temporal-mixing branch are integrated through bidirectional cross-attention and adaptive fusion, with the resulting EEG representations aligned to ground-truth image embeddings using contrastive and mean-squared-error loss supervisions. The aligned EEG representation is then used as a condition for a Transformer-based flow-matching model to predict the corresponding visual embedding for image synthesis. Experimental results show strong zero-shot EEG-to-image retrieval and improved visual reconstruction performance, outperforming several state-of-the-art baselines.

0.25

UNWIND: Any-Length Facial Video for Stress Detection without Temporal Windowing

cs.CV cs.AI ·2026-08-26 ·Stefanos Gkikas, Christian Arzate Cruz, Eric Nichols, Giorgos Giannakakis, Randy Gomez
similarity 0.54
Abstract

Automatic stress recognition from facial video provides a non-contact approach for affective monitoring. However, most existing video-based methods divide complete recordings into shorter temporal segments before performing classification. Such segmentation requires additional decisions concerning segment duration, overlap, and prediction aggregation, and may restrict the model from exploiting information distributed across the entire recording. We introduce UNWIND, a facial-video framework for stress detection that analyzes a complete recording as a single model input, eliminating the need for temporal windowing or external segmentation. UNWIND reorganizes the video by folding its temporal dimension into the channel dimension of a two-dimensional spatial representation, which is subsequently processed through a unified asymmetric-attention architecture. With a temporal stride of $τ=1$, the framework processes the entire $120$-second sequence, corresponding to $3{,}600$ frames sampled at $30$~fps, in a single input. We evaluate seven temporal-stride settings on a stress dataset comprising $58$ subjects, using a stratified subject-level protocol that covers configurations from dense frame retention to sparse temporal sampling. The highest test accuracy, $70.02\%$, is obtained at $τ=15$, while processing all frames at $τ=1$ achieves a comparable accuracy of $69.73\%$. Computational requirements range from $12.48$ to $348.78$ GFLOPs across the evaluated stride settings, illustrating the balance between temporal sampling density and computational efficiency. The findings show that effective facial-video stress recognition can be achieved without dividing recordings into temporal windows and that complete-recording inference can be performed within a single unified model.

0.32