CS Paper Radar

Breaking the Modality Barrier: Multimodal Fusion With Missing Modalities for Cybersickness Prediction

Purnata Saha, Rifatul Islam, John Quarles

cs.HC ·submitted 2026-11-05 03:36 UTC · v1

Abstract

Cybersickness is a major barrier to comfortable VR use, and reliable prediction often requires multiple physiological and behavioral sensors. However, existing models typically assume a fixed set of modalities, making them vulnerable when sensors are unavailable and difficult to generalize across datasets with different sensor configurations. We adapt a transformer-based model that handles arbitrary sensor combinations without imputing missing data. Anchor-based contrastive learning maps heterogeneous sensor embeddings into a shared latent space using eye tracking as the anchor modality, while masked self-attention excludes unavailable sensors during fusion. We evaluate our approach on three publicly available cybersickness datasets: VR.net, SIM21, and Blur-Rollercoaster, collected using different VR systems and sensor configurations. Across single-sensor dropout conditions, prediction error changes by less than 0.09 MAE. Joint training across all datasets achieves 0.51 MAE and 0.63 Pearson correlation, substantially outperforming single-dataset models. These results demonstrate robust and generalizable cybersickness prediction across diverse VR systems.

Closest papers

Nearest neighbours by meaning across the whole index.

Multimodal Federated Learning under Dual-Axis Modality Missingness

cs.LG cs.AI ·2026-08-10 ·Adiba Orzikulova, Jaehyun Kwak, Jaemin Shin, Yunqi Guo, Xiaomin Ouyang +3 more
code/data similarity 0.64
Abstract

Multimodal federated learning (FL) supports collaborative modeling in privacy-sensitive health-sensing and medical settings, but realistic deployments often exhibit dual-axis modality missingness: clients have different modality sets, and individual samples may contain only subsets of the modalities available locally. Existing methods typically address these two axes separately. We propose Flux, a multimodal federated learning framework built around two complementary components. First, modality-aware confidence tempering learns sample-specific confidence for each modality through mask-aware unimodal supervision and fuses the confidence estimates from observed modalities into a sample-adaptive temperature that adjusts predictive sharpness according to evidence quality and completeness. Second, gradient-decoupled private adaptation applies this temperature only to a client-private prediction pathway, while training the shared federated model with a standard, untempered objective. This enables sample-specific, client-local confidence adaptation without allowing confidence-dependent gradients to perturb shared representation learning. Across four multimodal datasets, Flux achieves the highest average macro-F1 on every dataset, outperforming the strongest dataset-specific baseline by 0.8~2.2 points and by 1.6 points on average. Additional analyses demonstrate favorable calibration, temperature sensitivity to both modality missingness and input corruption, and more stable shared optimization under private-only tempering. Our code is available at https://github.com/AdibaOrz/Flux.

0.24

How Early Can You Tell? Early Eye Gaze Dynamics and Cybersickness Progression in Virtual Reality

cs.HC ·2026-09-21 ·Nevzat Umut Demirseren, Isayas Berhe Adhanom
similarity 0.62
Abstract · 6 pages, 3 figures

Cybersickness remains one of the primary barriers to prolonged and comfortable virtual reality (VR) use, yet the temporal development of cybersickness remains poorly understood because its onset and progression vary substantially across individuals. Eye tracking provides a continuous and unobtrusive measure of user behavior, making it well suited for investigating how users respond throughout VR exposure. However, existing work has mainly focused on aggregate gaze measures or post-exposure assessments, leaving the relationship between early eye-movement dynamics and discomfort progression underexplored. In this paper, we investigate whether early eye-movement dynamics characterize the subsequent progression of cybersickness. Using trajectories of fixation duration, fixation dispersion, saccade amplitude, saccade velocity, and gaze eccentricity computed over the first one to five minutes of VR exposure, we evaluate their association with subsequent discomfort progression and compare early gaze dynamics between participants with low and high post-exposure cybersickness severity. Our results show that early fixation dispersion is consistently associated with subsequent discomfort progression, whereas no significant differences are observed between cybersickness severity groups. These findings suggest that early fixation dispersion can characterize the progression of discomfort rather than the final cybersickness severity. Overall, this work provides new insight into the temporal relationship between early eye-movement behavior and cybersickness progression, contributing to the understanding of gaze dynamics during immersive experiences.

0.19

Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling

cs.CV ·2026-08-24 ·Virmarie Maquiling, Zhuojiang Cai, Enkelejda Kasneci
benchmark similarity 0.62
Abstract · 8 pages, 1 figure, 1 table

Gaze is increasingly used as an input signal for vision and multimodal models, yet no consensus exists on how to represent it across datasets. Raw traces preserve detail but are noisy and device-dependent, while coarse event labels are easy to model but can discard local motion structure. We formulate event-aligned, fixed-horizon angular displacement as an interpretable, event-conditioned motion vocabulary and compare it with event-only, spatial, absolute-angle, learned vector-quantized, and continuous representations. To assess transfer alongside target predictability and token collapse, our evaluation combines next-token prediction with target-domain regret, low-order target references, paired bootstrap, order sensitivity, motif overlap, and frozen structural probes. In an event-aligned headset benchmark, angular-motion tokens have lower target-domain regret than frozen-codebook VQ tokens in one transfer direction, while the reverse direction is inconclusive. The probes reveal complementary representation properties, and event-only tokens show that low perplexity can retain little motion information. On a third egocentric dataset, a matched comparison of I-VT, native, and frame-span interfaces shows that event construction materially changes transfer: native events have the lowest regret into EGTEA, while frame-span events have zero motif overlap and fail severely as a source. Motion-based tokenization therefore provides a compact representation for event-aligned egocentric gaze streams, while the evaluation identifies how target predictability and event construction shape cross-dataset conclusions.

0.51

Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation

cs.CV cs.RO ·2026-08-25 ·Weizhi Tao, Zengwang Jin, Xiao Wang, Hailong Huang
benchmark similarity 0.61
Abstract · 17 pages, 9 figures, and 4 tables, including supplementary material. Submitted to IEEE Transactions on Vehicular Technology

End-to-end multimodal driving has progressed rapidly by fusing camera and LiDAR streams. Existing pipelines remain fragile under asymmetric sensor degradation, where either an entire modality or only a localized region is corrupted while other regions remain useful. The key difficulty is not simply to add an uncertainty head, but to obtain dense reliability supervision, calibrate this reliability against physical fault severity, and use it before unreliable features bias the planner. We propose Variance-Guided Spatial Attention Fusion (VG-SAF), in which dense heteroscedastic reliability estimates act as interpretable spatial gates. The framework couples three components. First, a physically grounded augmentor simulates representative camera and LiDAR failures and emits a continuous spatial mask, providing dense supervision without additional annotation. Second, modality-specific experts predict per-pixel reliability scales through cross-branch dense distillation in log space, enforcing a monotone severity-to-scale response. Third, calibrated reliability maps drive a hybrid attention mechanism that suppresses unreliable cells with a local spatial gate and arbitrates between modalities through a cross-modal trust softmax. A Laplace uncertainty head emits a systemic waypoint uncertainty scale that signals severe or combined sensor degradation, including severities outside the training ranges. On the CARLA Longest6 benchmark, VG-SAF consistently improves closed-loop robustness over the baselines across camera-only, LiDAR-only, and joint degradation regimes, as measured by driving score, route completion, and infraction score.

0.13

Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras

cs.CV cs.AI ·2026-09-07 ·Jiaqi Chen, Qinfu Xu, Hao Zhuang, Liyuan Pan
top venuebenchmark similarity 0.61
Abstract · 8 pages,5 figures,Accepted to ACM Multimedia 2026

Emotion analysis is a fundamental task in computer vision, but its practical deployment remains constrained by the privacy risks inherent to conventional RGB cameras. Bio-inspired event cameras present a promising hardware-level solution because they capture asynchronous brightness changes, thereby reducing exposure of facial identity details while leveraging high dynamic range for robust perception under challenging illumination conditions. Despite these advantages, existing event-based methods struggle in complex real-world settings due to limited dataset scales, simple acquisition conditions, and reliance on single-modality visual cues. To address these, we establish a challenging tri-modal benchmark with event, audio, and text modalities and propose the Information-Guided Gated Fusion (IGF) framework, which first pre-trains an event encoder on the FAU subset of Emo-DVS to capture fine-grained facial dynamics, then employs adaptive modality gating to suppress modality-specific noise, and finally leverages mutual information maximization to align robust cross-modal representations. To alleviate data scarcity, we introduce Emo-DVS, the first large-scale event-based emotion analysis dataset, which couples dynamic illumination with the Facial Action Unit (FAU) subset and emotion subset. Extensive experiments demonstrate that IGF achieves state-of-the-art performance.

0.45

Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition

cs.CV cs.AI ·2026-10-11 ·Yunxiang Luo, Constantino Álvarez Casado, Xiaoting Wu, Nhi Nguyen, Le Nguyen +2 more
hot direction similarity 0.61
Abstract · 6 pages, 2 figures, 3 equations, 4 tables, conference paper

Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of $75.23%\pm0.60$ averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.

0.36

FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

cs.CV ·2026-10-15 ·Xin Zhang, Robby T. Tan
publishedcode/data similarity 0.61
Abstract · NeurIPS 2026. 26 pages, 6 figures, 20 tables

Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose FuseAdapt, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs adaptation-space fusion by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce Predictive Modality Modeling (PMM), a direct representation-level objective for cross-modal redundancy. During training, PMM masks a target modality and predicts its embedding from the visible ones, supervised by a teacher target computed under full-modality input. PMM is used only during training and adds no inference-time overhead. Across three multi-modal segmentation benchmarks (MCubeS, DELIVER, MUSES), FuseAdapt consistently improves both modality-complete and modality-incomplete segmentation with a small trainable budget, yielding up to +3.01 mIoU under full modalities and +11.25 mIoU under missing modalities over prior state-of-the-art. Project page: https://devinxzhang.github.io/FuseAdapt-Page/

0.56

Slide to Report: Continuous Reporting Symptom-Specific Cybersickness in VR

cs.HC ·2026-11-05 ·Purnata Saha, Rifatul Islam, Qiliang He, John Quarles
similarity 0.61
Abstract

Cybersickness is often assessed during VR exposure through discrete overall sickness, which cannot capture specific symptoms or their changes over time. Post-exposure questionnaires such as the Simulator Sickness Questionnaire provide symptom-specific measures but cannot be collected during immersion. We investigate whether separating cybersickness into three symptom-specific dimensions -- nausea, oculomotor discomfort, and disorientation -- during immersion provides more informative continuous measurements. We conducted a between-subjects study with 52 participants in a VR-rollercoaster, comparing a three-slider interface with a single-slider interface. Concurrently, participants also completed an auditory Simon task to measure attention. We additionally collected multimodal set of physiological signals, including GSR, ECG, PPG, eye and head tracking. Results show that each slider correlated most strongly with its corresponding SSQ subscale, whereas the single slider correlated with all SSQ subscales. Using three sliders did not significantly affect usability, workload, or attention. We publicly release the multimodal dataset to support future cybersickness research.

0.20