CS Paper Radar

E3Sense: Head-Confined Multimodal Sensing of Learner Engagement

Sidharth Anupkrishnan, Itir Sayar, Jeongah Lee, Sri Harsha Musunuri, Guan-Ming Su, Madeline Endres, Ravi Karkar, Phuc Nguyen

cs.HC ·submitted 2026-09-22 15:19 UTC · v1

Abstract

Engagement-aware learning systems could provide hints or adjust pacing when learners struggle. Prior engagement sensing work distributes sensors across or outside the body rather than consolidating them at one site, or reduces engagement to shared affect or a single dimension (from behavioral, emotional, and cognitive engagement). We introduce E3Sense, a head-worn platform that co-locates electroencephalography, eye tracking, and electrodermal activity to personalize engagement measurement. During a lab study we collected 450 ratings of engagement levels on a five-level ordinal scale while participants watched educational videos. For fifteen held-out participants, E3Sense achieved a within-one-level prediction score of 75.0%, compared with 63.0% for always predicting the most common rating. In an exploratory analysis of 18 participants from the same study who defined engagement, conditioning on learners' definitions raised the same measure by 6.9 points, from 64.6% to 71.5%. Our work provides a proof-of-concept of a head-site, personalized multimodal sensing of engagement for adaptive educational interfaces.

Closest papers

Nearest neighbours by meaning across the whole index.

Preference vs. Performance: EEG-Based Classification of Learner Engagement in Multimodal Instruction

cs.HC cs.CY ·2026-10-04 ·Sri Jahnavi Adusumilli, Deepak Giri, Pallavi Vaswani, Pallavi Singh, Megha Moncy +2 more
published similarity 0.71
Abstract · Presented at The 26th IEEE International Conference on Advanced Learning Technologies (ICALT) 2026, July 6-9, 2026 at Hung Yen, Vietnam

Effective adaptive instructional systems require robust measures of learner engagement that go beyond static user profiles. This study employs Multimodal Learning Analytics (MMLA) to investigate the relationship between self-reported instructional modality preferences and objective neurophysiological markers of engagement. Thirty-seven participants engaged with learning content delivered via varying modalities (visual, auditory, reading/writing, kinesthetic). We captured real-time neural activity using two EEG devices: the Emotiv EpocX (14 channels, 128 Hz) and OpenBCI (16 channels, 125 Hz). Preferences were assessed using the VARK questionnaire. Consistent with literature challenging the "meshing hypothesis," aligning instructional modality with stated preferences did not significantly predict performance gains. However, spectral analysis of EEG data revealed divergent engagement patterns: when content aligned with preferences, distinct neural activity patterns emerged in theta and alpha frequency bands-markers associated with attention and cognitive processing. These signals were used to train a binary logistic regression classifier, achieving a mean accuracy of 83.21% with OpenBCI and 56.27% with Emotiv EpocX. These findings suggest that while self-reported preferences may not dictate learning outcomes, they significantly influence neurophysiological engagement, offering a viable, non-invasive input for adaptive educational algorithms.

0.30

Mind the Student: Behavioral and Contextual Cues for Automated Engagement Prediction in Online Learning

cs.CV cs.AIcs.HCcs.LG ·2026-08-25 ·Alperen Kantarci, Visvanathan Ramesh, Gemma Roig
publishedbenchmark similarity 0.63
Abstract · Accepted to ICMI 2026 (International Conference on Multimodal Interaction), October 5-9, 2026, Napoli, Italy. 5 pages, 1 figures

The prediction of student engagement from the online tutoring videos is difficult because engagement is a multidimensional construct comprising distinct behavioral, emotional, and cognitive states. A reliable prediction requires bringing together different types of behavioral signals as well as expressive cues. Through our analysis of the CASED dataset, it is clear that engagement prediction gets even harder due to the high inter-person variability as well as the subjectivity of the engagement annotation. To tackle these challenges, we develop a multimodal framework that integrates the implicit spatiotemporal features extracted from pretrained video, audio, and image encoders along with structured behavioral modalities like head pose, gaze, facial action units, emotion, and wavelet-based audio features. We integrate these modalities via a Perceiver IO latent bottleneck. Moreover, student and instructor personalities are modeled as variational posteriors over learnable embeddings to enable partial pooling across participants. We employ evidential regression and spectral-normalized Gaussian process classification heads for uncertainty-aware prediction to further improve robustness and calibration. Benchmark on the CASED challenge test set shows that all participating methods converge near random-chance performance, revealing the difficulty of the dataset. In this highly ambiguous regime, our framework achieves competitive performance while uniquely offering well-calibrated uncertainty metrics, demonstrating that reliable risk-quantification is an essential prerequisite for deploying engagement models in real-world educational tools.

0.63

Establishing a Dynamic Multimodal HRI Dataset for Engagement Analysis with a Humanoid Robot

cs.RO ·2026-09-03 ·Buwan Kim, Wonse Jo
similarity 0.59
Abstract

This paper presents an experimental design for constructing a multimodal dataset to analyze user engagement in human-robot interaction (HRI). Prior studies have mainly relied on observable behavioral cues, with limited frameworks integrating physiological signals. We therefore propose a structured data-collection protocol to build a multimodal dataset that includes wearable physiological signals, behavioral data, and self-report measures under different levels of task complexity defined in this experiment.

0.09

Modeling Learner-Technology Interaction Through Multilevel Gaze Transition Dynamics to Inform Usability in Online Learning

cs.HC ·2026-11-18 ·Yueqi Weng, Jie "Jennifer" Lu, fan Yang
similarity 0.59
Abstract

Gaze data provide fine-grained evidence of how learners engage with instructional content and interface elements, complementing conventional usability evaluation methods. However, eye-tracking analyses often examine within-slide visual attention and across-slide navigation separately, limiting understanding of learner interaction throughout extended learning experiences. This exploratory multi-method study integrated area-of-interest transitions, gaze entropy, slide-level navigation, and network-based representations to analyze seven eye-tracking recordings of high school students interacting with an AI-supported online biology lesson, complemented by three post-intervention focus groups. Results showed individual differences in within-slide gaze allocation and switching patterns. Learners followed shared navigation pathways but differed in revisiting and immediate return behaviors. The focus group findings highlighted positive perceptions of interactive materials and AI-generated feedback, alongside usability concerns related to navigation, interface clarity, and assessment guidance. Together, these outcomes show how integrating multilevel gaze analytics with learner perspectives can provide complementary evidence for identifying interaction trends relevant to usability evaluation. Future research should test this approach with larger, more diverse samples and examine relationships among gaze patterns, usability, and learning outcomes.

0.27

Tracking Human Daily Cognitive Activity from EEG and Biometric Data

cs.HC ·2026-10-02 ·Alina Gutoreva, Zhaniya Omar
published similarity 0.58
Abstract

Understanding human cognitive activity in everyday life remains challenging due to the dynamic, context-dependent, and multimodal nature of cognition. Laboratory-based studies often fail to capture real-world cognitive processes, while single-modality approaches provide only partial insight into cognitive states. Advances in wearable sensing now enable the collection of heterogeneous data streams for a more comprehensive view of daily cognition. This paper presents a multimodal framework for tracking human cognitive activity using electroencephalography (EEG), wearable physiological signals, behavioral context, and self-reported measures. A preliminary pilot study was conducted with observational data from three participants (N = 3) over 280 annotated 10-minute intervals spanning nine activity domains across two weeks. Results reveal consistent temporal patterns, including a discernible mid-day decrease in motivation and energy at 13:00, followed by afternoon recovery. Work and IADLs yielded the highest flow state rates (51% and 50%), while ADLs produced the lowest (15%). Motivation correlated strongly with arousal (r = 0.78) and attention (r = 0.74), whereas perceived stress showed a weaker negative relationship (r = -0.33). A linear regression model predicting motivation from arousal, attention, energy, and stress achieved R2 = 0.76 (MAE = 9.84, RMSE = 12.85). Lag-based analysis indicates that prior energy levels positively predict subsequent motivation, confirming temporal dependencies in cognitive dynamics. These findings demonstrate the feasibility of multimodal cognitive activity analysis in real-world environments and highlight the importance of integrating physiological and behavioral indicators. The proposed framework provides a foundation for future large-scale multimodal systems and applied intelligent solutions.

0.22

Engagement Estimation in Human-Robot and Human-Human Interactions across Domain, Age and Culture: MultiMediate'26 Challenge

cs.HC cs.CVcs.MMcs.RO ·2026-08-25 ·Marius Funk, Daksitha Withanage Don, Michal Balazia, Victor Oei, Shogo Okada +4 more
top venuebenchmarkcode/datahighly cited similarity 0.56
Abstract · 6 pages, 1 figure, ACM MultiMedia 2026 Grand Challenge Baseline Paper

Understanding engagement in human interaction is essential for modelling assistive and context-aware AI systems. Previously, the MultiMediate challenge has focused on estimating momentary continuous engagement values in adult conversation across cultures, achieving high levels of generalisability across domains. However, engagement can be measured in different metrics and manifest differently between children and adults. MultiMediate'26 therefore expands the evaluated domain set to include the PInSoRo dataset of Child-Child and Child-Robot play sessions in English. These sessions were annotated categorically with task and social engagement, diversifying engagement types, age ranges and interaction types. In this paper, we describe the MultiMediate'26 challenge and its goals, introduce the NoXi, NoXi-J, MPIIGI, and PInSoRo datasets, showcase extracted multi-modal features, present our baseline model for engagement estimation across all domains, and discuss challenge submissions and approaches. In an effort to provide a transparent and welcoming challenge environment, baseline code, leaderboard, datasets, and extracted features are publicly available at https://github.com/hcmlab/MultiMediate_26.

0.66

When Attention Guardrails Become Barriers to Learning: Towards the Tipping Point

cs.HC cs.CY ·2026-09-30 ·Meenakshi V., Pavani Ayinampudi, Aditya B. M. V., Jinal Gupta, Prakash Hegade +3 more
position similarity 0.56
Abstract · 10 pages, 2 figures, 4 tables

Online learning offers flexibility but lacks the structure of a classroom, where a teacher's presence guides attention. The platform we study restores that structure by monitoring the learner through the webcam during ordinary coursework, interrupting or restarting a video when the learner appears distracted. What such monitoring does to a learner across a whole course, rather than in a single examination, is largely unexamined. We report a convergent mixed-methods study of one monitored course pipeline in a summer internship program. We read two free-text surveys alongside three platform channels. The dropout exit survey yielded 15 analyzable responses; the persisting-learner reflection survey, 36. The channels are camera-verification telemetry (14,529 flags from 448 students), an in-video emotion widget (615 submissions from 273 students), and a mandatory end-of-course survey (up to 634 respondents per item). Neither survey named monitoring, so every mention analyzed here was raised by the respondent. Focus-monitoring was raised by 18 of the 51 free-text respondents: 7 of 15 dropouts and 11 of 36 persisting learners. Among the dropouts who raised it, focus-monitoring was the stated primary cause of departure in 4 of 7 cases. None of the 18 questioned being observed in principle. What learners contest is the misreading of ordinary actions, drinking water or moving the head, and the severity of what follows a flag: a video already watched returns to the start of its segment. These findings identify two targets for redesign: the severity of the response to a flag, and the environment check, which can flag a learner before any content has been seen. We argue that the proportionality of that response marks the point at which an attention guardrail becomes a barrier to learning, the tipping point this study approaches.

0.33

EE-Motion: An EEG-based Web Architecture for Live Affective Music Recommendation

cs.IR cs.HC ·2026-11-16 ·Sabino Roccotelli, Potito Aghilar, Vito Walter Anelli, Alejandro Bellogín, Tommaso Colafiglio +2 more
unusual similarity 0.56
Abstract

Music listening is intrinsically tied to emotion, which drifts continuously within a single session. Although modern browsers interface with wearable brain-computer interfaces, recommendation systems remain structured around discrete logs and lack the infrastructure to integrate continuous physiological signals. Furthermore, existing affect-aware models rely on static user profiles and offline training, failing to serve anonymous newcomers without model retraining. We present a web platform that resolves both challenges. Pairing with a consumer electroencephalogram (EEG) headband over Web Bluetooth, the system captures live EEG, mapping frontal alpha asymmetry and arousal index to normalized valence-arousal proxy coordinates on Russell's circumplex plane. Driven by these coordinates, an affective adapter unifies structurally diverse recommendation backbones under a shared live scoring mechanism. By parameterizing anonymous visitors through a few favorite tracks, the module recommends tracks conditioned on live proxy coordinates. This demonstrates that real-time EEG-derived control signals can steer recommendations without persistent user tables or model retraining.

0.26