CS Paper Radar

Why Personalization Matters: Cross-Subject Challenges in EMG-IMU-based HRI Activity Recognition

Ruan Rithelle Chagas de Faria Carminati, Giovanni Braglia, Luigi Biagiotti, Ronnier Frates Rohrich, Andre Schneider de Oliveira, Mikael Nedel Hartmann, André Eugenio Lazzaretti

cs.RO ·submitted 2026-08-21 20:36 UTC · v1
publishedcode/data

Abstract

This paper investigates wearable-based recognition of human activities and gestures to support Human-Robot Interaction (HRI) in object-handover and assembly-like scenarios. Electromyography (EMG) and Inertial Measurement Unit (IMU) signals were collected using a Myo armband, culminating in a novel dataset introduced as MAGIC-HRI (Multimodal Activity, Gesture and Intention Collection) with a large taxonomy of 53 movement classes, including Brazilian Sign Language (LIBRAS) numbers (0-9), hand gestures, object/tool handover actions (pick up/give/hold), tool-manipulation tasks, and generic assembly/idle motions, collected from 11 participants with 10 samples per class (530 samples per participant). Signals are segmented by detecting muscle activation via an EMG energy envelope, then processed using sliding windows; time- and frequency-domain features are extracted. Multiple classical classifiers are tuned via cross-validated grid search, with Random Forest as the strongest baseline. A Leave-One-Subject-Out (LOSO) protocol reveals a large generalization gap, indicating substantial subject dependence. A personalized adaptation experiment suggests that injecting a small number of samples from a new user can markedly improve recognition. Overall, the study contributes a broad, HRI-driven multimodal dataset, a rigorous evaluation emphasizing generalization, and practical evidence that personalization is likely required for robust deployment in practical HRI.

Comment
Data collection was approved by the Federal University of Technology-Paraná Ethics Committee (CAAE 91430125.0.0000.0177). The MAGIC-HRI (Multimodal Activity, Gesture, and Intention Collection for HRI) dataset is available at [https://github.com/ruancarminati/MAGIC-HRI-V01.git](https://github.com/ruancarminati/MAGIC-HRI-V01.git). This paper will be presented at IEEE RO-MAN 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Synchronized AMG and EMG Dataset of Lower-limb Muscle Activities in Everyday Training

cs.HC ·2026-08-12 ·Dongxu Tang, Shih Ying-Lei, Zhuoyi Ren, Jianting Liao, Yitian Shao
benchmarkdataset similarity 0.66
Abstract

Understanding how lower-limb muscle groups coordinate is important for studying movement impairment, rehabilitation, and physical performance. Reproducible analysis of this coordination requires multimodal recordings that relate local muscle-related signals with body-level kinematics. Complementing neural-level electrical activation captured by EMG, AMG provides a valuable mechanical approach to monitoring muscle activity. Here, we introduce a synchronized, multimodal dataset for healthy-adult lower-limb activities. For data collection on the left leg, 16 triaxial accelerometers were evenly divided into four muscle-site clusters for AMG recording, complemented by four surface EMG channels. A 15-marker optical motion-capture (MoCap) system captured lower-body kinematics, with the resulting marker trajectories used to compute bilateral knee and ankle joint angles. Our dataset contains 1,918 trials from 30 subjects across 16 task conditions. We benchmark the dataset by estimating four joint angles from 300 ms windows of the 5-100 Hz band-pass-filtered AMG data and assess matched EMG features in a separate modality ablation. In the primary cross subject benchmark, the four reference models achieved mean absolute errors of 8.840$^\circ$-9.591$^\circ$. The benchmark and ablation results characterize performance across subjects, tasks, and joint angles and examine the effects of sensor configuration, modality, the number of training subjects, and frequency representation. The release includes documented timing definitions, processed data, and reproducible benchmark resources. https://dongxutang918-afk.github.io/SAME-Limb/

0.16

EMGBlend: Heterogeneity-Aware Self-Supervised Pretraining for Gesture and Force Decoding

cs.LG cs.AI ·2026-09-22 ·Yuwei Jia, Cheng Zhong, Jinyang Yu, Zhe Cui
code/data similarity 0.66
Abstract

Public surface electromyography (EMG) datasets vary widely in electrode layout, channel count, frequency support, and size. Simply mixing them for pretraining can misalign channel semantics, introduce spectral targets that some devices cannot observe, and let large or high-channel-count datasets dominate learning. We introduce EMGBlend, a self-supervised framework designed around these differences. It combines shared channel patches with geometry-aware attention, restricts spectral targets to each recording's supported frequency band, and balances exposure across data sources. We pretrain a 109M-parameter model on 11 public EMG sources and evaluate it on gesture recognition, continuous-force regression, and contact classification. EMGBlend consistently outperforms matched random initialization and waveform reconstruction controls. Fixed-budget source controls show that multi-source pretraining improves gesture recognition and remains competitive for force decoding. Ablations confirm that geometry, band-aware targets, and source balancing each contribute to transfer, although cross-person NinaPro force estimation remains difficult. Overall, EMGBlend shows how heterogeneous EMG datasets can be combined through explicit mechanism design rather than simple concatenation. Code is available at https://github.com/tamanano/EMGBlend

0.25

MotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding

cs.CV cs.RO ·2026-10-30 ·Eadom Dessalene, Michael Maynord, Amir-Hossein Shahidzadeh, Botao He, Yasamin Tabatabaee +4 more
publishedcode/data similarity 0.66
Abstract · NeurIPS 2026. 27 pages, 9 figures, 5 tables. Project page: https://www.cs.umd.edu/~edessale/motorsense/

Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense, the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with egocentric video and yields synchronized video, bimanual EMG and IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producing an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.

0.28

EVFormer: An Egocentric Vision-EMG Bidirectional Attention Model for Bimanual Hand Pose Estimation

cs.LG cs.HC ·2026-10-03 ·JiaCheng Ge, SiYu Zhang, ShengJie Li, XinTong Yang
similarity 0.66
Abstract

Egocentric bimanual hand pose estimation is important for virtual interaction, wearable control, and rehabilitation, but visual observations are often degraded by self-occlusion, hand-hand contact, and object manipulation. We propose EVFormer, a multimodal framework that combines the current RGB frame with the preceding 200 ms of bilateral wrist surface electromyography (sEMG) to estimate 44 finger and wrist joint angles. EVFormer separately encodes visual spatial features and sEMG temporal features, enables cross-modal information exchange through sequential bidirectional cross-attention, and integrates the two modalities using feature-wise gated fusion. We evaluate EVFormer in a single-participant feasibility study using one synchronized public EgoEMG recording with chronologically separated training, validation, and test splits. On 296 test samples, EVFormer achieves a mean absolute error of 11.482 degrees, compared with 13.228-13.610 degrees for vision-only, sEMG-only, late-fusion, and training-mean baselines. This corresponds to relative error reductions of 13.20% compared with the vision-only model and 14.23% compared with late fusion. EVFormer also achieves the lowest error in four of the five evaluated gesture classes. These results provide preliminary evidence that feature-level interaction between egocentric vision and sEMG can improve bimanual hand pose estimation. Further evaluation across participants, recording sessions, sensor placements, and real-world interaction conditions is required to establish the generalizability of the approach.

0.19

LiteEMG-FM: An Efficient and Deployable Foundation Model for Robust EMG Sensing

cs.LG ·2026-10-01 ·Tianhao Wu, Xu Wu, Amirmohammad Radmehr, Jiawei Yu, Yi Wu +2 more
similarity 0.64
Abstract · The authors are withdrawing this manuscript because the current experimental evaluation and validation are insufficient to adequately suppor…

Electromyography (EMG) signals vary substantially across individuals, body regions, recording sessions, and sensing hardware, limiting the generalization of models for assistive devices and human-computer interaction. Existing time-series foundation models are also computationally expensive for real-time wearable deployment and often fail to capture EMG-specific time-frequency characteristics. We present LiteEMG-FM, an efficient hybrid CNN-Transformer foundation model for practical EMG sensing. Pretrained on 16 diverse upper- and lower-limb EMG datasets, LiteEMG-FM learns representations that generalize across users and datasets. For resource-constrained deployment, we implement a hierarchical wake-up architecture in which a lightweight, always-on 1D-CNN filters rest and non-target activity and activates LiteEMG-FM only for valid gestures. We evaluate full inference offloading, split inference, and full on-device processing, characterizing their trade-offs in latency, power consumption, and memory footprint. Across diverse evaluation settings, LiteEMG-FM outperforms state-of-the-art time-series foundation models and supervised baselines, particularly under zero-calibration cross-participant and data-scarce conditions. These results demonstrate that LiteEMG-FM is an effective, efficient, and deployable foundation model for EMG applications.

0.11

Wearable Multimodal Human-Machine Interface for Integrated Hand Intentions Decoding in Dynamic Teleoperation

cs.RO ·2026-09-07 ·Jiaxuan Li, Yinshi Wu, Xiao Zhang, Hongyu Wang, Renzhen Le +2 more
similarity 0.63
Abstract

Under ubiquitous teleoperation environments with optically challenging conditions, an interface for tele-operated grasping that combines wearability with precise decoding of hand intentions (hand pose, gestures, and grasping force) is essential. Yet, existing interfaces often fall short in meeting these demands, compromising either the diversity of multiple intentions decoding or wearability. To address this, we developed a novel Multiple Intentions Decoding Human-Machine Interface (MI-DHMI) that integrates high-throughput surface electromyography (sEMG) sensors with hand-mounted and forearm-mounted inertial measurement units (IMUs). The developed interface is supported by a unified framework for simultaneous multiple intentions decoding. By employing multimodal deep learning and hardware design with a low noise floor, the decoding framework selectively focuses on the sEMG components that are genuinely associated with finger movements. This effectively reduces decoding errors caused by sEMG variability during unconstrained upper-limb motions, thereby significantly enhancing robustness. Even under unconstrained wrist and forearm motion, the interface achieves a gesture recognition accuracy exceeding 97%, grasping force estimation with $R^2 = 0.95$, and hand pose decoding consistent with the actual hand pose, outperforming baseline devices and algorithms. Ablation studies further validate the effectiveness of the proposed decoding framework. Finally, two online experiments were conducted to validate the device, demonstrating its superior performance in high-stability tasks, including a pouring task and object grasping. The developed interface provides a new solution of a fully wearable, multiple intentions decoding system, offering effective support for ubiquitous teleoperation and contributing to the advancement of human-machine interaction research.

0.27

Ampe-DB: A Dataset for Human Activity and Interaction Analysis Using the Ghanaian Game Ampe

cs.CV cs.LG ·2026-10-23 ·Ruth Senior Biney, Isaac Kojo Yedonu Aboah
benchmarkdatasetcode/data similarity 0.63
Abstract · 74 pages, 9 figures, 7 numbered tables. Ampe-DB V1 dataset (637 interaction clips) available at https://huggingface.co/datasets/RuthBiney/Am…

Human activity recognition benchmarks underrepresent culturally situated forms of movement and interaction. We introduce Ampe-DB, a multimodal dataset based on Ampe, a traditional Ghanaian game involving coordinated jumping, leg movements, and two-person interaction. It contains 637 canonical clips from 15 participants, with RGB video, two-person pose representations, participant and pair metadata, four-class movement labels, and derived binary movement-relation labels. We evaluate four-class movement recognition and binary movement-relation classification using three protocols: a duplicate-safe fixed benchmark, four-fold pair-held-out evaluation, and participant-held-out diagnostics for participants with complete class coverage. Baselines comprise a frozen Kinetics-400-pretrained R(2+1)D-18 RGB encoder, a two-layer bidirectional GRU for pose sequences, and validation-selected RGB-pose late fusion. For four-class recognition, RGB macro-F1 falls from 0.955 on the fixed benchmark to 0.418 under pair-held-out evaluation and 0.384 under participant-held-out evaluation. Pose is more robust, achieving 0.957, 0.811, and 0.839 across the same protocols. Fusion reaches 0.976 on the fixed benchmark and 0.824 under pair-held-out evaluation, but does not consistently improve on pose under participant shift. Binary movement-relation recognition also becomes substantially harder under held-out evaluation, especially for different-leg configurations. These findings show that strong fixed-benchmark scores can mask failures to generalize across people and pairings. Ampe-DB supports research on fine-grained movement recognition, human-human interaction, multimodal learning, and identity-aware generalization.

0.33

TransfHAR: Self-Supervised Wrist Representations for On-Demand Activity Recognition

cs.LG ·2026-08-16 ·Aidan Bradshaw, Riku Arakawa, Xin Liu, Karan Ahuja
top venue similarity 0.62
Abstract · Accepted to the ACM Symposium on User Interface Software and Technology (UIST '26)

Fine-grained wrist activity recognition can support applications such as procedural step guidance and context-aware assistance, yet acquiring labeled data for every new task, user, and activity granularity remains a bottleneck. We present TransfHAR, a self-supervised wrist IMU framework for on-demand, fine-grained activity recognition by learning transferable motion priors from global, unlabeled activities. We show that self-supervised pretraining on coarse wrist IMU activities (e.g., sitting, walking, exercise) learns motion structure rich enough to transfer to fine-grained manipulative, gestural, and procedural activities (e.g., snapping, stirring, waving) that are absent from pretraining. We implement TransfHAR as a real-time smartwatch application that lets users define and expand their own activity set for personalized recognition from only a few demonstrations. Across three offline cross-dataset evaluations, TransfHAR matches or exceeds fully supervised baselines that use complete label sets with equal or additional sensor channels, by 6.2 balanced-accuracy points on average. In an in-lab study with 10 participants each performing seven novel wrist activities, TransfHAR reaches 86.7% balanced accuracy across participants with five examples per class and 90.4% when updated from a single one-minute recording per class. These results indicate that broad self-supervised wrist pretraining provides an effective foundation for on-demand fine-grained activity recognition.

0.64