CS Paper Radar

Orthogonal Ensembles and Tested Explanations for Performer-Independent Body-Motion Emotion Recognition

Naoto Nishida, Yoshio Ishiguro

cs.CV cs.HCcs.LG ·submitted 2026-09-02 12:15 UTC · v2
publishedcode/data

Abstract

We study body-only, 12-class acted-emotion classification from skeleton motion under leave-performer-out (LPO) evaluation, a hard, underdetermined setting: chance is 8.3%, and a protocol-matched reproduced STGCN++ baseline reaches only 25.73 +/- 4.03% Macro-F1. We show that reliable gains come not from a new architecture but from combining eleven models with orthogonal error modes: under 10-fold LPO cross-validation on the labeled training performers, an equal-weight logit-mean ensemble reaches 36.80 +/- 4.00% per-fold Macro-F1, a protocol-matched +11.07 pp (+43% relative) over the same-split reproduced baseline. Our central contribution is a tested explanation suite: for a strong ensemble member, part-masking and counterfactual edits show (rather than assert) that its decisions depend on motion-grounded body-region evidence, and this region saliency aligns with rule-based Laban Movement Analysis (LMA) attributes far more than with classical kinematics: region-level saliency-LMA Spearman rho = +0.500 versus +0.033, roughly 15x, and the alignment holds for the submitted 11-way ensemble itself at rho = +0.517; the audit is post hoc and needs no retraining. The same suite faithfully reports a negative: within-window temporal saliency is diffuse rather than localized. On the hidden challenge test set the submitted ensemble scored 37.23 % Macro-F1 and received the Best Performance Award of the MMAC Challenge 2026 (Human score is 39 %). Code is available at https://github.com/nawta/diema-challenge and the presentation at https://nawta.github.io/mmac2026/.

Comment
8 pages, 3 figures, 3 tables. Accepted to ACII2026 workshop.received the Best Performance Award of the MMAC Challenge 2026
DOI
10.1109/ACIIW71585.2026.11712085

Closest papers

Nearest neighbours by meaning across the whole index.

Fine-Grained Action Recognition with Cross-Attentive Latent Sparse Experts

cs.CV ·2026-08-13 ·Imtiaz Ul Hassan, Tasweer Ahmad, Nik Bessis, Ardhendu Behera
published similarity 0.62
Abstract · Accepted manuscript: Workshop on Affective & Behavior Analysis in-the-wild (ABAW), as part of European Conference on Computer Vision (ECCV) …

Fine-grained human action recognition (FHAR) must distinguish visually similar actions that differ mainly in body configuration, timing, or local appearance. RGB representations retain visual context but often suppress joint-level geometry, whereas skeleton representations encode kinematics but discard dense spatial detail. We introduce FineX, which factorizes fine-grained cues into RGB appearance, pose heatmap geometry, and skeletal-graph topology. Pairwise cross-attention enables symmetric, stream-preserving information exchange, followed by a streamwise latent sparse Mixture-of-Experts that routes each representation to a content-dependent subset of shared experts, regularized by a load-balancing objective. FineX achieves state-of-the-art results on Gym99, Gym288, and Diving48. On the long-tailed Gym288, it raises mean class accuracy from 68.6% to 76.2% (+7.6 points) without textual supervision or large-scale vision-language pre-training, demonstrating the benefit of structured visual-pose-graph fusion and conditional expert refinement for FHAR.

0.20

Ampe-DB: A Dataset for Human Activity and Interaction Analysis Using the Ghanaian Game Ampe

cs.CV cs.LG ·2026-10-23 ·Ruth Senior Biney, Isaac Kojo Yedonu Aboah
benchmarkdatasetcode/data similarity 0.62
Abstract · 74 pages, 9 figures, 7 numbered tables. Ampe-DB V1 dataset (637 interaction clips) available at https://huggingface.co/datasets/RuthBiney/Am…

Human activity recognition benchmarks underrepresent culturally situated forms of movement and interaction. We introduce Ampe-DB, a multimodal dataset based on Ampe, a traditional Ghanaian game involving coordinated jumping, leg movements, and two-person interaction. It contains 637 canonical clips from 15 participants, with RGB video, two-person pose representations, participant and pair metadata, four-class movement labels, and derived binary movement-relation labels. We evaluate four-class movement recognition and binary movement-relation classification using three protocols: a duplicate-safe fixed benchmark, four-fold pair-held-out evaluation, and participant-held-out diagnostics for participants with complete class coverage. Baselines comprise a frozen Kinetics-400-pretrained R(2+1)D-18 RGB encoder, a two-layer bidirectional GRU for pose sequences, and validation-selected RGB-pose late fusion. For four-class recognition, RGB macro-F1 falls from 0.955 on the fixed benchmark to 0.418 under pair-held-out evaluation and 0.384 under participant-held-out evaluation. Pose is more robust, achieving 0.957, 0.811, and 0.839 across the same protocols. Fusion reaches 0.976 on the fixed benchmark and 0.824 under pair-held-out evaluation, but does not consistently improve on pose under participant shift. Binary movement-relation recognition also becomes substantially harder under held-out evaluation, especially for different-leg configurations. These findings show that strong fixed-benchmark scores can mask failures to generalize across people and pairings. Ampe-DB supports research on fine-grained movement recognition, human-human interaction, multimodal learning, and identity-aware generalization.

0.33

HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark

cs.RO cs.AIcs.CV ·2026-08-13 ·Dairu Liu, Zekun Qi, Jiayu Zeng, Ruixi Yu, Yu Guan +9 more
top venuebenchmark similarity 0.61
Abstract · Accepted to ECCV 2026

Humanoid motion tracking is central to teleoperation and whole-body imitation, yet evaluation often disagrees with what people perceive in videos. Kinematic errors average per-frame pose differences but miss the physical artifacts that matter most, particularly unstable support and incorrect contacts such as foot skating and mistimed touch-downs. Meanwhile, widely used test suites are small and lack the diversity needed to stress contact-rich, long-horizon behaviors. We introduce HumanTracker to make humanoid tracking evaluation both perceptually aligned and scalable. The HumanTracker benchmark contains approximately 153 hours of optical motion trajectories from multiple professional performers, organized into four motion families with text labels for fine-grained diagnosis. We further propose HumanScore, a preference-aligned metric trained on 12K motion pairs containing 24K motions. Across representative state-of-the-art trackers, HumanScore better predicts human preferences and reveals contact and stability failures that kinematic metrics often miss.

0.64

EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

cs.GR cs.CVcs.LGcs.MM+3 ·2026-08-19 ·Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier Perrotin
similarity 0.61
Abstract

Embodied conversational agents require synchronized full-body motion (body gestures and facial expressions) that aligns with speech and emotional state. Omni-modal large language models excel at multimodal understanding but produce only linguistic outputs, leaving a critical gap in embodied response generation. We identify and address a failure of emotion conditioning: like other conditional generators that under-use weak conditioning signals, a flow-matching model given both a rich audio embedding and a discrete emotion label suppresses the emotion, generating near-identical motion regardless of the specified emotion. We present EMODY Flow, a lightweight (around 35M parameters) flow-matching framework that attaches to a frozen Qwen-3 Omni model and reuses its internal Mimi audio-codecs to condition two parallel DiT generators - one for SMPL-X body pose, one for FLAME facial expressions. A training-time auxiliary emotion classifier restores emotion sensitivity by forcing generated motion to be emotion-identifiable. EMODY Flow sets a new state of the art on BEAT2 gesture quality, with FGD 0.302, Beat Correlation 0.853, and Diversity 24.62 - improving over the best prior results by 26%, 5%, and 62% respectively - and transfers to zero-shot facial animation on TFHP without domain-specific fine-tuning. Beyond these quantitative gains, the classifier yields clearly emotion-separated motion, which we demonstrate qualitatively through a multidimensional-scaling analysis of the generated gestures.

0.23

EgoActionBench: Benchmarking Dense Action Captioning for Egocentric Human Videos

cs.CV ·2026-11-02 ·Yujiao Shen, Jiahe Song, Haitian Li, Jingkang Yang, Ziwei Liu
benchmarkcode/data similarity 0.61
Abstract · 32 pages, 16 figures. Project page: https://egoaction-bench.github.io

Learning from human data offers a scalable path toward embodied intelligence, yet it requires models to accurately understand fine-grained human actions from egocentric observations. However, existing benchmarks remain limited in action atomicity, temporal precision, and hand-action association. We introduce EgoActionBench, a benchmark that represents continuous human manipulation as temporally localized atomic actions associated with the left hand, right hand, or both hands. EgoActionBench contains 1,407 episodes with 41,132 human-refined action annotations. To evaluate such dense predictions, we propose an evaluation protocol that retrieves temporally eligible candidates, selects a prediction by content agreement, and scores semantic fidelity and temporal overlap alongside action precision and recall, jointly capturing what happens, when it happens, and which hand performs it. We benchmark 20 baseline models and systems on EgoActionBench and further adapt two backbones, Qwen3-VL-4B and LLaVA-OneVision-2-8B, through supervised fine-tuning followed by reinforcement learning on automatically annotated training episodes. EgoAction improves EgoActionScore by 15.9 and 23.2 points on the two backbones, and both adapted models outperform all open-source baselines. Our experiments reveal substantial limitations of current models in action atomicity, temporal precision, and hand-action association, highlighting the gap between general video understanding and the action understanding required for embodied intelligence.

0.25

MotionBench-XAI: Benchmarking What Shapes Shapley Explanations in Spatiotemporal Machine Learning

cs.LG ·2026-11-05 ·Mathilde Papillon, Louisa Cornelis, Nina Miolane
benchmarkcode/data similarity 0.60
Abstract

Predicting outcomes from spatiotemporal data, such as assessing Parkinson's severity from human motion, often requires explainability in addition to accuracy. Practitioners need to know what drove the prediction, whether it be spatial (a body joint), temporal (a specific window), or both. Shapley values are attributions computed by treating input features as players in a cooperative game and measuring how the model output changes as coalitions of players are removed. Computing them involves choosing what counts as players, how absent players are imputed, and how the game is evaluated. How these choices affect explanation quality is poorly understood for spatiotemporal data, and real data carries no ground-truth attributions to check against. We present MotionBench-XAI, a benchmark built on i) five synthetic spatiotemporal datasets with exact ground-truth Shapley values, ii) three real datasets across clinical gait, electrocardiogram, and audio, iii) ten attribution methods across three player sets, and iv) three trained architectures. We show that grouping features into players, across time, space, or both, matters more than the choice of imputer. Generative imputers improve explanations only when their conditional models are accurate. On real data, the standard proxy metrics select contradictory best methods. We distill these findings into practical insights and provide an open-source pipeline.

0.29

Why Personalization Matters: Cross-Subject Challenges in EMG-IMU-based HRI Activity Recognition

cs.RO ·2026-08-21 ·Ruan Rithelle Chagas de Faria Carminati, Giovanni Braglia, Luigi Biagiotti, Ronnier Frates Rohrich, Andre Schneider de Oliveira +2 more
publishedcode/data similarity 0.60
Abstract · Data collection was approved by the Federal University of Technology-Paraná Ethics Committee (CAAE 91430125.0.0000.0177). The MAGIC-HRI (Mul…

This paper investigates wearable-based recognition of human activities and gestures to support Human-Robot Interaction (HRI) in object-handover and assembly-like scenarios. Electromyography (EMG) and Inertial Measurement Unit (IMU) signals were collected using a Myo armband, culminating in a novel dataset introduced as MAGIC-HRI (Multimodal Activity, Gesture and Intention Collection) with a large taxonomy of 53 movement classes, including Brazilian Sign Language (LIBRAS) numbers (0-9), hand gestures, object/tool handover actions (pick up/give/hold), tool-manipulation tasks, and generic assembly/idle motions, collected from 11 participants with 10 samples per class (530 samples per participant). Signals are segmented by detecting muscle activation via an EMG energy envelope, then processed using sliding windows; time- and frequency-domain features are extracted. Multiple classical classifiers are tuned via cross-validated grid search, with Random Forest as the strongest baseline. A Leave-One-Subject-Out (LOSO) protocol reveals a large generalization gap, indicating substantial subject dependence. A personalized adaptation experiment suggests that injecting a small number of samples from a new user can markedly improve recognition. Overall, the study contributes a broad, HRI-driven multimodal dataset, a rigorous evaluation emphasizing generalization, and practical evidence that personalization is likely required for robust deployment in practical HRI.

0.33

Recognition-Conditioned Reasoning: A Training-Free Multimodal-LLM Pipeline for Fine-Grained Micro-Action Understanding

cs.CV cs.MM ·2026-08-21 ·Fengshun Wang, Jin'ang Han, Zhigang Tu
top venue similarity 0.60
Abstract · Accept at ACM Multimedia 2026

Micro-actions are subtle, short, low-amplitude body movements, such as a fidgeting hand or a slight head tilt, that humans perform with little conscious intent yet that reliably leak emotional and psychological state. Understanding them goes beyond assigning a label: a model must also describe which body parts move and reason, faithfully, about why a clip warrants a particular fine-grained category. We present the training-free, prompt-only system that won first place in the fine-grained understanding track (MA-Bench) of the MAC~2026 Micro-Action Challenge, where both fine-tuning and ground-truth supervision are disallowed. Built entirely upon frozen multimodal large language models (MLLMs), the system dynamically routes each of the eight sub-tasks to the MLLM empirically best suited for that task: a discriminative MLLM for closed-ended recognition tasks and a generative MLLM for open-ended description and reasoning tasks. This architecture achieves a statistically significant performance advantage on open-ended tasks, attaining an average score of 2.68 (on a five-point scale) compared to 1.44 for the second-best approach.

0.52