CS Paper Radar

MICE: Multi-animal Interaction Context Encoder - A Hierarchical Foundation Model for Mouse Behavior

Yen-Shuo Su, Weihan Li, Anqi Wu

cs.CV cs.LGq-bio.NC ·submitted 2026-11-08 21:59 UTC · v1
top venuecode/data

Abstract

Quantifying mouse behavior from pose is a foundational tool in neuroscience, ethology, drug discovery, and animal welfare. Self-supervised foundation models are a natural fit, but a foundation model for mouse behavior must handle social behavior - many behaviors that matter (aggression, courtship, mounting, allogrooming) are inherently multi-animal. Existing self-supervised pose models fall short: they encode each animal independently and discard interaction signal, hard-code the animal count, and pretrain one dataset at a time. We introduce MICE (Multi-animal Interaction Context Encoder), a hierarchical foundation model for mouse behavior that pretrains jointly on six mouse-pose corpora spanning one to four mice and 7 to 27 keypoints. MICE combines a hierarchical individual encoder capturing per-mouse kinematics at multiple temporal scales with a Perceiver-style social encoder whose learned latent codebook decouples parameter count from animal count, so a single checkpoint serves any group size without retraining. Across four evaluation protocols spanning six datasets, MICE matches or exceeds prior keypoint-based baselines, and layer-wise probing shows the social stage incrementally improves discriminability beyond the individual encoder. Code, weights, and data are available at https://github.com/eric0708/MICE.

Comment
NeurIPS 2026 (Spotlight). 25 pages, 4 figures. Code: https://github.com/eric0708/MICE

Closest papers

Nearest neighbours by meaning across the whole index.

Sparse-View Interpretable 3D Animal Behavior Representations for Neural Encoding and Decoding

cs.CV ·2026-09-28 ·Xinming Dai, Qihang Jin, Tianshu Tan, Baiyuan Chen, Hanrui Lyu +6 more
similarity 0.67
Abstract

A deeper understanding of brain function requires a precise, structured characterization of behavior. Yet, extracting behavioral representations from video in a form suitable for scientific analysis remains a fundamental challenge. Many prior studies represent behavior via pose estimation or nonlinear video embeddings. However, pose tracking discards rich information beyond predefined keypoints, while nonlinear video embeddings lack interpretability. We address this limitation with SABLE (Sparse-view Animal Behavior Latent Embeddings), a self-supervised framework that leverages a geometric inductive bias to learn behavior representations. By augmenting a multi-view transformer with priors from monocular depth and pose estimation, SABLE reconstructs 3D animal behavior from extremely sparse views while learning explicit 3D latent structure. Without ground-truth 3D labels, it reliably recovers 3D behavior from two-view videos, whereas state-of-the-art (SOTA) methods fail or yield degenerate solutions. Across the International Brain Lab and Cheese3D datasets, we demonstrate that SABLE learns 3D representations that match or exceed prior SOTA performance in neural encoding and decoding. Once pretrained across animals, SABLE serves as an off-the-shelf model that generalizes zero-shot to unseen animals without animal-specific calibration or retraining. Our method establishes 3D-aware video embeddings that capture complex behavior, opening new avenues for studying brain-behavior relationships.

0.21

Representing short-term movement to reveal animal identity and learning progress

cs.LG cs.AIq-bio.NC ·2026-10-19 ·Ping Zhang, Quang-Huy Nguyen, Yingtie Lei, Sameera Shridhar, Zachary Leveroni +3 more
code/data similarity 0.66
Abstract · The first three authors contribute equally to this work

How learning progresses over time and varies across individuals is a longstanding question in cognitive and behavioral science. However, a major challenge is identifying observable behavioral measures that faithfully capture learning progression. Existing approaches commonly rely on predefined behavioral features or task-level performance, while fine-scale posture and movement in freely behaving animals remain less explored. In this paper, we ask: can short-term movement trajectories reveal learning progress and individual identity? We introduce a self-supervised representation learning framework built on a simple hypothesis: behavioral characteristics associated with identity and learning remain relatively stable over nearby time intervals. Using temporal proximity as supervision, our framework learns compact representations directly from movement trajectories without manually defining behavioral states or kinematic features. Using behavior data from freely behaving rats, we show that the learned representations not only improve prediction of identity and learning-related readouts over existing methods, but also provide a tool for investigating broader scientific questions about when and how these signals appear in behavior. Specifically, trajectories as short as one second contain informative learning and identity signals. Egocentric pose dynamics are more predictive of identity, whereas allocentric motion is more predictive of learning-related variation, and both signals are strongest at decision junctions in the alternation cohort. We further use a representation-conditioned diffusion model to inspect movement patterns associated with identity and learning progress. Our findings establish short-term movement as an informative behavioral "fingerprint" for studying learning and individual differences. Our code is available at https://github.com/pizhn/TRAIL.

0.28

Training-free motion statistics match self-supervised behavior encoders

cs.CV cs.LG ·2026-11-18 ·Ziwen Chen
benchmark similarity 0.65
Abstract · 10 pages, 7 tables, 1 figure. Code will be released with the conference version

Self-supervised encoders for animal behavior are often trained on, or fed, hand-built movement features. We ask whether they beat those ingredients when compared fairly. A training-free representation, made of generic per-frame measurements summarized over fixed time windows and reduced with PCA, is compared with retrained and released encoders (BAMS, hBehaveMAE) under a preregistered protocol: frozen representations of equal size and one shared classifier. On CalMS21 it reaches 85.8 macro F1 against 82.4 for the best encoder, ties with four labelled videos, and leads when the annotator changes or new behaviors must be learned from a few videos; the reading holds for encoders retrained with three seeds. On the MABe22 mouse triplets it scores 83.7 against 80.3 for the released hBehaveMAE (gap +3.4, 95% interval +2.1 to +4.7). On a second species, the MABe22 fly groups, the same recipe scores 51.6 mean F1 over 50 tasks against 48.2 for BAMS, the best published method. On Shot7M2, the synthetic basketball benchmark built by hBehaveMAE's authors, it scores 68.8 against 64.7 for hBehaveMAE, the best method in that paper, though hBehaveMAE remains better on the shortest movements. Short time windows carry most of the result. Raw keypoints in each body's own frame, summarized over the same windows, already match or beat the encoders on both mouse datasets and on Shot7M2; on the fly groups they fall short, and the measurements add 5.3 points. On these benchmarks, current behavior encoders have yet to beat the hand-built features they learn from, and such baselines should be reported alongside them.

0.34

BrainWideBench: Benchmarking large-scale pretraining and across-animal transfer in multi-region neural recordings

cs.LG q-bio.NC ·2026-09-18 ·Alexandre Andre, Shivashriganesh P. Mahato, Vinam Arora, Keshav Balaji, Divyansha Lachi +37 more
benchmark similarity 0.62
Abstract

Advances in large-scale neural recording have made it possible to collect data across many animals and distributed brain regions, raising the question of whether this scale can be exploited to learn general-purpose neural representations transferable across diverse downstream tasks. Yet, progress toward this goal has been limited by fragmented evaluation protocols and a narrow focus on individual task domains. Here, we present BrainWideBench, a benchmark for evaluating across-animal transfer on multi-region neural recordings, built on the International Brain Laboratory Brainwide Map dataset of neural and behavioral recordings spanning 276 brain regions from 139 mice performing a sensory-guided decision-making task. The benchmark is organized around three complementary task suites that evaluate whether learned representations support downstream decoding of behavior, can predict masked or future neural activity, and can recover biologically meaningful anatomical organization. With this benchmark, we systematically evaluate pretraining methods across transfer settings, including finetuning on downstream objectives and zero-shot generalization to unseen animals. Our results confirm pretraining improves performance over matched single-session baselines, but we show current methods exhibit heterogeneity in transfer capabilities: gains depend strongly on the alignment between pretraining objectives and downstream tasks. No single approach performs uniformly well across all three suites, and most methods are designed to only address a subset of them. Together, these findings suggest that learning representations that jointly generalize across behavior, dynamics, and anatomy remains an open challenge. By providing a unified and reproducible evaluation suite, BrainWideBench establishes a framework for measuring progress toward general-purpose models of the mouse brain.

0.40

BAL: Bidirectional Autoregression in Latent Space for Learning Human Movement Representations

cs.CV cs.LG ·2026-11-13 ·Genki Kinoshita, Ko Nishino
top venuetheorycode/data similarity 0.62
Abstract · Accepted to NeurIPS 2026. Project page: https://github.com/kyotovision-public/bal

Learning semantically meaningful human movement representations from 3D pose sequences is essential for human behavior modeling. Existing self-supervised methods derive supervision from artificial assumptions, e.g., invariance to augmentation within a certain range or recoverability of masked regions at a certain temporal scale, introducing heuristic hyperparameters that make the learned semantics sensitive to their settings. We propose BAL, a self-supervised framework that instead exploits supervision intrinsic to human movement: the natural ordering structure of everyday behavior. BAL autoregressively predicts in a semantically abstract latent space, analogous to how humans anticipate others' actions rather than their precise joint configurations. Bidirectional autoregressive prediction, both forward and backward in time, fully leverages these ordering constraints while enabling representation extraction using both past and future context at inference. To suppress representation collapse, BAL decomposes each prediction target into a global component capturing sequence-level semantics and a local component capturing context-independent movement. We prove that the local targets retain nontrivial angular diversity unless the representations are completely collapsed, preserving a meaningful training signal. Experiments on large-scale datasets demonstrate that BAL's representations are highly effective for motion captioning, forecasting, and interpolation, outperforming baselines on most evaluation metrics.

0.31

PRIMA: Pose Refined Inductive Motion Adapter for Zero Shot Social Bodily Behavior Recognition

cs.CV cs.AI ·2026-10-18 ·Mingcong Ding, Babette Bühler, Tim Fütterer, Ulrich Trautwein, Enkelejda Kasneci
top venue similarity 0.62
Abstract · 10 pages, 5 figures. Accepted at ACM Multimedia 2026 (ACM MM 2026)

Recognizing fine-grained bodily behaviors, such as self-grooming, postural shifts, and backchanneling, is a useful intermediate step for computational social signal analysis. But the behaviors that matter vary across social contexts, and retraining a model for each new taxonomy is expensive. Vision-language models (VLMs) can adapt to new categories through text prompts, yet they lack sensitivity to body structure and often confuse subtle postural distinctions. We propose PRIMA (Pose Refined Inductive Motion Adapter), a lightweight module that injects a body-structure prior into frozen VLMs using human pose as a training-time supervisory signal, requiring no pose input at inference. PRIMA separates a model-specific projection head from a shared core that encodes transferable body-configuration knowledge; adapting to a new VLM requires retraining only the head. We train PRIMA on a micro-action dataset and evaluate entirely through zero-shot prompting. The adapter improves recognition on its training domain, transfers to additional VLM backbones at different scales with the core frozen, and generalizes to two unseen target datasets from social psychology and daily activity recognition. In a supporting downstream study, the improved behavior predictions also yield higher correlations with human ratings of rapport and dominance in a collaboration context. Within the scope of the tested models and indoor datasets, the results indicate that pose-guided training can produce body-structure priors that generalize across unseen behavior vocabularies and can be partially reused across VLM backbones while preserving RGB-only inference.

0.41

HUI360: A 360° Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation

cs.CV ·2026-08-11 ·Raphael Lorenzo-Louis, Fabio Amadio, Bertrand Luvison, Serena Ivaldi
publishedbenchmarkdatasetcode/data similarity 0.59
Abstract

As robots increasingly operate in human-populated environments, anticipating human intentions is essential for enabling proactive and socially aware behavior. Automatic anticipation of human-robot interactions is thus emerging as a crucial perception challenge for embodied agents. To this end, we introduce HUI360, the largest dataset for human-robot interaction anticipation in the wild and its set of baselines. The dataset was collected from a mobile robot, in the wild, over multiple days within a 3-month period, and in several environments, capturing natural, spontaneous behaviors from both passersby and users, and encompassing a diverse range of individuals. This variety enables evaluating and improving the generalization capabilities of interaction anticipation models. We designed a pipeline and share code for automatic interaction annotation in arbitrary 360-degree equirectangular videos, along with interfaces for manual refinement. Using this pipeline, we release the HUI360 open set of 1M pre-processed annotations, including detailed 2D poses, facial keypoints, and segmentation masks, obtained using state-of-the-art computer vision methods and manually curated to ensure high-quality tracking and interaction annotation. Additionally, we release the raw panoptic 360-degree images captured from the robot's egocentric viewpoint (on demand, for research purpose only in compliance with GDPR). Finally, we establish benchmark baselines for interaction anticipation, including the first cross-dataset evaluations for this task: to this end, we also release 6M annotations for another existing in-the-wild outdoor dataset collected from a mobile robot (SSUP-HRI). Dataset and code can be found at https://hucebot.github.io/hui360.

0.76

Scaling Neural Motor Decoding via Decoupled Behavioral Pretraining

cs.LG cs.AI ·2026-11-05 ·Divyansha Lachi, Vinam Arora, Shivashriganesh P. Mahato, Alexandre Andre, Eva L. Dyer
code/data similarity 0.58
Abstract

Recent neural decoding approaches achieve strong performance by training on large collections of paired neural-behavioral recordings, but such datasets are expensive, invasive, and difficult to scale. In contrast, behavioral data is abundant and easy to collect, raising the question of whether neural decoding can benefit from decoupling behavioral and neural representation learning. We introduce BEEMO, a flexible multi-session framework that enables training with arbitrary mixtures of paired neural-behavioral recordings and unpaired behavioral data. A behavior decoder learns movement dynamics directly from behavioral trajectories, while neural activity is incorporated through cross-attention layers when available, allowing behavioral and neural supervision to scale independently. Across multiple intracortical motor datasets and tasks in nonhuman primates, we show that incorporating unpaired behavioral data improves decoding performance, particularly when transferring to new tasks, and that adding even a small amount of behavioral data from the target task further improves transfer. By decoupling neural and behavioral data, we unlock a new dimension for scaling: unpaired behavioral data, which is substantially cheaper and easier to collect than paired neural recordings, can be used directly to improve decoding performance. We further show that behavior-only pretraining, without any neural pretraining, achieves performance comparable to models pretrained on substantially larger neural datasets. Together, our results suggest that scaling behavioral data offers a practical and cost-effective path toward neural decoding models that generalize across subjects and tasks. Code is available at https://github.com/nerdslab/beemo.

0.26