CS Paper Radar

Semantic-Enhanced Sparse Temporal Modeling for Dynamic Engagement Estimation in Multimodal Interactions

Yangchen Yu, Taoliang Wang, Jia Li, Nor Mazlina Ghazali, Mohd Nazim Mohtar, Bingkun Bao, Richang Hong

cs.HC cs.CVcs.MM ·submitted 2026-08-29 10:24 UTC · v1
top venuebenchmark

Abstract

Dynamic engagement estimation aims to infer participants' evolving engagement states from continuous multimodal interaction streams. Existing approaches typically combine acoustic, pose, and visual representations with temporal models to capture observable interaction patterns. However, conversational semantics are difficult to align with frame-level engagement annotations in the same manner as continuous audio-visual signals, leaving dialogue context less effectively exploited in existing temporal modeling pipelines. Moreover, engagement annotations often evolve smoothly over time, making dense frame-level modeling computationally redundant and potentially sensitive to local annotation noise. To address these issues, we propose SEST, a Semantically-Enhanced Sparse Temporal framework that models engagement-related information from both semantic and temporal perspectives. SEST leverages Multimodal Large Language Models to extract multimodal semantic embeddings, providing dialogue-aware contextual information beyond conventional frame-level features. In parallel, SEST replaces dense-to-dense temporal modeling with a sparse-to-dense pathway, where continuous frame sequences are first encoded into compact temporal dynamics to reduce redundant computation and broaden temporal coverage, then fused with semantic context and recovered to the original label resolution for dense engagement prediction. Experiments on the MEE2026 benchmark demonstrate the effectiveness of SEST, which achieved first place in the challenge.

Closest papers

Nearest neighbours by meaning across the whole index.

DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

cs.AI ·2026-10-20 ·Yuhang Chen, Guangyue Liu, Shilong Liu
benchmark similarity 0.79
Abstract · Preprint. 19 pages, 4 figures

Recent advances in multimodal large language models (MLLMs) have substantially improved spoken interaction and video understanding. Existing online and streaming benchmarks primarily assess causal multimodal perception through timestamped question answering, whereas real-world interaction requires models to continuously identify task-relevant multimodal cues and decide whether, when, and how to respond. This leaves open how well current models can sustain real-time interaction throughout an evolving multimodal stream. To address this gap, we introduce DIVE, a benchmark for evaluating full-duplex interaction with visual engagement under continuous and realistic multimodal streams. It organizes six tasks into three complementary interaction scenarios. (1) Proactive Intervention evaluates whether models can identify emerging visual or semantic cues and intervene at the appropriate moment. (2) Continuous Engagement evaluates sustained participation through continuous visual tracking and visually grounded query interaction. (3) Interaction Coordination evaluates how models coordinate listening and speaking under overlapping and multi-party interactions. DIVE contains 525 audio/video samples with fine-grained human annotations and precise temporal metadata. Its evaluation protocol separates response quality, participation decisions, and response timing. We evaluate representative real-time models using response quality, window response rates, and reaction latency. Experiments show that leading real-time models achieve sub-second average response latency, yet their average window response rates remain at or below 53.77%. Response quality varies substantially across models and tasks, with adapting ongoing responses to newly arriving information remaining particularly challenging. We hope DIVE facilitates progress toward more natural and capable real-time multimodal interactive systems.

0.16

Multimodal Temporal Modeling for Continuous Group Emotion Recognition in Multi-party Dialogues

cs.MM ·2026-09-10 ·Soma Iwata, Koji Inoue, Muyun Wu, Taiga Mori, Divesh Lala +1 more
similarity 0.70
Abstract · 9 pages, 6 figures, 12 tables. To appear in the Companion Proceedings of the 28th ACM International Conference on Multimodal Interaction (IC…

To realize natural behavior in dialogue agents in multi-party dialogue scenarios, it is important to understand group emotion such as valence and arousal as a whole. Most prior work addressed this task at the utterance level or using a coarse-grained time window, which is not sufficient to capture emotional dynamics. In this study, we formulate continuous recognition of the Group Emotion at a one-second resolution. Moreover, we also introduce the Mixed state, which captures the emotional divergence among participants in the group. We constructed a dataset with frame-level soft labels based on the TEIDAN corpus and propose a multimodal temporal framework that integrates audio and video information using a sliding-window context. Experimental results demonstrate that the temporal Transformer outperforms simple baselines and shows stronger temporal agreement with the ground-truth labels than the LLM-based model. The effect of context length is limited, whereas audio-visual input outperforms either unimodal input on the continuous-label metrics. Additionally, our analysis shows larger Group Emotion recognition errors in intervals with high Mixed values, exposing emotional divergence as a key challenge for group emotion recognition.

0.32

MEDR: Query-Independent Frame Selection via Multi-Signal Event Modeling and Dynamic Rescoring

cs.CV ·2026-08-15 ·Xinlei Pu, Weijie Shi, Wen Yang, Yi Cao, Yao Zhao +5 more
similarity 0.67
Abstract · 11 pages, 3 figures, 4 tables; revised manuscript and updated author list

Frame selection is a fundamental component of multimodal large language models, enabling long videos to be processed under limited visual-token and computational budgets. Uniform sampling preserves temporal coverage but may miss informative content that appears only briefly. To alleviate this limitation, query-dependent methods can retrieve question-relevant frames. However, because the selected frames depend on the current question, the same visual input cannot be directly shared across different questions, and frame selection must be repeated in multi-turn video dialogue. This motivates us to seek a query-independent frame selection method that preserves the reusability of a fixed visual input while improving the coverage of informative events beyond uniform sampling. We propose Multi-Signal Event Modeling and Dynamic Rescoring (MEDR), a training-free and query-independent frame selection method. Multi-Signal Event Modeling organizes complementary visual, motion, and text signals into signal-specific temporal events. Dynamic Rescoring then iteratively reevaluates each candidate relative to the current selected set, updating its score according to frame-level signal strength, additional event coverage, and balanced temporal coverage. The resulting fixed frame set is constructed without observing the query and can be reused across different questions. Experiments on Video-MME and LongVideoBench demonstrate that MEDR achieves competitive performance across two video MLLMs. Under the 64-frame budget, MEDR improves Video-MME accuracy over the strongest evaluated query-independent baseline by 1.33-1.78 percentage points. We further evaluate the reuse of a fixed frame set in multi-question video dialogue.

0.40

YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

cs.CV ·2026-11-03 ·Shengyu Wang, Tengyue Nong, Junming Liu, Haosong Peng, Siyu Li +8 more
benchmarkcode/datacommunity pick similarity 0.67
Abstract · Project page: https://rowletqwq.github.io/YANAMI

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal stateful interaction, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce YANAMI, the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising YANAMI-400K and YANAMI-Bench. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates Dialogue State Understanding and Visual Realization, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states.

0.31

Engagement Estimation in Human-Robot and Human-Human Interactions across Domain, Age and Culture: MultiMediate'26 Challenge

cs.HC cs.CVcs.MMcs.RO ·2026-08-25 ·Marius Funk, Daksitha Withanage Don, Michal Balazia, Victor Oei, Shogo Okada +4 more
top venuebenchmarkcode/datahighly cited similarity 0.67
Abstract · 6 pages, 1 figure, ACM MultiMedia 2026 Grand Challenge Baseline Paper

Understanding engagement in human interaction is essential for modelling assistive and context-aware AI systems. Previously, the MultiMediate challenge has focused on estimating momentary continuous engagement values in adult conversation across cultures, achieving high levels of generalisability across domains. However, engagement can be measured in different metrics and manifest differently between children and adults. MultiMediate'26 therefore expands the evaluated domain set to include the PInSoRo dataset of Child-Child and Child-Robot play sessions in English. These sessions were annotated categorically with task and social engagement, diversifying engagement types, age ranges and interaction types. In this paper, we describe the MultiMediate'26 challenge and its goals, introduce the NoXi, NoXi-J, MPIIGI, and PInSoRo datasets, showcase extracted multi-modal features, present our baseline model for engagement estimation across all domains, and discuss challenge submissions and approaches. In an effort to provide a transparent and welcoming challenge environment, baseline code, leaderboard, datasets, and extracted features are publicly available at https://github.com/hcmlab/MultiMediate_26.

0.66

AIM: Adaptive Interaction in Multi-Agent Debate for Multimodal LLM Inference

cs.MA ·2026-11-06 ·Wei Fan, JinYi Yoon, Bo Ji
top venuebenchmark similarity 0.66
Abstract · Accepted in The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS 2026)

Multi-Agent Debate (MAD) improves reasoning on multimodal tasks, but its accuracy and token efficiency depend on how agents interact during debate. A largely overlooked design choice is which interaction modalities agents use to exchange evidence. Existing methods rely on fixed text- or graph-based interaction, which fails to convey fine-grained multimodal evidence and thus degrades accuracy while inflating token cost due to verbose descriptions. To address this, we propose Adaptive Interaction in Multi-Agent Debate (AIM), an adaptive routing framework that dynamically selects the optimal modality combination for inter-agent interaction in multimodal MAD. Beyond text and graph, AIM introduces task-dependent modality interaction, where agents exchange targeted regions of interest from the task-specific multimodal input, such as image regions, audio segments, or spatio-temporal video clips. To determine the best combination for each instance, AIM first generates a structured baseline response and extracts interpretable routing features that capture which evidence the instance requires, how confident the baseline agent is, and how much new evidence each modality would add beyond the baseline response. A lightweight router then uses these routing features to select a modality combination that avoids both under-fusion (i.e., agents miss decisive evidence) and over-fusion (i.e., redundant modalities inflate token cost) risks. Across eight benchmarks spanning three multimodal task families, AIM achieves the highest accuracy. Combining modalities improves accuracy by up to 6.4% over the best single-modality baseline, and adapting the combination per instance adds up to 4.2% over the most accurate fixed combination of each benchmark with up to 53.4% fewer tokens.

0.24

FORTE: Adaptive Scoring and Exact Keyframe Selection for Long-Video Question Answering

cs.CV ·2026-09-30 ·Haifeng Huang, Biyin Xu, Chunsheng Xin, Yang Li
similarity 0.65
Abstract

Query-aware keyframe selection enables multimodal large language models (MLLMs) to process long videos using only a small set of question-relevant frames. Existing score-based methods, however, typically search within a fixed, uniformly sampled candidate pool, preventing evidence outside this pool from ever being selected. Given a limited relevance-scoring budget, the key challenge is to allocate evaluations adaptively to promising frames while continuing to explore underrepresented temporal regions. We introduce FORTE, a training-free framework that addresses this challenge through two stages: adaptive relevance scoring and global keyframe optimization. Starting from sparse, uniformly distributed observations, our efficient Gaussian-process relevance predictor estimates relevance for unscored frames, exploiting temporal locality and the approximately banded kernel structure to reduce the core computation from cubic to linear time in the number of frames for fixed bandwidth. The scoring stage then selects which frames to score next by balancing predicted relevance with temporal coverage, prioritizing promising regions while also exploring less-represented parts of the video. The optimization stage selects the final keyframes by maximizing an objective that jointly captures measured relevance and temporal coverage. We derive an exact algorithm that leverages the logarithmic coverage structure to identify the optimal subset of the scored candidate pool in time linear in the pool size, for a fixed final-frame budget. Experiments on four long-video question-answering benchmarks show that FORTE achieves the highest observed mean accuracy among the compared selectors under every tested scoring budget. Further evaluations demonstrate its consistent effectiveness across different relevance scorers and downstream MLLMs.

0.15

Kairos: A Dataset for Fine-Grained Video-Language Modeling over Space, Time, and Dynamics

cs.CV cs.AI ·2026-09-08 ·Ruibo Ming, Lei Sun, Deheng Zhang, He Zhang, Jialu Li +6 more
dataset similarity 0.65
Abstract

Many emerging video language modeling tasks require systems to move beyond clip-level abstraction and model visual content as it unfolds over extended time horizons. However, most existing video datasets rely on coarse or sparsely aligned supervision, which compresses temporal variation and limits the ability of models to learn reusable representations of continuous visual dynamics. We introduce Kairos, a video dataset for video-language modeling with time-resolved annotations. Kairos consists of long-duration videos, ranging from ten minutes to half an hour, annotated with fine-grained temporal alignment. The annotations capture ongoing actions, entity appearances and attributes, interactions, and evolving contextual cues along the video timeline. This time-resolved structure supports fine-grained evaluation, long-range modeling and reasoning, instruction data construction, representation learning, and video generation. Kairos provides a general-purpose foundation for modeling visual experiences over time.

0.11