CS Paper Radar

Time-Dynamic Semantic-Activity Entanglement: Quantifying Communication-Based Synchrony in Collaborative VR

Pooja Pol, Jianing Zhang, Qing Huang

cs.HC ·submitted 2026-10-27 15:40 UTC · v1

Abstract

Collaborative virtual reality (VR) produces rich multimodal traces of team behavior, yet practical methods for characterizing time-varying team communication remain limited. We present time-dynamic semantic-activity entanglement (TSAE), inspired by the entanglement concept introduced by Gloor et al. (2022), to describe communication-based team synchronization within discrete time windows. TSAE is modeled by combining relative speaking activity with contextual semantic alignment for participant pairs, which is aggregated into participant-level trajectories and summarized through the Gini coefficient to understand the distribution of team-level communication synchrony. This approach is employed with German-language transcripts from fifteen collaborative VR gameplay sessions. Transcript utterances are represented using a long-context embedding model with a late-chunking strategy and analyzed within fixed game-time windows. A representative session illustrates shifting dyadic alignment and low-to-moderate participant contributions, with sporadic periods of high entanglement. Across fifteen game sessions, the metrics indicate complementary insights, offering a qualitative contrast to the core-periphery pattern reported in the foundational work. Hence, these descriptive findings support the study of communication as a dynamic process and provide a mathematically grounded approach to analyze its evolving distribution in shared interactive VR spaces.

Comment
6 pages, 3 figures. This work has been accepted as an extended abstract in the "Poster" track for publication in the Companion Proceedings of ACM ISS 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Computational Measurement of Team-Process Phase Dynamics in Collaborative Virtual Reality

cs.LG ·2026-08-19 ·Qing Huang, Jianing Zhang, Pooja Pol
similarity 0.80
Abstract

Collaborative virtual reality (VR) environments make team communication observable as it unfolds, but conventional transcript analyses often summarize entire trials or divide them into fixed temporal windows. Such approaches can obscure changes in team communication and coordination over time. This article presents a computational framework for detecting and interpreting dynamic team-process phases from timestamped dialogue in a collaborative VR game. The framework uses late chunking to generate context-aware transcript representations, aggregates them into temporal chunks, and applies penalized Gaussian-kernel change-point detection to identify semantic transitions in team communication. After boundary detection, term frequency--inverse document frequency (TF-IDF), non-negative matrix factorization (NMF), and representative transcript segments provide structured evidence for phase interpretation. A locally deployed large language model (LLM) uses in-context learning to generate initial interpretations that are subsequently reviewed by humans. Independently recorded interaction logs are then aligned with the detected phases to examine corresponding task-action patterns. The evaluation compares representations, pooling strategies, segmentation methods, parameter settings, reviewed phase interpretations, and phase-aligned interaction profiles. The results show that the framework identifies coherent and interpretable phase structures while preserving traceability to the underlying transcript evidence. The correspondence between transcript-derived phases and interaction behavior further supports their relevance for analyzing collaborative activity. The framework therefore offers a transparent and transferable approach for studying temporal changes in teamwork from timestamped transcripts across collaborative task settings.

0.23

PUBG Ally: A Conversational Embodied Agent as an AI Teammate

cs.AI cs.CLcs.HC ·2026-09-24 ·PUBG Ally Team, Irene Chen, Youngin Cho, Seungjun Chung, Jimin Hong +20 more
similarity 0.58
Abstract · Authors are listed alphabetically. Project leads are Kangwook Lee and Hyunseung Kim

We introduce PUBG Ally, an embodied agent for PUBG: BATTLEGROUNDS that can reason, act autonomously, and play alongside players as a voice-enabled teammate. Building such a teammate requires combining two difficult capabilities: it must perceive and respond to a constantly changing game world under strict latency constraints while interacting naturally with players, keeping its speech synchronized with its actions. Ally therefore combines agentic tool use with real-time game control. A language-model agent uses a controlled interface to inspect game information, interpret player speech, maintain context, decide what to say, and issue high-level action choices that steer a faster control layer for movement, combat, and recovery. Because the player's and Ally's speech and actions continually shape each other and the course of the match, training requires data from actual gameplay. We therefore collect data across nearly 39k sessions in which real players play alongside Ally, recording gameplay, player speech, agent decisions, tool use, actions, and player feedback, and use these records for iterative training. To evaluate teammate quality, we use player feedback and preference comparisons to identify gaps between offline evaluations and player preferences, and iteratively refine the evaluation criteria. Deploying Ally in live service further requires low-latency on-device execution and safeguards for player-facing communication, which we address through model compression, context compaction, targeted safety training, runtime guardrails, and memory redaction. During the live service, we surveyed players in 141 countries. Among respondents whose play with Ally was confirmed in game records, positive responses exceeded negative responses by 25.1 percentage points when asked whether they would recommend Ally, with players describing Ally not only as a tool but also as a teammate or companion.

0.26

Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue

cs.CL ·2026-09-14 ·Leon Hammerla, Patrick Schrottenbacher, Alexander Mehler
similarity 0.57
Abstract · To be submitted to the October 2026 cycle of ACL Rolling Review (ARR)

Negation is typically modeled through its linguistic realization, although spoken interaction is accompanied by tightly coordinated nonverbal behavior. We ask whether contexts centered on spoken negation cues contain measurable multimodal behavioral information: whether they can be distinguished from matched control contexts without lexical or acoustic input, where this information occurs in time, which modalities carry it, and whether it extends to the dialogue partner. We study 27 human-human interviews conducted in virtual reality, comprising temporally aligned gaze, facial, head, body, hand, and finger behavior and 964 annotated negation cues. Treating classification as a predictive probe, we compare 20 time-series models while excluding lexical and acoustic information, and then systematically vary temporal context, interactional source, modality availability, and event timing. Across grouped 10-fold cross-validation, the strongest probes reach up to .75 mean held-out AUROC from speaker-side behavior. Temporal analyses show that predictive information is concentrated around cue onset but remains detectable over a broader surrounding interval, while dialogue-partner behavior carries weaker predictive information with a comparatively diffuse temporal profile. Ablation and timing perturbations further show that facial features produce the largest modality-ablation effect and that the trained probe is sensitive to the temporal organization of the observed events.

0.45

Esports World Model Dataset

cs.CV ·2026-10-12 ·Weiyu Ma, Zeyuan Gong, Xuhui Liu, Wenxuan Zhang, Jian Ding +3 more
dataset similarity 0.57
Abstract

Interactive world models must capture how multiple participants jointly change a shared environment while observing different parts of it. Competitive games make this challenge concrete: players cooperate and compete, while fog of war, team vision and camera placement determine what each can see. Modeling these interactions requires data that connect individual observations and actions to shared state and events. We introduce the Esports World Model Dataset (EWM), a collection of human gameplay spanning StarCraft II, Dota 2 and Counter-Strike 2, with 3,053 games and maps and approximately 4,880 player-view hours, including estimated hours for the ongoing CS2 extension. It includes 4,628 completed StarCraft II views, all ten perspectives for 303 Dota 2 matches, and up to ten views per round in CS2. Across these views, replay-clock alignment connects rendered video with game-native actions, structured state and events from the same match. The three games expose complementary observation conditions: private fog of war, shared team vision and first-person viewpoints. In StarCraft II, player-specific entity records further distinguish live observations from remembered snapshots and link co-observed entities across opposing views. Together with temporal alignment, these records allow researchers to distinguish legitimate information differences from disagreement about shared state. By connecting human decisions, shared dynamics and player-specific information, EWM supports research toward jointly controlled multiplayer world models and interactive environments populated by teammates and opponents learned from human play.

0.21

A study on human-agent teaming through spoken interaction: the impact of human individual traits and agent characteristics

cs.HC cs.AIcs.DB ·2026-10-02 ·Lara Gauder, Martin Bernardo Meza, Javier Krick, Alejandro Masin, Luciana Benotti +2 more
similarity 0.56
Abstract

Human-agent teams are collaborative systems where humans and agents work interdependently to achieve shared goals. The success of such teams is associated with the human's perception of the agent as a legitimate teammate. This perception is thought to depend not only on the agent's capabilities and reliability but also on social factors. We explore whether simple changes in an agent's communicative behavior can influence the human's perception of the agent's teammate-likeness. To this end, we developed a protocol based on a collaborative game requiring spoken interaction, in which a human subject and a virtual agent collaborate to identify a target object and place it on a board. Each subject interacted with two agents which provided the same task-relevant information but differed in behavior. While one was a neutral tool-like agent, the other was a team-building agent that used simple social strategies such as empathy, politeness, and positivity. Our analysis shows that the team-building agent was perceived as significantly more teammate-like, even in terms of its ability, despite both agents having identical capabilities for game-solving. Further, subjects with a higher propensity to trust automated systems and lower neuroticism tended to have a more favorable perception of the agent. Finally, we found that subjects tended to change their prosodic patterns when talking to the team-building agent, as a classifier based on prosodic features was able to predict the type of agent with better-than-random performance. A practically relevant conclusion is that minimal changes in spoken communication behavior of the agent, easily implemented in a variety of scenarios, can have a significant positive impact on the human's perception of its teammate-likeness.

0.18

InterView-C: A Synchronized Multimodal Corpus of VR Avatar-Mediated Survey Interviews

cs.CL ·2026-10-07 ·Patrick Schrottenbacher, Leon Hammerla, Lydia Kleine, Doris Stingl, Alexander Mehler
similarity 0.55
Abstract

We present InterView-C, a German multimodal corpus of 27 survey interviews conducted entirely in virtual reality, with both interlocutors represented by avatars. The corpus aligns spoken interaction with synchronized behavioral data, including gaze, head and body movement, facial behavior, hand and finger tracking. Its reference transcripts and linguistic annotations provide a reliable interface between this multimodal spoken interaction and predominantly text-based NLP methods. This interface is important because automatically transcribing speech can distort linguistically relevant information, while downstream models trained on existing resources may additionally face transfer challenges when applied to transcribed spoken data. InterView-C therefore provides word-timed and manually post-edited verbatim transcripts for all 54 recordings, interview-item timings, questionnaire responses and negation cue and scope annotations for 1,422 sentences, 1,398 of them doubly annotated (α=0.87 for cues; α=0.81 for scopes). We demonstrate both challenges empirically: nine open-weight ASR systems disproportionately misrecognize short closed answers and number words, while negation models trained on existing corpora show lower and highly variable performance on our transcribed interviews than a model trained on the InterView-C annotations. InterView-C thus enables linguistic analyses of spoken interaction while retaining their alignment with rich multimodal behavior.

0.07

SynerGest: Temporal-Spatial Synergistic Gesture Generation for Dyadic Conversations

cs.CV cs.SDeess.AS ·2026-10-16 ·Chengxiang Guo, Jian Wang, Lanmiao Liu, Zhan Tang, Junhua Fei +1 more
top venue similarity 0.55
Abstract · accepted by ACM MM 2026

Co-speech gesture generation in dyadic conversations for creating immersive virtual avatars remains highly challenging. Existing models often struggle to balance kinematic stability with rhythmic responsiveness and suffer from semantic entanglement, leading to either generic mean-pose regression or unnatural micro-jitters. To address these challenges, we propose SynerGest, a high-fidelity dyadic gesture generation framework grounded in Contrastive Flow Matching (CFM). Specifically, our CFM objective explicitly penalizes mismatched semantic pairs during training, discouraging spatial shortcuts and over-smoothed predictions while improving conditional semantic alignment. Furthermore, to overcome architectural bottlenecks, we introduce the Impulse-Trend Synergistic Attention (ITSA) module. Grounded in the key insight of decoupling "what moves" from "when it moves," ITSA models sharp, speech-driven rhythmic impulses separately from continuous kinematic trends, supporting precise audio-motion synchronization and global structural stability. Extensive qualitative and quantitative evaluations on two public datasets demonstrate that SynerGest achieves state-of-the-art performance, delivering contextually responsive and kinematically fluid dyadic interactions.

0.33

Comparing Experiences of Speech-Driven Game Modification in Solo Play, Paired Play, and Paired Watching Using the Fighting Game PATCH FIGHTER

cs.HC ·2026-10-19 ·Maya Grace Torii, Takahito Murakami, Yoichi Ochiai
similarity 0.55
Abstract · 17 pages, 5 figures, 4 tables

Players often have ideas about a game while playing it, but turning them into changes usually means stepping out of play to edit it. We built PATCH FIGHTER, a two-player fighting game defined by 91 types of adjustable parameters. A large language model (LLM) reads players' conversations during play and rewrites these parameters between games, so players can change the game just by talking. We conducted a within-subjects experiment with 27 pairs (54 participants) comparing three conditions, solo play, paired play, and paired watching. Participants rated enjoyment after each game and answered a questionnaire on speech and changes after each condition. Paired play was the most enjoyable condition. In paired play, participants also felt more able to say what they thought about the game than in solo play. In interviews, participants valued talking with the partner and checking changes through their own play. Paired watching was less preferred because they could not try the changes themselves. As our design cannot separate the enjoyment of editing from that of play, we outline a design that measures them separately and the evaluation of the resulting games as future work.

0.23