CS Paper Radar

PUBG Ally: A Conversational Embodied Agent as an AI Teammate

PUBG Ally Team, Irene Chen, Youngin Cho, Seungjun Chung, Jimin Hong, Hyeonbin Hwang, Hyeojung Im, Insub Im, Jaeseung Jeon, Seohyeon Jung, Beomsoo Kim, Byeongju Kim, Dohyun Kim, Dongwon Kim, Eunchong Kim, Hongmin Kim, Hyeonghwan Kim, Hyunseung Kim, Sungwoo Kim, Kangwook Lee, Minkyoung Park, Sue Hyun Park, Hyoseok Seol, Yujeong Son, Kiyoon Yoo

cs.AI cs.CLcs.HC ·submitted 2026-09-24 14:06 UTC · v2

Abstract

We introduce PUBG Ally, an embodied agent for PUBG: BATTLEGROUNDS that can reason, act autonomously, and play alongside players as a voice-enabled teammate. Building such a teammate requires combining two difficult capabilities: it must perceive and respond to a constantly changing game world under strict latency constraints while interacting naturally with players, keeping its speech synchronized with its actions. Ally therefore combines agentic tool use with real-time game control. A language-model agent uses a controlled interface to inspect game information, interpret player speech, maintain context, decide what to say, and issue high-level action choices that steer a faster control layer for movement, combat, and recovery. Because the player's and Ally's speech and actions continually shape each other and the course of the match, training requires data from actual gameplay. We therefore collect data across nearly 39k sessions in which real players play alongside Ally, recording gameplay, player speech, agent decisions, tool use, actions, and player feedback, and use these records for iterative training. To evaluate teammate quality, we use player feedback and preference comparisons to identify gaps between offline evaluations and player preferences, and iteratively refine the evaluation criteria. Deploying Ally in live service further requires low-latency on-device execution and safeguards for player-facing communication, which we address through model compression, context compaction, targeted safety training, runtime guardrails, and memory redaction. During the live service, we surveyed players in 141 countries. Among respondents whose play with Ally was confirmed in game records, positive responses exceeded negative responses by 25.1 percentage points when asked whether they would recommend Ally, with players describing Ally not only as a tool but also as a teammate or companion.

Comment
Authors are listed alphabetically. Project leads are Kangwook Lee and Hyunseung Kim

Closest papers

Nearest neighbours by meaning across the whole index.

A study on human-agent teaming through spoken interaction: the impact of human individual traits and agent characteristics

cs.HC cs.AIcs.DB ·2026-10-02 ·Lara Gauder, Martin Bernardo Meza, Javier Krick, Alejandro Masin, Luciana Benotti +2 more
similarity 0.69
Abstract

Human-agent teams are collaborative systems where humans and agents work interdependently to achieve shared goals. The success of such teams is associated with the human's perception of the agent as a legitimate teammate. This perception is thought to depend not only on the agent's capabilities and reliability but also on social factors. We explore whether simple changes in an agent's communicative behavior can influence the human's perception of the agent's teammate-likeness. To this end, we developed a protocol based on a collaborative game requiring spoken interaction, in which a human subject and a virtual agent collaborate to identify a target object and place it on a board. Each subject interacted with two agents which provided the same task-relevant information but differed in behavior. While one was a neutral tool-like agent, the other was a team-building agent that used simple social strategies such as empathy, politeness, and positivity. Our analysis shows that the team-building agent was perceived as significantly more teammate-like, even in terms of its ability, despite both agents having identical capabilities for game-solving. Further, subjects with a higher propensity to trust automated systems and lower neuroticism tended to have a more favorable perception of the agent. Finally, we found that subjects tended to change their prosodic patterns when talking to the team-building agent, as a classifier based on prosodic features was able to predict the type of agent with better-than-random performance. A practically relevant conclusion is that minimal changes in spoken communication behavior of the agent, easily implemented in a variety of scenarios, can have a significant positive impact on the human's perception of its teammate-likeness.

0.18

ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games

cs.MA ·2026-10-16 ·Dawei Wang, Di Zhao, Jianran Yu, Chengming Zhou, Marci Chi Ma +6 more
published similarity 0.69
Abstract · NeurIPS 2026

Large multimodal models (LMMs) are increasingly deployed as agents that perceive, reason, and act in interactive multi-agent environments. Yet most existing benchmarks for LMM-based agents in multi-agent settings remain turn-based: the environment and other agents pause while an agent performs inference. Such protocols obscure a fundamental challenge of real-time constraints, where the world continues to evolve and delayed decisions may become outdated before execution. Motivated by this limitation, we introduce ROAR, a multiplayer action-shooter game environment for evaluating LMM-based agents in multi-agent settings under real-time constraints. ROAR covers cooperative, competitive, mixed cooperative-competitive, and basic single-agent scenarios. We further propose a Multi-Agent Asynchronous Gym interface that decouples environment evolution from agent reasoning by running the environment and each agent in separate threads, enabling the environment to advance at fixed time intervals without waiting for LMM inference and allowing agents to reason independently without blocking one another. ROAR also features convenient deployment, multimodal observations, and customizable levels. ROAR offers a practical testbed for advancing LMM-based agents beyond static, turn-based evaluation toward real-time multi-agent decision-making.

0.48

Exploring Collaboration between a language and a non-language agent

cs.CL cs.AI ·2026-08-31 ·Harini S, Somesh Singh, Yaman K Singla, Rajiv Ratn Shah, David Doermann +1 more
top venuebenchmark similarity 0.66
Abstract · Accepted at EMNLP 2026

LLMs are increasingly deployed as orchestrators that coordinate specialized subagents to solve complex tasks through natural language. However, in many important domains like game playing and robotics, the strongest available agents are not language models. Integrating non-language agents with LLMs would require \emph{verbalization}: compressing their rich continuous representations into sparse textual summaries at each interaction step. To study whether verbalization constitutes a bottleneck, we introduce \textsc{LLAMIA-Bench}, a suite of six diverse collaborative chess tasks spanning three facets: behavioral imitation, state assessment, and natural-language explanation. Each task instantiates a well-established chess problem that neither the LLM nor the chess engine can solve alone. To solve LLM collaboration with non-language agents, we introduce \emph{latent state internalization}, which projects the subagent's continuous representations directly into the LLM's token stream as learned state tokens, with dynamic re-encoding as actions advance the environment state. Comparing internalization to verbalized integration, our experiments reveal a consistent \emph{verbalization debt}: the performance gap widens throughout training and persists as the LLM scales from 4B to 14B parameters. A single 14B model, \textsc{LLAMIA}, trained with latent state internalization, matches or exceeds task specialists and frontier models including GPT-5.1 with tool access across all benchmark tasks, and generalizes out-of-distribution where task-specific finetunes collapse

0.64

Joi-Bench-Task: Benchmarking Task Coordination in Full-duplex Voice Agents through Interactive Simulation

cs.AI ·2026-11-09 ·Xiangheng He, Mingjun Zhao, Zhehao Fan, Zhengjia Li, Xiaopeng Sun +3 more
benchmark similarity 0.66
Abstract

Full-duplex voice agents enable natural, hands-free interaction, allowing users to refine requests, redirect ongoing work, and seek clarification while the agent is speaking. However, existing voice-agent benchmarks insufficiently isolate coordination between task execution and conversation. We introduce a benchmark that assesses this coordination by examining whether agents track what is being executed, what has been completed, what has been communicated, and what remains to be reported as the conversation evolves. We propose an interactive user simulator that introduces configurable interventions at designated timing in tool execution and spoken result delivery, supporting controllable and reproducible evaluation conditions. We evaluate five leading commercial voice agents along three dimensions: execution responsiveness, coordination under request changes, and coordination under interruptions. Agent evaluations reveal persistent coordination failures. Under request changes, the highest result completion rate is only 55%, indicating difficulty for current voice agents adapting to new requests while preserving earlier obligations. After user interruption, four of the five agents exhibit average information omission rates of at least 23%. These findings highlight task coordination as a central challenge for current voice agents in complex, evolving conversations.

0.13

Beyond What to Say: Proactive Utterance Timing with Inner Thoughts for Werewolf Agents

cs.CL ·2026-10-23 ·Yoshiki Tanaka, Shuya Uchiyama, Daichi Hamaguchi, Michimasa Inaba
similarity 0.66
Abstract · Accepted to the 4th International AIWolfDial Workshop at INLG 2026

Participating in a conversation involves not only deciding what to say but also whether to speak at all. This judgment carries even greater significance in environments like the Werewolf game, where participants' interests conflict, because both utterances and silence are subject to interpretation and assessment by others. Therefore, determining when to speak, as well as what to say, is crucial for an LLM agent playing the Werewolf game. In this study, we introduce a Werewolf AI agent developed for the Anytime Utterance Track of the AIWolfDial shared task. Our system builds on the existing Inner Thoughts framework, which determines utterance timing during conversations by generating internal thoughts and evaluating the value of each thought for expression at that moment. Since existing methods were designed for general dialogue, we have redesigned the system specifically for the competitive environment and incomplete information setting of Werewolf games. We call our adapted method Adversarial Inner Thoughts. Through experiments involving analysis of self-play logs and human evaluations, we analyze and discuss the effectiveness of our approach.

0.28

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

cs.AI eess.AS ·2026-09-30 ·Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang
benchmark similarity 0.66
Abstract

Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.

0.12

CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI

cs.AI ·2026-09-02 ·Austin Tudor David Andrews, Liam Wilkinson, Jamie Heagerty, Harry Coppock, Jakob Nicolaus Foerster +1 more
benchmarkcode/data similarity 0.66
Abstract

We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP). A single episode can span over 300 turns and produces thousands of tool calls over a large action space, requiring sustained planning, state monitoring, and execution under partial observability. The environment exposes 76 MCP tools and a deterministic narration layer that retrieves and formats game state as structured text. We use CivBench to characterise agent behaviour across four model families in 23 admissible runs. The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale. Instead, we introduce two interface-level metrics that the environment makes measurable: Proactive Monitoring Rate (PMR), capturing whether agents actively query latent strategic state, and RAG@10, capturing whether commitments stated in structured planning reflections are executed within ten subsequent turns. Across runs we observe two consistent patterns under a shared playbook protocol. Agents under-monitor strategically relevant state that is available but requires explicit querying: despite playbook guidance to query victory progress every 20 turns, agents do so only every 30-75 turns, and in 8 of 20 detectable defeats they failed to query within the 20-turn warning window before game end. Agents also frequently fail to execute near-term commitments stated in their own planning reflections (RAG@10 between 48.2% and 65.8% across models). Both patterns arise despite tool access and explicit guidance, and we interpret them as deviations under instruction rather than absences of capability. We release the environment, scenarios, logs, metrics, and analysis pipeline at https://github.com/lmwilki/civ6-mcp.

0.50

Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction

eess.AS cs.AIcs.CLcs.SD ·2026-09-21 ·Lujia Bao, Qian Chen, Luyao Cheng, Chong Deng, Yuxiang Kong +13 more
similarity 0.65
Abstract · 25 pages, technical report

Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M$^{2}$-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns whether, when, and how the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of $τ$-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.

0.50