CS Paper Radar

Human-robot conversation with multiple participants in noisy public spaces

Divesh Lala, Yogeeswaran Muthukumaran, Vincent Fernandes, Kazushi Kato, Shota Fujiki, Zihao Chi, Masaya Iwasaki, Taiken Shintani, Megumi Kawata, Kazuki Sakai, Koji Inoue, Yuicihiro Yoshikawa, Tatsuya Kawahara

cs.RO cs.HC ·submitted 2026-09-01 03:25 UTC · v1

Abstract

For noisy real-world environments such as those in open public spaces, spoken dialogue systems for both autonomous robots and avatars should be carefully designed to provide enhanced speech signals. These signals can be used either for speech recognition or, in the case of an avatar system, transmitted as clean speech to a remote operator. This work proposes an audio system that can be used for both these scenarios and was demonstrated as a proof-of-concept at the 2025 World Expo in Osaka. The first scenario is an attentive listening system with the android ERICA, and the second is a conversation support system with mobile Teleco robots, with one of them acting as an avatar for a remote operator. Both systems feature multi-party conversation and use a single multi-channel microphone array. We describe how our audio system not only enhances the speech of multiple speakers in a noisy environment, but provides a form of spatial audio which allows for more immersiveness in avatar-based conversational interactions.

Closest papers

Nearest neighbours by meaning across the whole index.

Towards an Extensible Benchmark for Spoken Dialogue with Social Robots

cs.RO ·2026-10-06 ·Casey Kennington, Ross Mead, Saad Elbeleidy, Jesse Thomason
publishedbenchmark similarity 0.66
Abstract · Accepted to and presented at the IROS 2026 Workshop on Human-Robot Dialogue

Language models provide a plug-and-play interface between humans and robots, but important challenges remain when speech, dialogue, fast interaction, and collaboration are required. We propose a benchmark for the community to use as a way to explore common spoken dialogue artifacts between robots and humans, including requests for clarification, interruptions, embodied signals (e.g., head nods or facial cues), and time constraints. We also explain our vision to extend the benchmark for other aspects of human-robot interaction that are important to the larger research community. To facilitate the benchmark, we further propose using \textit{Retico}, a real-time communication framework that fulfills important technical requirements to enable robots to have spoken dialogue capabilities.

0.20

Multimodal Socially Aware Framework for Human-Robot Interaction in Virtual Reality

cs.RO ·2026-10-25 ·Richard Xu, Dakarai Crowder, Alexis E. Block, Wenzhen Yuan
similarity 0.66
Abstract

Social robots must coordinate speech and movement while interpreting multimodal human input. We present a multimodal framework that integrates speech, vision, and touch to generate contextually appropriate robot speech and movement. We evaluated the framework in virtual reality with 18 participants across five social interaction scenarios, comparing its performance against a human-teleoperated baseline. Although participants rated the autonomous system above the neutral midpoint for both naturalness and social appropriateness, the teleoperated baseline received significantly higher ratings on both measures. Behavioral coding and qualitative feedback revealed that conversational timing, clarification of ambiguous requests, and spatial coordination were important factors contributing to this performance gap. These findings highlight key challenges in developing autonomous robots capable of natural, multimodal social interaction.

0.18

MIRA: Real-Time Full-Duplex Human-Robot Interaction for Embodied Companions

cs.RO ·2026-09-21 ·Lijian Lin, Ye Zhu, Fan Zhang, Yunfei Liu, Baofeng Li +3 more
similarity 0.65
Abstract

% !TEX root = ../main.tex Real-time embodied companion interaction requires a robot to infer user intent from streaming speech, generate timely responses, and execute expressive, interruptible motions. Existing systems typically decouple dialogue orchestration from gesture synthesis, relying on offline motion generation from complete audio. This separation leaves open how a deployed robot can dynamically synchronize response content, prosodic timing, and physical safety under incremental inputs and uncertain turn boundaries. We present MIRA, a unified framework for real-time full-duplex embodied companion interaction. Given streaming user speech, dialogue history, and vocal affect, MIRA predicts both the response text and an explicit embodiment cue that routes the response to the appropriate physical behavior. Discrete social behaviors (\eg listening and greeting) are mapped to validated robot trajectories, while speaking responses are accompanied by streaming, generative co-speech motion. For co-speech motion generation, we propose ROSCO, a prefix-conditioned diffusion model for streaming audio-to-joint motion generation. We further design RHPC, an inference scheme that maintains a sufficiently long temporal context for motion prediction while bounding physical commitment to a short, interruptible prefix. At the interaction level, we design CORTEX, a dual-timescale interaction policy that combines low-latency barge-in preemption and streaming response generation with deliberative turn decisions, backed by a robot-side execution layer that enforces physical safety constraints during execution. MIRA is deployed on an Astribot S1 humanoid robot. Quantitative evaluations demonstrate competitive audio-motion alignment relative to state-of-the-art motion-generation baselines, while real-robot deployment measurements characterize streaming responsiveness and interruption handling.

0.17

KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction

cs.SD cs.CL ·2026-09-07 ·Ryuichiro Higashinaka, Shinnosuke Takamichi, Tetsuji Ogawa
published similarity 0.63
Abstract · Accepted at APSIPA ASC 2026

Realizing full-duplex spoken dialogue requires large amounts of two-channel, one-speaker-per-channel conversational speech data. Although conversational text-to-speech (TTS) engines have been developed, they are not necessarily robust to two-party simultaneous phenomena such as backchannels, interruptions, and overlaps that occur while the interlocutor is speaking. In this work, aiming at conversational speech synthesis that reproduces human-like overlap, we propose KABURI-TTS. KABURI-TTS takes a per-speaker phoneme raster as input and renders the speech of the two speakers on separate channels, conditioned on the per-frame phonemes and the voice activity derived from them. Because the phoneme raster is supplied by a separate module, the proposed method enables controllable generation of one-speaker-per-channel, two-party spoken dialogue. A user evaluation shows that, compared with strong baselines, the proposed method attains higher naturalness at both the utterance and the interaction level. Furthermore, an analysis of voice activity confirms that the proposed method produces more overlap and more frequent turn-taking.

0.32

Robots That Talk Too Much and Don't Listen Enough

cs.RO cs.AIcs.HC ·2026-11-12 ·Graham Wilcock, Kristiina Jokinen
published similarity 0.63
Abstract · 4 pages, 3 figures, presented at EMF Workshop, HRI 2026, Edinburgh, Scotland

Until recently, the biggest obstacles to natural spoken interaction with social robots were speech recognition errors and turn-taking failures. Advances in speech recognition have enabled much better spoken interaction, but turn-taking failures and other misunderstandings continue to be far too prevalent. This short paper looks back at earlier methods for improving speech recognition and turn-taking in interaction with social robots, and then looks at what can be done today, arguing that multimodal bidirectional streaming is needed to reduce the frequency of turn-taking failures.

0.35

Multimodal Voice Activity Projection for Social Robot Mediation: Expected Behavior and Deployment Constraints

cs.RO ·2026-09-23 ·Antonio Cano, Guillermo Perez, Luis Merino, Randy Gomez
similarity 0.61
Abstract · Presented at IEEE RO-MAN 2026 at 3rd Workshop on Nonverbal Cues for Human-Robot Cooperative Intelligence (NOC) - Best Workshop Paper Award

Turn-taking prediction is especially relevant for social robots that act as mediators in human-human interaction, where the expected action is often not to speak, but to orient, wait, avoid interruption, or prepare a balanced intervention. This paper presents Multimodal Voice Activity Projection (MM-VAP) as a human state-aware perception layer for future robot mediation behavior. The model estimates the future evolution of the conversational floor from synchronized audio-visual evidence and derives turn-taking events such as Hold, Shift, Shift prediction, Backchannel prediction, and overlap-related states. The approach uses VA-related pretrained audio-visual encoders, LoRA adaptation, inter-speaker attention, and zero-shot event inference from future voice activity projections. Experiments on NoXi, NoXi+J, and Haru EDR support the feasibility of this formulation, especially for floor management events that can be connected to gaze preparation, active listening, and conservative intervention. Finally, the paper defines the expected robot output interface and discusses the main deployment constraints, including real-time inference, preprocessing latency, multimodal synchronization, and input-quality monitoring.

0.26

Industry case studies of adapting TTS for smooth interactions

cs.HC cs.ROcs.SDeess.AS ·2026-08-26 ·Paige Tuttösi, Eléonore Ferrier-Barbut, Anais Scipioni, Olivia Yung, Ha Eun Shim +3 more
published similarity 0.61
Abstract · Published at RO-MAN 2026

Modern, neural-based ASR and TTS systems for speech communication are now very sophisticated, but they are frequently designed for average users in more idealized scenarios, which can underperform when interacting with users having specialized listening needs, or who speak in non-standard ways. Here we review recent TTS research, and note that industrial, production-level implementations of adaptive TTS systems are scarce. We describe how we implemented changes to TTS systems based on prior research, and then report qualitative results from two case studies, where adapted TTS techniques are applied to real-world HRI settings: First, in an older adult care home, and second, in an airport, where noise levels create comprehension problems. Results suggest promising directions using relatively simple modifications to pitch, duration, and intensity of the robot's speech, and offer a framework for larger, more formal studies of how TTS adaptations can improve HRI.

0.33

SciPhi-Chat: Training a Spatial Audio Agent for Question Answering, Dialogue, and Tool Use

cs.SD ·2026-11-03 ·Gijs Wijngaard, Hannes Gamper, Sebastian Braun
benchmarkcode/data similarity 0.60
Abstract

Spatial audio question answering requires associating sound identities with locations and reasoning about their relations. This paper introduces SciPhi-Chat, a spatial audio agent that answers questions, conducts multi-turn dialogue, and calls tools on first-order ambisonic recordings. The accompanying MASU benchmark contains 12,000 samples across 42 task categories, the widest task coverage of any spatial model, with real recordings and simulated audio scenes. SciPhi-Chat achieves a score of 44.9%, 7.4 percentage points above the previous state-of-the-art. On the external ASA-Bench, SciPhi-Chat achieves 74.5% azimuth and 60.0% elevation accuracy, the highest among locally evaluated models. Removing reasoning supervision reduces ASA-Bench accuracy by 17.4 percentage points. All materials will be released via https://github.com/microsoft/sciphi-chat

0.21