CS Paper Radar

VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents

Yate Ge, Run Yuan, Yueran Qi, Wenjie He, Jiaqi Mo, Yangshuo Chen, Wenbin Zuo, Xiaohua Sun, Weiwei Guo, Qi Wang

cs.HC cs.AI ·submitted 2026-10-03 18:39 UTC · v1

Abstract

People write and sketch while speaking to explain, organize, and develop content together. Inspired by these practices, we investigate how voice agents can use a canvas alongside speech in multi-turn conversations with users. We conducted a two-part formative study: an observational study of how pairs coordinated speech and boardwork, followed by a design workshop that informed a design space for speech-canvas interaction with voice agents. Building on these insights, we developed VoCa, a voice agent that coordinates speech with visual object creation, annotation, and attention guidance. A five-day deployment with 18 participants examined usability, experiences of speech-canvas interaction, patterns of use, and desired improvements. Participants' experiences highlighted opportunities for speech-canvas interaction in learning, work, and daily life, alongside challenges in coordinating what agents say and show in ways users can follow and influence. These findings inform how voice agents can use a canvas alongside speech in conversation.

Comment
22 pages, 11 figures, 5 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Souffleur: A Glanceable Display for Mobile Voice Conversations

cs.HC ·2026-11-05 ·Anton Wolter, William Benjamin, Niklas Elmqvist
similarity 0.68
Abstract

The speech capabilities of mobile devices have grown, but their displays still rely on a text-based chat. When resuming a voice chat after a context switch, the last messages may not convey the key points of the conversation. Autobiographical use and sessions with external participants led to design goals for a mobile-first, speech-driven, glanceable display that helps users continue their reasoning from where they left off. We implement these goals in Souffleur and evaluate it in a between-subjects study with 48 participants. Compared with a conventional chat transcript, we found no reliable difference in how easily participants followed and resumed short voice conversations, nor in effort, attention or recall, although Souffleur reduces the conversation to a few short phrases.

0.09

CommSketch: How Speaking while Sketching Steers Human--AI Design Ideation

cs.HC ·2026-09-29 ·Weiyan Shi, Darryl Lim, Geraldine Quek, Kenny Tsu Wei Choo
similarity 0.66
Abstract · work in progress

Designers often speak while sketching when explaining ideas, yet AI design tools often rely on sketches or prompts, overlooking context expressed as ideas develop. We developed a sketch-based AI design interface that jointly interprets sketches and concurrent speech. Through a between-subjects study ($N=24$), we examined how speaking while sketching steers human--AI design ideation compared with sketches alone. For creativity support, concurrent speech supported natural expression of design intent and efficient visualisation. For human--AI collaboration, speech helped establish a shared understanding of design intent, supported significantly higher perceived alignment ($p<.05$), and enabled participants to guide AI contributions as ideas co-evolved. We discuss how future human--AI design tools could support dynamic alignment, broader multimodal expression, and human--AI co-creativity.

0.20

ABRA: Agent-Bound UI for Human-Agent Communication beyond Chat with Reusable Views

cs.HC cs.MA ·2026-10-22 ·Seon Gyeom Kim, Hannah Kim
top venuecode/data similarity 0.65
Abstract · EMNLP 2026 System Demonstrations

Agents can create rich representations, yet these representations are rarely used as a medium for expressing user intent. We present Agent-Bound UI, an interaction model that separates reusable interaction structures from task-specific meanings. Agents bind task-specific meanings to interactions in reusable views and publish the resulting bindings as shared state, enabling humans and agents to understand and act on shared surfaces whose meanings vary across tasks. We implement ABRA, a framework-agnostic prototype of Agent-Bound UI, in which users and agents collaborate through chat and shared interactive canvases. We demonstrate ABRA through a representative use case and evaluate four LLM-backed agents on canvas authoring, owner-side consequence resolution, and cross-agent invocation. The code and demo video are available at https://github.com/megagonlabs/abra.

0.56

Inkitiate: Initiating and Steering Agent Assistance through Ink in a Spatial Workspace

cs.HC ·2026-10-29 ·Liangwei Wang, Nan Chen, Luna K. Qiu, Zilong Wang, Fugee Tsung +2 more
similarity 0.64
Abstract · 18 pages, 11 figures

Handwriting and sketching help people develop ideas around information. As this work unfolds, users may seek assistance with finding, understanding, and organizing materials. We investigate how ink can initiate and steer agent assistance in this process. An eight-participant formative study explored desired assistance and interaction expectations, informing three connected activities: bringing materials in, thinking with materials, and shaping the space. We present Inkitiate, a spatial workspace for ink-based agent assistance in knowledge work. Through ink, users retrieve external materials, build on accumulated content and results, and reorganize the canvas, while inspecting and guiding agent actions. In a study with eighteen participants, users reported favorable usability and low perceived workload, and used retained ink and materials to develop further ink queries. An ink-query vocabulary and cases reveal how users combine linguistic and graphical expressions to direct agents. These findings inform ink-based agent interaction design for expressive requests and continued work with materials.

0.31

From Artifacts to Coordination: Exploring How Shared Canvases Drive Collaboration during Group Ideation

cs.HC ·2026-10-29 ·Saramsh Gautam, Andrew M. Webb, Mahmood Jasim
published similarity 0.63
Abstract · Accepted to ACM GROUP 2027. To appear in Proc. ACM Hum.-Comput. Interact. 11, GROUP (January 2027). 21 pages, 9 figures. DOI: 10.1145/385940…

Recent years have seen a surge in tools that support collaborative ideation through shared canvases. While these canvases offer flexible workspaces, how collaborators leverage them to iteratively develop ideas by externalizing thoughts, organizing artifacts, and coordinating tasks remains underexplored. To address this gap, we conducted an exploratory study with seven triads planning a travel itinerary using an infinite-canvas technology probe. Drawing on participant interviews and video analysis of their canvas interactions, we observed that collaborators contributed in parallel and arranged artifacts freely without a predefined structure. This spatial organization allowed collaborators to coordinate implicitly, made less vocal participants' contributions visible, helped distinguish private exploratory regions from shared working areas, and supported the emergence of distinct roles through repeated patterns of contribution. We discuss design implications for canvas-based ideation tools, including surfacing unnoticed contributions, supporting emergent roles without formalizing assignments, and keeping spatial cues legible for AI integration in collaborative canvases.

0.45

"Just Like This": Manner Deixis at the Graphics Interface

cs.HC cs.GR ·2026-10-06 ·Hamza El Alaoui, Jeffrey P. Bigham, Jun Rekimoto
hot direction similarity 0.63
Abstract

People communicate how things should move by combining words with demonstrations: "open it like this." We present an interaction technique that brings this expressive resource to conversational 3D authoring. Building on "Put-That-There," our system combines speech, pointing, and spatiotemporal demonstrations to specify editable behavior. A hand movement supplies evidence for a mechanism's axis, pivot, range, and pace; an animated preview makes the interpretation inspectable. Users refine behavior through further words or demonstrations, and the system can request a demonstration to clarify intent. In a twelve-participant study comparing three input configurations, our system achieved 89% participant-declared completion versus 47% with speech alone, had the highest observed match rates on six categorical accuracy measures and tied on two, and was preferred by nine participants. This work makes demonstration part of an ongoing authoring conversation: behavior can be shown, inspected, and revised.

0.32

Seeing the Voice, Preserving the Self: A Participatory Design Approach to Deaf-Centric Text-to-Speech

cs.HC ·2026-09-09 ·Shela Atemnkeng, Patrick Boudreault, Paige DeVries, Lloyd May, Christian Vogler
top venue similarity 0.62
Abstract · Accepted for publication at ACM ASSETS 2026

We describe a participatory design approach toward developing Deaf-centric text-to-speech (TTS) technologies. While TTS is growing rapidly in the mainstream, it has received little attention to date in the deaf and hard of hearing (DHH) technology space. Critical problems have remained unaddressed for DHH users, including the ability to manipulate tone, emotions and delivery via non-auditory means. Verifying that the generated speech matches intent and is appropriate for a given situation without having to listen to it is another challenge. Respecting cultural and identity factors in the generated speech is also important. This work explores the design space with DHH participants through two focus groups, three co-design sessions, and four one-on-one early-stage design evaluation sessions. Participants included people both familiar and unfamiliar with TTS, as well as DHH content creators. We describe key findings, design ideas, results, and implications for future Deaf-centric TTS development. We also identify unmet technology requirements that pose barriers to adoption of Deaf-centric TTS technology.

0.46

Agentic-DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech

cs.CL eess.AS ·2026-08-17 ·Pengcheng Wang, Sheng Li, Jiyi Li, Takahiro Shinozaki
similarity 0.62
Abstract

Synthetic conversational speech has become an important resource for developing and evaluating conversational speech systems. However, existing dialogue synthesis pipelines typically generate dialogue content first and then insert interruptions, overlap, and backchannels using handcrafted markers or timing rules, making conversational timing prescribed rather than interaction-driven. We present Agentic-DuplexGen, a dialogue synthesis framework that explicitly decouples content, timing, and acoustics. An LLM first generates the dialogue script, and then two full-duplex conversational models perform the script while listening to each other in real time. This allows conversational timing to emerge naturally while preserving the scripted content. Finally, a high-fidelity text-to-speech model re-renders the interaction without altering its timing. As a demonstration of the proposed framework, we construct a patient--clinician conversational speech corpus with construction-time annotations, including word timestamps, speaker activity, overlap regions, and interaction events. Experimental results show that the proposed framework produces conversational dynamics closer to real dialogue than conventional stitching-based synthesis.

0.26