CS Paper Radar

"Just Like This": Manner Deixis at the Graphics Interface

Hamza El Alaoui, Jeffrey P. Bigham, Jun Rekimoto

cs.HC cs.GR ·submitted 2026-10-06 20:52 UTC · v1
hot direction

Abstract

People communicate how things should move by combining words with demonstrations: "open it like this." We present an interaction technique that brings this expressive resource to conversational 3D authoring. Building on "Put-That-There," our system combines speech, pointing, and spatiotemporal demonstrations to specify editable behavior. A hand movement supplies evidence for a mechanism's axis, pivot, range, and pace; an animated preview makes the interpretation inspectable. Users refine behavior through further words or demonstrations, and the system can request a demonstration to clarify intent. In a twelve-participant study comparing three input configurations, our system achieved 89% participant-declared completion versus 47% with speech alone, had the highest observed match rates on six categorical accuracy measures and tied on two, and was preferred by nine participants. This work makes demonstration part of an ongoing authoring conversation: behavior can be shown, inspected, and revised.

Closest papers

Nearest neighbours by meaning across the whole index.

Say It or Grab It? Conversational Control of UI Adaptations in Mixed Reality

cs.HC ·2026-10-17 ·Sina Elahimanesh, Joao Marcelo Evangelista Belo, Isidora Jeknic, Alexander Koller, Anna Maria Feit
similarity 0.65
Abstract

Mixed reality interfaces must continuously adapt to a user's changing task and environment, yet adaptation policies are fixed in advance, either preset by the system or configured by the user. Natural language offers a way to state changing needs mid-task, but whether users can express interface behavior this way, and what it gains over direct manipulation, is unknown. We first elicited how users verbalize adaptation needs (N=19). We then built ConverseMR, which translates a spoken request into adaptation objectives and constraints and satisfies them continuously as the user's context changes. In a within-subjects study (N=24), conversational and direct control proved complementary: direct manipulation for precision, ConverseMR for reach and expressive breadth, with their combination rated highest. Spoken commands were rarely final and users frequently refined and replaced them upon experiencing the result, characteristic of, what we call, user-led adaptation, where users communicate adaption needs and the system identifies a solution.

0.11

VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents

cs.HC cs.AI ·2026-10-03 ·Yate Ge, Run Yuan, Yueran Qi, Wenjie He, Jiaqi Mo +5 more
similarity 0.63
Abstract · 22 pages, 11 figures, 5 tables

People write and sketch while speaking to explain, organize, and develop content together. Inspired by these practices, we investigate how voice agents can use a canvas alongside speech in multi-turn conversations with users. We conducted a two-part formative study: an observational study of how pairs coordinated speech and boardwork, followed by a design workshop that informed a design space for speech-canvas interaction with voice agents. Building on these insights, we developed VoCa, a voice agent that coordinates speech with visual object creation, annotation, and attention guidance. A five-day deployment with 18 participants examined usability, experiences of speech-canvas interaction, patterns of use, and desired improvements. Participants' experiences highlighted opportunities for speech-canvas interaction in learning, work, and daily life, alongside challenges in coordinating what agents say and show in ways users can follow and influence. These findings inform how voice agents can use a canvas alongside speech in conversation.

0.36

Evaluation-Guided Demonstration Refinement for Vision-Language-Action Policies: The Winning Entry in the MetaFine Open Competition

cs.CV ·2026-10-27 ·Lifan Zheng, Guotao Chen, Zhenyang Li, Shujie Cui, Qiang Huang +1 more
similarity 0.62
Abstract · 1st place in the MetaFine Open Competition, RoboFineMani Workshop @ CoRL 2026

We fine-tuned a separate $π_{0.5}$ policy for each of the five tasks in the MetaFine Open Competition, with a task-specific training mixture and number of actions executed per prediction. Training started from the official demonstrations and expert procedures. Local evaluation exposed recurring grasping, pushing, and insertion failures. We addressed them by revising the demonstrated motions and by adding expert continuations from states the policy reached, while keeping nominal demonstrations in training. Checkpoints were compared on every instruction, so that a gain on one instruction could not hide a loss on another. Our submission, MAI, ranked first overall with a score of 96.66.

0.13

DiaHeads: Jointly Generating Coordinated 3D Dialogue Heads with Non-verbal Cues from Dual Audio

cs.MM ·2026-11-04 ·Jiahe Wang, Yuxin Wang, Xin Yun, Binyan Chen, Tong Pan +2 more
similarity 0.62
Abstract

Audio-driven 3D head animation has made considerable progress in generation realism, yet generating coordinated behavior for two-person dialogue remains difficult, because current dialogue head generation methods overlook the modeling of non-verbal behavior semantics between the two participants. To address this limitation, we present DiaHeads, a framework that generates the behavior of both participants jointly. The two audio streams are concatenated and decoded together, so each participant's motion is conditioned on the other, which makes the interaction part of the generation itself. We further introduce four discrete non-verbal cues, gaze-on, head shaking, nodding, and smiling, and fuse them with the continuous parameters in a shared latent space through a product-of-experts posterior. These cues carry the interaction semantics and constrain how the continuous motion is generated. A flow-based prior over the shared latent variable replaces the standard Gaussian prior and makes the latent motion temporally dependent. These designs therefore yield coordinated 3D dialogue heads. To support this task we build CANDOR-FG, a dialogue corpus derived from CANDOR and augmented with synchronized FLAME parameters and 3D gaze for both interlocutors. Experiments on CANDOR-FG show that DiaHeads achieves competitive per-participant fidelity and substantial gains in interpersonal coordination over single-speaker and dyadic baselines. In addition, ablations confirm that joint generation, the flow-based prior, and multi-cue supervision are the sources of this coordination. Finally, a user study shows that the advantage of DiaHeads comes from perceived interactional coordination rather than from raw motion fidelity alone.

0.09

Souffleur: A Glanceable Display for Mobile Voice Conversations

cs.HC ·2026-11-05 ·Anton Wolter, William Benjamin, Niklas Elmqvist
similarity 0.62
Abstract

The speech capabilities of mobile devices have grown, but their displays still rely on a text-based chat. When resuming a voice chat after a context switch, the last messages may not convey the key points of the conversation. Autobiographical use and sessions with external participants led to design goals for a mobile-first, speech-driven, glanceable display that helps users continue their reasoning from where they left off. We implement these goals in Souffleur and evaluate it in a between-subjects study with 48 participants. Compared with a conventional chat transcript, we found no reliable difference in how easily participants followed and resumed short voice conversations, nor in effort, attention or recall, although Souffleur reduces the conversation to a few short phrases.

0.09

A Benchmark for Spatially Grounded Gesture Generation

cs.CV cs.GRcs.HC ·2026-10-02 ·Anna Deichler, Rishabh Dabral, Fethiye Irmak Dogan, Anindita Ghosh, Jonas Beskow
publishedbenchmarkcode/data similarity 0.61
Abstract · 13 pages, 9 figures. Benchmark of the Referential Gesture Challenge at the HSI Workshop, ECCV 2026. Data and video: https://huggingface.co/d…

Communication in shared space interweaves verbal and non-verbal signals, and pointing gestures anchor language to the environment: "put the cup on that one" is uninterpretable without the gesture that fixes the referent. Yet no common framework exists for evaluating whether generated gestures indicate their intended referent; distributional metrics reward a gesture aimed at the wrong object as long as it looks natural. We introduce a benchmark for spatially grounded gesture generation, comprising ~2K pointing-annotated clips from naturalistic VR dialogue with ground-truth 3D referents, a task in which systems must decide when, how and where to point within conversational speech, and a protocol that separates temporal alignment, spatial grounding and perceived naturalness. We also provide a flow-matching baseline, MM-Conv-Flow. Evaluating it alongside an independent retrieval-based system and captured human motion, we find that geometric grounding can exceed that of human pointing without any gain in perceived naturalness, showing that referential gesture quality must be measured along separate dimensions.

0.53

DimSteer: Steering LLM Authoring with Automatically Discovered Stylistic Controls

cs.HC cs.CL ·2026-10-03 ·Ajit Mallavarapu, Ziwei Gu
similarity 0.61
Abstract · 24 pages, 5 figures

Large language model writing interfaces often make users steer outputs by repeatedly articulating desired changes in natural language. Yet writers may recognize useful stylistic directions only after seeing alternatives, making revision recall-heavy. We present DimSteer, an authoring interface that samples prompt-local completions, discovers high-variance activation-space axes of variation, labels them, and exposes them as sliders with pole previews, diff comparison, and reset controls. Users can manipulate discovered dimensions, reducing the need to reformulate prompts for each stylistic adjustment. In a within-subjects study with 16 participants against a matched prompt-only baseline, DimSteer reduced mental demand, effort, and frustration while preserving comparable perceived success. Participants valued the surfaced dimensions, yet 15 of 16 disagreed that they would have thought to request the same changes in a prompt. Results suggest prompt-local controls can shift LLM authoring from recall-based prompting toward recognition-based exploration and direct manipulation, while preserving prompting for open-ended edits.

0.30

PolyPresentation: A Multimodal AI Platform for Slide-Aware Iterative Presentation Practice

cs.HC ·2026-08-13 ·Chen Chen, Jihao Li, Zhiyuan Wen, Tianhui Zhang, Di Zou +1 more
similarity 0.61
Abstract

Presentations are essential for students, researchers, and professionals to communicate ideas persuasively, yet delivering them effectively requires repeated practice that coordinates content, delivery, visual materials, and audience interaction. Existing AI-assisted rehearsal tools provide scalable feedback, but they often treat presentations as single-run delivery performances, offering limited support for linking feedback to the slide deck or planning what to practice in the next iteration. To address this gap, we introduce PolyPresentation, a multimodal AI platform for slide-aware iterative presentation practice. PolyPresentation organizes slide-by-slide practice, full rehearsal, audience Q&A, and feedback into a unified practice loop, using slide-grounded evidence to help presenters diagnose performance issues and prepare for subsequent practice. We evaluate PolyPresentation through a rubric-based comparison with four baseline systems on 20 academic presentation rehearsals, and additionally assess its alignment with human ratings. Results suggest that PolyPresentation provides more actionable, context-aware, and practice-oriented support for improving presentations. The demonstration video is available at https://youtu.be/MmWj9O_PJxw.

0.16