CS Paper Radar

CommSketch: How Speaking while Sketching Steers Human--AI Design Ideation

Weiyan Shi, Darryl Lim, Geraldine Quek, Kenny Tsu Wei Choo

cs.HC ·submitted 2026-09-29 15:26 UTC · v1

Abstract

Designers often speak while sketching when explaining ideas, yet AI design tools often rely on sketches or prompts, overlooking context expressed as ideas develop. We developed a sketch-based AI design interface that jointly interprets sketches and concurrent speech. Through a between-subjects study ($N=24$), we examined how speaking while sketching steers human--AI design ideation compared with sketches alone. For creativity support, concurrent speech supported natural expression of design intent and efficient visualisation. For human--AI collaboration, speech helped establish a shared understanding of design intent, supported significantly higher perceived alignment ($p<.05$), and enabled participants to guide AI contributions as ideas co-evolved. We discuss how future human--AI design tools could support dynamic alignment, broader multimodal expression, and human--AI co-creativity.

Comment
work in progress

Closest papers

Nearest neighbours by meaning across the whole index.

DuoSketch: How Pairs Navigate Challenges in AI-Supported Collaborative Design Ideation

cs.HC ·2026-10-07 ·Weiyan Shi, Darryl Lim, Geraldine Quek, Kenny Tsu Wei Choo
similarity 0.75
Abstract · work in progress

Generative AI offers new resources for collaborative design ideation, yet how designers work with it as a shared concept develops remains underexplored. We developed DuoSketch, integrating a shared canvas and live transcripts with a separate AI for each member of a pair. An exploratory qualitative study with 12 pairs identified three challenges: unresolved fit between AI proposals and shared design purposes, shared meanings not expressed in AI responses, and agreed changes missing from later outputs. We found that pairs navigated these challenges through bridging work in two directions: AI -> Shared Design (reworking features, developing uses, and assessing fit with shared requirements) and Shared Design -> AI (supplying materials, explaining meanings, and communicating decisions). We discuss how future AI systems can support the joint exploration and articulation of design ideas.

0.04

Designing UIs with GenAI: Understanding the Use, Benefits, and Limitations of Prompt-to-Design Generators

cs.HC ·2026-11-09 ·Jingzhou Ye, Yao Li, Xueqiang Wang
similarity 0.68
Abstract

The integration of generative AI (GenAI) into user interface (UI) design has led to prompt-to-design generators (P2dGens), tools that enable designers to create UI prototypes from textual prompts efficiently. Despite their growing adoption, little is known about how designers use these tools in everyday practice, their benefits, and limitations. We conducted semi-structured interviews with 17 UI designers to investigate P2dGens use, focusing on workflow integration, attitudes toward generator-assisted design, and the organizational climate. Findings show that designers primarily use P2dGens as ideation accelerators rather than end-to-end solutions, and they adopt a variety of individual and collaborative prompting strategies. Designers expressed complex attitudes, noting gains in efficiency and inspiration while also highlighting concerns about loss of personal creativity and design quality. We also observed emerging governance practices and varying attitudes within organizations toward P2dGens. These insights illuminate how P2dGens reshape UI design workflows and contribute to understanding human-AI collaboration in UI design.

0.27

Creative Intent as an Evolving Shared State: Supporting Cross-Stage Human-AI Video Co-Creation

cs.HC ·2026-11-09 ·Yuhang Jin, Mengfei Chen, Hanwei Zheng, Songruoyao Wu, Kejun Zhang
similarity 0.67
Abstract

As generative AI becomes increasingly involved in video creation, creators face challenges in expanding, translating, and negotiating creative intent across stages. We therefore view creative intent as an evolving shared state throughout human-AI co-creation. A multi-stage design was employed: semi-structured interviews with 12 creators informed 3 design goals, followed by expert co-design and the development of VideoMaker. VideoMaker organizes creation into a six-stage workflow that progressively translates creative intent. At key stages, creators negotiate with AI through the Seed Panel, explicitly confirming, revising, and updating intent. A shared AI Agent, supported by the Multi-style Script Library and Cinematic Language Knowledge Base, enables differentiated expansion of initial intent and faithful translation. A within-subjects study with 30 participants showed that VideoMaker effectively supported the expansion and cross-stage translation of creative intent, with stronger intent preservation, while negotiated boundaries between creator-confirmed intent and AI initiative remained difficult to sustain.

0.06

ReVision: Supporting Designers' Interpretation and Exploration of Visuals in Concepts and Forms

cs.HC ·2026-09-27 ·Yaqing Yang, Mei-Xi Chia, Aniket Kittur
top venue similarity 0.67
Abstract · UIST'26 Poster

Visual designers get inspiration from references to expand their design space. They decompose what makes a reference evocative into conceptual and visual elements, ranging from explicit attributes such as objects and colors, to less readily articulated concepts and visual motifs. They then create different visual forms to explore how the selected elements could be combined differently. Novices often struggle with these moves, instead focusing on surface features or producing limited visual variation, thus becoming fixated on the reference. Existing tools support editable visual attributes and high-level themes, but provide limited control over how conceptual interpretations relate to expressive visual motifs or how their combinations can be systematically re-expressed. We present ReVision, an AI-based tool that decomposes visual and textual references into editable conceptual interpretations and visual motifs, enables their recombination across conceptual and visual spaces, and renders each direction as divergent visual-form variations, supporting more divergent exploration during the creation process.

0.27

Reflecting on Creative-Boundaries with an AI Co-Doodler

cs.HC cs.AIcs.GR ·2026-10-04 ·Samia Menon, Samyukta Jayaram, Chetan Goenka, Shm Garanganao Almeda
similarity 0.67
Abstract · In Proceedings of The First Reflection in Creative Experience (RiCE) Workshop (RiCE W1) arXiv:2607.24558

In this pictorial, we consider how the negotiation of creative boundaries with a co-creative AI system can create moments for personal creative reflection. We ground this in our experiences with Froggi-Draw, a single-initiative co-doodling system that gives users power to decide when and how much an AI "collaborator" (Froggi) contributes to their drawing. From a 1-week pilot study where (n=8) novice and experienced artists doodled daily with the system, we report on ways participants navigated creative risk and uncertainty, and how their usage and perceptions of Froggi shifted over time. In moments of disruption, participants described the system as encroaching on their creative territory. We consider how the design of a supportive, co-creative AI "collaborator" might look like the design of a supportive power dynamic---and finding ways to offer users control to find, reflect upon, and flexibly negotiate the boundaries of that dynamic.

0.12

PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models

cs.AI cs.CLcs.HC ·2026-09-28 ·Shane K. A. Dalumura Hettige, Jonas Oppenlaender
benchmark similarity 0.67
Abstract · 25 pages, 7 figures, 11 tables

Figural divergent thinking is the ability to develop a given shape fragment into an original drawing. In humans, this ability is assessed with incomplete-drawing tasks. We introduce PainterBench, a benchmark that ports the incomplete-drawing task to the agentic setting. The agent draws on a canvas through tool calls and observes the result after every turn. The canvas includes a starting shape which cannot be erased, and the agent's goal is to incorporate this shape into the most original drawing it can produce. The task is open-ended, and the agent itself decides when the drawing is finished. The benchmark tests incremental visual planning over a short horizon and the transfer of creative ability from pretraining to multi-turn tool use. We evaluate 14 multimodal language models from small to frontier scale. Across the primary study and six sensitivity analyses, we collect 2,700 drawings and crowdsource creativity and recognizability ratings for every drawing and for 300 human reference drawings. We also present ViDrA-adapted, an automated scorer that predicts human creativity ratings of agent drawings (r = 0.85 on random held-out test split). Figural divergent thinking varies widely across the 14 models, and GPT-6 Astra produces the most creative drawings. Relative to the human drawings, the agent drawings score higher in creativity but lower in recognizability. We release the final drawings, per-round canvas snapshots, tool call traces, stimulus bank, benchmark harness, crowdsourced ratings (N = 72,000), and ViDrA checkpoint.

0.25

Surprise2Refine: Axis-Centered Exploration-To-Refinement for Agent-Assisted Creative Scaffolding

cs.HC ·2026-08-12 ·Yuzhe You, Gromit Yeuk-Yin Chan, Shunan Guo, Anlan Zhang, Eunyee Koh +2 more
top venue similarity 0.66
Abstract · Accepted to ACM UIST 2026

Designers require different design spaces across creative stages: broad during exploration, and targeted during refinement. Yet existing agent-driven tools assume a fixed or continuously expanding space, leaving designers to manage and navigate it themselves. Informed by a formative study with five designers, we propose an axis-centered workflow that adaptively broadens and narrows the design space to support structured exploration and refinement. We implemented this workflow in Surprise2Refine, a prototype that allows users to build and reshape an nxn design space through a set of axis-centered interactions as their creative intent evolves. A within-subjects study with 14 designers shows that Surprise2Refine enhances users' sense of control, supports tracking of scaffolding paths, and improves the perceived creativity of design outcomes. We further distill design insights to guide future agent-assisted tools for creative scaffolding.

0.34

VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents

cs.HC cs.AI ·2026-10-03 ·Yate Ge, Run Yuan, Yueran Qi, Wenjie He, Jiaqi Mo +5 more
similarity 0.66
Abstract · 22 pages, 11 figures, 5 tables

People write and sketch while speaking to explain, organize, and develop content together. Inspired by these practices, we investigate how voice agents can use a canvas alongside speech in multi-turn conversations with users. We conducted a two-part formative study: an observational study of how pairs coordinated speech and boardwork, followed by a design workshop that informed a design space for speech-canvas interaction with voice agents. Building on these insights, we developed VoCa, a voice agent that coordinates speech with visual object creation, annotation, and attention guidance. A five-day deployment with 18 participants examined usability, experiences of speech-canvas interaction, patterns of use, and desired improvements. Participants' experiences highlighted opportunities for speech-canvas interaction in learning, work, and daily life, alongside challenges in coordinating what agents say and show in ways users can follow and influence. These findings inform how voice agents can use a canvas alongside speech in conversation.

0.36