CS Paper Radar

"I'm Very Happy for It to Start Hallucinating a Little Bit": Using ClayFlect to Negotiate Multimodal AI Representations in Material Meaning-Making

Kellie Yu Hui Sim, Quoc-Nam Nguyen, Shuenn Yuen Han, Kenny Tsu Wei Choo

cs.HC ·submitted 2026-10-06 18:06 UTC · v1

Abstract

As AI enters reflection and emotional support, understanding how it can participate in personal meaning-making while preserving users' authority over interpretation is increasingly important. We present ClayFlect, a novel MLLM-powered system integrating tactile clay-making with conversational and visual generative AI, and report a mixed-methods study with 50 participants. Reflection developed across material, conversational, and generated forms rather than through AI interaction alone. Participants treated AI representations as provisional: they compared, redirected, reinterpreted, selectively incorporated, or left them aside as their artefacts and meanings evolved. Clay provided a directly manipulable space in which participants could continue developing meaning independently of the AI, while generated representations externalised possibilities beyond what they could readily make or visualise. We show how generative AI can participate through representations that remain negotiable, and derive implications for supporting movement across representations, preserving parallel sites of control, and allowing AI support to recede or deepen as reflection unfolds.

Comment
32 pages, 16 figures, 4 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Exploring the Affordances of Generative Image AI for Supporting Early-stage Architect-client Communication

cs.HC ·2026-09-28 ·Chengzhi Zhang, Weijie Wang
similarity 0.68
Abstract · 18 pages

Text-to-image generative AI can produce renderings from natural-language prompts in near real time, making it increasingly popular for rapidly visualizing concepts in early-stage architectural design. Meanwhile, exchanging ideas efficiently and building shared understanding have long been central challenges in architect-client communication. How might the speed of generative image AI change this communication? To explore this question, we conducted a study with 11 architect-client pairs, in which each pair used generative image AI over video conference to collaboratively produce early-stage renderings of the client's "dream house." Our findings suggest that generative image AI helped pairs develop a solid shared understanding by providing concrete visual materials and supporting the exchange of ideas. It also shifted conversation dynamics, enabling clients to participate more actively in shaping design direction. However, challenges emerged, including a stylistic bias toward particular types of images and unpredictable shifts in design direction caused by variation across generations. We conclude with implications for the design of future generative image AI-based systems that support architect-client communication.

0.13

OmniScribble: Mixed-Initiative Authoring of Animated Sketches through Inspectable and Repairable AI Decisions

cs.HC ·2026-11-03 ·Hien Trinh, Tan-Hiep To, Trung-Nghia Le
similarity 0.66
Abstract

AI-assisted creative tools reduce the effort of authoring content, but delegating creative decisions to AI can make it difficult for authors to understand, correct, or redirect work. We present OmniScribble, a system for animating multi-object sketches that treats object decomposition and motion planning as editable artifacts. Rather than generating animation end-to-end, OmniScribble presents AI proposals at verification checkpoints, where authors can accept, repair, or regenerate them. Both manual and AI-assisted authoring operate on the intermediate representation and rendering pipeline, allowing us to isolate AI delegation. In a within-subject study with eight participants, AI-assisted authoring yielded a median within-participant reduction in completion time of 71.4\%, while perceived control and motion-intent alignment remained high. Interaction traces and interviews show that participants adapted to AI errors by repairing local issues, regenerating mismatches, and accepting proposals. These findings suggest that inspectable, repairable delegation enables generative automation while maintaining control over the outcome.

0.08

Sensemaking as Artifact: Accumulated Influence in AI-Mediated Information Environments

cs.HC ·2026-09-14 ·Manling Yang, Remco Chang
publishedposition similarity 0.65
Abstract · 5 pages, 3 figures. Accepted at the CSCW 2026 Workshop on Broader Impacts of GenAI in Communication

Generative AI is changing what can happen after a source artifact reaches its audience. A viewer's interpretation can now be externalized into a derivative artifact, allowing private sensemaking to become part of subsequent communication. Once such a derivative artifact circulates, it can enter subsequent viewers' information environments and shape the conditions under which their later sensemaking occurs. In this paper, we examine how this shift changes visual information communication. We first consider the viewer's immediate interaction with a source artifact and generative AI. We then examine what becomes consequential when the viewer's sensemaking takes communicative form, including communicative commitment, the legibility of transformations and source relationships, and the literacy required to interpret already-mediated information. Finally, we broaden the unit of analysis to consider how repeated and distributed AI mediation may accumulate over time, shaping what subsequent viewers notice, consider plausible, trust, and carry into subsequent sensemaking. We argue that understanding these longer-term forms of influence is a research direction for AI-mediated visual communication.

0.27

Reflecting on Creative-Boundaries with an AI Co-Doodler

cs.HC cs.AIcs.GR ·2026-10-04 ·Samia Menon, Samyukta Jayaram, Chetan Goenka, Shm Garanganao Almeda
similarity 0.65
Abstract · In Proceedings of The First Reflection in Creative Experience (RiCE) Workshop (RiCE W1) arXiv:2607.24558

In this pictorial, we consider how the negotiation of creative boundaries with a co-creative AI system can create moments for personal creative reflection. We ground this in our experiences with Froggi-Draw, a single-initiative co-doodling system that gives users power to decide when and how much an AI "collaborator" (Froggi) contributes to their drawing. From a 1-week pilot study where (n=8) novice and experienced artists doodled daily with the system, we report on ways participants navigated creative risk and uncertainty, and how their usage and perceptions of Froggi shifted over time. In moments of disruption, participants described the system as encroaching on their creative territory. We consider how the design of a supportive, co-creative AI "collaborator" might look like the design of a supportive power dynamic---and finding ways to offer users control to find, reflect upon, and flexibly negotiate the boundaries of that dynamic.

0.12

More Than Just Access: Generative AI as Communication Intermediary for Blind and Low-Vision Users

cs.HC cs.AIcs.ET ·2026-09-14 ·Protik Dey, Mohd Saifuzzaman, Taslima Akter
published similarity 0.64
Abstract · Proceedings of the CSCW 2026 Workshop on Broader Impacts of GenAI in Communication:Building Agendas for Research, Design, and Policymaking, …

Generative AI (GenAI) tools are increasingly woven into how blind and low-vision (BLV) people communicate, not only with digital information, but with the physical world and with other people. Tools such as ChatGPT, Google Gemini, Be My AI, and Seeing AI translate visual and textual content into accessible form, and are beginning to substitute for interpersonal requests for help, such as asking a family member to read a label or describe a scene. Drawing on semi-structured interviews with 19 BLV participants, we examine GenAI as a communication intermediary and how it succeeds and fails as an alternative for reading, describing, and even asking another person for help. We also investigated what BLV users gain and risk when these tools take over that role. We conclude with design and policy implications for GenAI systems that communicate uncertainty honestly, protect information, and support BLV users' independence rather than substitute for it unsafely.

0.38

From Semantic Editing Intentions to Persistent Parametric Controls for AI-Assisted Image Editing

cs.HC ·2026-10-14 ·Rifat Mehreen Amin, Michelle Dutoit, Anton Rockenstein, Baptiste Caramiaux, Andreas Butz
similarity 0.64
Abstract

AI-powered image editing increasingly allows users to express high-level editing intentions, yet how these intentions are translated into concrete editing actions often remains difficult to understand. We present Atelier, a semantic image editing tool that uses image semantics and user intent to determine which editing controls to surface and how to initialize them. Guided by a formative study with eight image-editing practitioners, Atelier composes context-specific, directly manipulable controls that users can refine rather than committing AI interpretations directly to finished edits. In a within-subjects study with 12 practitioners, participants preferred AI-composed controls, described them as supporting creative exploration, and actively refined AI proposals. However, experiences of control were mixed, and interviews highlighted limited insight into why particular controls and values were proposed. Our findings suggest that making AI interpretations actionable through manipulable controls can support exploration and refinement, but actionability alone does not make those interpretations transparent to all users.

0.25

Controlling Proactive Multimodal AI: Separating Perception from Participation

cs.HC ·2026-11-17 ·Muawiz Feroze Khan, Fatima Ali, Yun Huang
similarity 0.64
Abstract · 30 pages, 4 figures, 2 tables

Proactive multimodal AI can watch a user's screen, listen to speech, track interaction, and decide whether to enter unfolding work. Little is known about how people want to control what such AI may perceive and how far it may participate. Informed by a 14-person Wizard-of-Oz study, we built SpotMe, a design probe that senses screen, speech, and interaction; chooses Silence, Ask, or Assist; and lets users separately configure what it may perceive, how far it may participate, and what types of support it may provide. In a 16-person study on a civic dashboard, participants first experienced proactive help, then inspected decisions to act or remain silent, and then configured SpotMe's perception and participation. We found that participants could keep all perception channels available while lowering participation to Ask or Silence; that exposing suppressed decisions made silence interpretable and contestable; and that seeing what SpotMe used led some to change how they clicked, highlighted, and spoke. We contribute an empirical account of inspecting and controlling proactive multimodal AI by decoupling perception from participation.

0.42

Visual Scaffolding: Toward Native Multimodal Reasoning

cs.CV ·2026-10-18 ·Run Shao, Zhaoyang Zhang, Ziyu Li, Linrui Xu, Ji Qi +4 more
similarity 0.63
Abstract

The success of chain-of-thought (CoT) in complex reasoning can be attributed to a textual scaffolding effect: explicit intermediate results can be examined, reused, and composed to support subsequent reasoning, much as scaffolding supports a building under construction. However, mainstream multimodal LLMs lack an equivalent scaffold for visual distinctions that are difficult to express in concise verbal descriptions. We therefore ask: Can native visual representations become first-class citizens alongside language, providing a visual scaffold for reasoning? We define native multimodal reasoning as reasoning in which native visual representations and language jointly carry reusable intermediate states. As a concrete implementation, the visual scaffolding mechanism we proposed lets the MLLM autonomously initiate a retrieval action at any reasoning step. A lightweight retriever selects native visual features, which are then incorporated directly into the evolving CoT and remain available to subsequent steps. To evaluate this mechanism under controlled visual demands, we introduce NMR-Bench, comprising Burden Counting and Path Tracing. With 88M added parameters (approximately 2%) on Qwen3-VL-4B-Thinking, our model achieves 66.43% and 72.50% accuracy, respectively, surpassing the evaluated GPT-5.5 and Gemini 3.1 Pro baselines. Controlled interventions show that changing selected visual content affects task accuracy and can redirect subsequent visual selection even when intervening text is held fixed. These findings show that native visual states can function as intermediate reasoning material and point toward reusable scaffolds for additional modalities.

0.15