CS Paper Radar

TouchTwin: Human-AI Haptic Authoring for 3D-Scanned Tabletop Scenes through Language and Touch

Wanli Qian, Xinlei Yu, Heather Culbertson

cs.HC ·submitted 2026-10-12 18:54 UTC · v1
published

Abstract

Creating haptic experiences from real-world environments remains difficult: visual scans capture appearance but not material feel, while manual haptic authoring requires specialized expertise. We present TouchTwin, a human-AI approach that turns phone scans of tabletop scenes into editable haptic scenes. TouchTwin uses AI to propose material regions that users can experience and revise through language and spatial edits. A model trained on measured tool-surface interactions generates the haptic responses, allowing users to work with material descriptions rather than low-level signal parameters. In a study with N=12 participants and physical reference objects present, automatic drafts scored 4.0/7 for realism and 5.4/7 as useful starting points. After a median of 6.6 minutes of editing, realism increased to 5.6/7, while scoring the saved material layers against ground truth showed material-family accuracy improving from 52% to 92%. Participants also used the same workflow to author material feels without a physical reference.

Comment
submission CHI2027

Closest papers

Nearest neighbours by meaning across the whole index.

The Potential of Haptic Foundation Models

cs.RO cs.CVcs.MM ·2026-08-23 ·Jianquan Wang, Haiwei Dong, Abdulmotaleb El Saddik
publishedbenchmark similarity 0.63
Abstract · accepted by IEEE Consumer Electronics Magazine

Despite the success of foundation models in language and vision, their expansion into embodied AI is bottlenecked by a lack of generalized touch sensing. This limitation is especially relevant to consumer electronics, where smartphones, wearables, VR controllers, home robots, and health monitoring devices require safe and adaptive physical interaction. Constrained by hardware heterogeneity and the necessity of active physical data collection, current haptic models remain rigidly task-specific. To overcome these limitations, this article explores the transformative potential and developmental trajectory of Haptic Foundation Models (HFMs). We detail the paradigm shift required to transition from passive Large Language Models and Vision Language Models into active HFMs across four core dimensions: action coupling, physical dynamical representation space, continuous time-series data granularity, and action-conditioned future state prediction. Furthermore, we synthesize existing large-scale tactile datasets and benchmark UniTouch, AnyTouch, T3, and Sparsh on TacBench for force estimation, slip detection, and relative pose estimation.

0.29

BAT-WAM: Bridging Action and Touch in World-Action Models for Dexterous Hands

cs.RO ·2026-11-07 ·Dong Fang, Feixuan Huang, Yuanxin Zhong, Yuxiang Gao, Qi Song +3 more
similarity 0.63
Abstract

Dexterous manipulation with world action models (WAMs) relies on hand configuration and contact information, yet joint-state and whole-hand tactile observations may be partially or entirely missing. RGB images provide scene context and hand-object geometry but are insufficient to reliably recover these missing modalities. We further observe a correlation between hand state and tactile information. Motivated by this observation, we introduce BAT-WAM, a framework that bridges visual, joint-state, and tactile representations for WAMs. BAT-WAM learns separate codebooks and decoders for the tactile and action streams. An RGB-conditioned continuous latent bridge connects the two streams for bidirectional learning, while visual information serves as continuous context. Under the HTBench evaluation protocol, BAT-WAM reduces the root mean squared error (RMSE) of tactile reconstruction by 34.73% under fully masked conditions and 38.84% under partially masked conditions, relative to an image-conditioned HandTouch checkpoint. For future tactile prediction conditioned on target-time RGB and historical actions, it reduces mean RMSE by 85.56% relative to OpenWAM with HandTouch. Real-robot experiments further show that BAT-WAM consistently outperforms related WAM baselines in dexterous manipulation. Generalization experiments on unseen objects and scenes also demonstrate substantial improvements over baseline models. These findings support joint modeling of visual context, hand state, and touch as a representation approach for multimodal WAMs.

0.09

T-WLA: Bridging Slow Vision and Fast Touch

cs.RO ·2026-10-20 ·Kaichen Zhou, Yuzhen Chen, Hannah Yao, Chonghe Jiang, Rao Fu +12 more
code/data similarity 0.62
Abstract

In contact-rich manipulation, contact conditions can change before a visual policy updates its action plan. Recomputing the plan through the visual policy adds latency, while tactile feedback alone does not specify how the ongoing motion should be revised for the task. We introduce T-WLA, a world-language-action architecture that uses shared visual memory to guide both action generation and tactile correction. At each visual update, the visual branch integrates the current image, task instruction, and retrieved history into a visual memory that conditions base-action generation. The same memory is cached for a lightweight tactile editor, which combines it with the active action chunk and incoming touch to predict residual corrections to unexecuted actions without rerunning the visual branch. Future-image and future-pressure prediction provide training-only supervision, encouraging the visual memory and tactile features to capture interaction dynamics relevant to action generation and correction. We evaluate T-WLA on tactile-conditioned hand-motion prediction using EgoTouch and Tachin, and on four real-world dexterous manipulation tasks. T-WLA reduces finger-joint MPJPE by 30% relative to the strongest reported baseline and achieves an average success rate of approximately 50.0% across real-robot manipulation tasks. Videos and code are available at https://anonymous-submission-20.github.io/T-WLA.github.io/.

0.22

HapticWAM: Distilling Imagined Touch into a World-Action Model without Inference-Time Tactile Sensing

cs.RO ·2026-09-20 ·Mikhail Sannikov, Ilya Mikhalchuk, Konstantin Gubernatorov, Petr Kovalev, Ogunwoye Faith Oluwatobi +1 more
hot direction similarity 0.61
Abstract · 10 pages, 5 figures, 4 tables, submitted to IEEE ICRA 2027

Contact-rich manipulation requires estimating forces, slip and contact geometry that can remain ambiguous in scene images. Optical tactile sensors provide both visual observations of the contact surface and mechanical measurements, yet learning from these signals raises two challenges: representing contact beyond appearance and transferring its benefits to a policy that does not require fingertip observations at deployment. We introduce HapticWAM, a world-action model that combines heterogeneous tactile encoding, structured contact prediction and teacher-student distillation. Its teacher encodes gel images together with deformation, shear, distributed forces, resultant wrench and derived contact state into a frozen video backbone. Rather than predicting tactile pixels alone, the model jointly generates actions and a contact package describing future events and mechanics. Anticipatory Contact Coupling uses the previously imagined package to condition attention, preserving a contact-related input when direct tactile observations are unavailable. Haptic-Imagination Distillation transfers both contact futures and action predictions to a student that retains the generative contact head but removes its fingertip input branches. On a real-world setup, across three contact-rich pick-and-place tasks, HapticWAM Student achieves a 77% per-task mean success rate (41 of 50 starts, 82% pooled), reaching 95% on one of the tasks, outperforming the evaluated teacher and baseline configurations.

0.28

Vibes on Demand: Adding Vibrotactile Encoding to Line Charts Shows Experiential Benefits Without Performance Costs

cs.HC ·2026-08-15 ·Anchit Mishra, Oliver Schneider, Matthew Brehmer
similarity 0.61
Abstract

Details on demand is a common design pattern in visualization design, especially useful when interacting with visually-saturated or small displays. Beyond visualization, another common approach for saturated displays is to incorporate other modalities, such as haptic feedback. While haptic rendering in visualization has primarily targeted accessibility needs, with haptics as a substitute for visual feedback, studies using haptics outside of a visualization context have shown value in experiential factors, such as increased confidence in ambiguous contexts and higher engagement. We explore vibrotactile feedback as a reinforcing information channel for communicating trends in details-on-demand tooltips on touchscreens. We identify preferred parameter configurations for our haptic encoding, informed by a study where participants identified parameter configurations that they perceived to most accurately reflect the dynamics of line charts appearing in tooltips. In a second study, we evaluated participant performance in a pairwise comparison task, finding that incorporating vibrotactile encoding improves involvement without affecting accuracy. We discuss the implications of these findings for future visualization design, and propose directions for applications and future studies.

0.04

Comparing Haptic Feedback Across Hand Tracking and Controllers in VR Object Interaction Tasks

cs.HC ·2026-09-18 ·Natalia Ocampo, J. Felipe Gonzalez, Robert J. Teather, Kiyoshi Kiyokawa
published similarity 0.60
Abstract

Hand tracking offers natural VR interaction but lacks controllers' inherent tactile feedback, while haptic feedback across input methods remains understudied. We conducted a participant study comparing vibration, impulse, and no feedback in hand- and controller-based VR interaction across grasp-, pinch-, and tap-like gestures assessing performance, workload, and preference. A custom glove and controller-mounted device provided both feedback types, aiming for consistency across input modalities. Controllers were faster for grasp and pinch, whereas hand tracking was more accurate for pinch and preferred overall. Haptics had limited performance effects, although impulse reduced grasp accuracy with controllers. Participants preferred vibration and impulse over no haptic feedback, favouring vibration overall. Our findings reveal a more nuanced relationship between haptic feedback, input device, and interaction context than suggested by previous work.

0.36

From Semantic Editing Intentions to Persistent Parametric Controls for AI-Assisted Image Editing

cs.HC ·2026-10-14 ·Rifat Mehreen Amin, Michelle Dutoit, Anton Rockenstein, Baptiste Caramiaux, Andreas Butz
similarity 0.60
Abstract

AI-powered image editing increasingly allows users to express high-level editing intentions, yet how these intentions are translated into concrete editing actions often remains difficult to understand. We present Atelier, a semantic image editing tool that uses image semantics and user intent to determine which editing controls to surface and how to initialize them. Guided by a formative study with eight image-editing practitioners, Atelier composes context-specific, directly manipulable controls that users can refine rather than committing AI interpretations directly to finished edits. In a within-subjects study with 12 practitioners, participants preferred AI-composed controls, described them as supporting creative exploration, and actively refined AI proposals. However, experiences of control were mixed, and interviews highlighted limited insight into why particular controls and values were proposed. Our findings suggest that making AI interpretations actionable through manipulable controls can support exploration and refinement, but actionability alone does not make those interpretations transparent to all users.

0.25

AVT-Fabric: Active Visuo-Tactile Perception via Adaptive Evidence Selection for Efficient Robotic Fabric Comparison

cs.RO ·2026-09-18 ·Chang Gao, Zhuo Chen, Suhang Xia, Jihong Zhu, Jiankang Deng +1 more
similarity 0.60
Abstract · Project website: https://zhuochenn.github.io/AVT-project/

Robotic fabric comparison needs to actively combine visual appearance and tactile cues. Here, we present AVT-Fabric, an RGB-first framework that allocates tactile evidence according to the difficulty of each comparison. A dual-scale gate evaluates answer-token confidence and raw logit separation to determine whether another force-tagged GelSight observation is needed. Compact textual memory preserves the executed history, and majority voting consolidates the selected predictions. On 400 held-out comparisons, AVT-Fabric achieves 98.0% accuracy with a compact 7B Multimodal Large Language Model (MLLM), surpassing the 94.0% reported by the 90B MLLM-Fabric baseline by 4.0 percentage points while processing only 1.60 of five available stages on average. It improves on matched passive inference by 9.25 percentage points and reduces model-side latency by 61.8%, while also improving on RGB-only accuracy. Four additional MLLM backbones support the generalizability, accuracy, and efficiency of the framework. This framework is also deployed on a real robotic system, achieving 78.1% pairwise ranking accuracy and correct fabric selection in seven of eight application scenarios. AVT-Fabric demonstrates that adaptive evidence selection can improve both the accuracy and efficiency of robotic visuo-tactile reasoning.

0.33