CS Paper Radar

Touching Projected Agents through the DAT Screen -System Design and Questions of Interaction Boundaries-

Takaya Tamamoto, Tomoko Yonezawa

cs.HC cs.ARcs.ROeess.SY ·submitted 2026-11-02 04:51 UTC · v1

Abstract

Touching a virtual agent involves more than detecting a finger on a display: the contact surface can deform and resist the user's push, and the agent can respond to that action. We describe the Desktop Amid Touch (DAT) screen, which combines spatial projection onto a semi-transparent, deformable mesh with touch sensing and motorized control of resistance to indentation. An initial prototype detected a single touch and estimated its position and depth from tension changes around a flexible projection screen. A subsequent prototype added motorized tension control to vary the screen's resistance to indentation. We describe the tension-controlled DAT implementation and its demonstrations, distinguishing what has been observed from what remains an open design question. These capabilities raise a design question: should touchable virtual agents respond as though they had preferences or boundaries, and under what conditions should such behavior be implemented on systems like DAT. The present paper describes a system and frames this question; it does not establish how users interpret an agent's refusal or consent.

Comment
The 2nd International Workshop on Trust and Blame in Multi-Person Multi-Agent Societies (TBSA_2nd) in conjunction with HAI 2026, 4 pages, 4 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Selfrionette: A Fingertip Force-Input Controller for Continuous Full-Body Avatar Manipulation and Diverse Haptic Interactions

cs.HC ·2026-11-02 ·Takeru Hashimoto, Yutaro Hirao
top venuehighly cited similarity 0.65
Abstract · 14 pages, 12 figures. Author-prepared version of the UIST 2024 paper. Both authors contributed equally

We propose Selfrionette, a controller that uses fingertip force input to drive avatar movements in virtual reality (VR). This system enables users to interact with virtual objects and walk in VR using only fingertip force, overcoming physical and spatial constraints. Additionally, by fixing users' fingers, it provides users with counterforces equivalent to the applied force, allowing for diverse and wide dynamic range haptic feedback by adjusting the relationship between force input and virtual movement. To evaluate the effectiveness of the proposed method, this paper focuses on hand interaction as a first step. In User Study 1, we measured usability and embodiment during reaching tasks under Selfrionette, body tracking, and finger tracking conditions. In User Study 2, we investigated whether users could perceive haptic properties such as weight, friction, and compliance under the same conditions as User Study 1. Selfrionette was found to be comparable to body tracking in realism of haptic interaction, enabling embodied avatar experiences even in limited spatial conditions.

0.61

TouchTwin: Scaling Human Touch Data via Real-to-Sim-to-Real

cs.RO ·2026-11-13 ·Huiwen Han, Nuo Chen, Ziyao Zeng, Dayou Li, Hao Wang +4 more
similarity 0.62
Abstract

Human demonstrations are becoming the main source of data for dexterous manipulation, and real-to-sim-to-real pipelines multiply each demonstration in simulation. These pipelines rebuild the interaction from vision, which recovers how the hand moves. Which fingers touch, when, and how hard remain unobserved, because a hand pose shows where each finger stops on the surface rather than how far it is driven to close. This gap is a force gap: on held-out recordings, a hand policy that imitates the observed pose grips with 0.31x the recorded force, and a policy trained on physics replays of vision-tracked poses grips with 1.55x. TouchTwin closes the gap with measured human touch. Each selected TouchScale recording, captured with head and wrist RGB-D cameras and an 880-taxel glove, is rebuilt in MuJoCo. Coding agents write the objects as physics-tested programs, and a per-subject tactile hand replays the interaction with the recorded touch setting the contact and the closing force of each finger. In simulation, a policy trained on touch-grounded replays grips at 0.99x the recorded force and succeeds in 20/20 trials on held-out recordings and 35/36 unseen perturbations; a contact predictor trained on them misses 13% of real contact frames, against 32% for vision-based replays. Each replay also seeds perturbed demonstrations with simulated contact and force labels, rendered as realistic video in new views and scenes.

0.09

TapID: Rapid touch interaction in virtual reality using wearable sensing

cs.HC eess.SP ·2026-10-19 ·Manuel Meier, Paul Streli, Andreas Fender, Christian Holz
top venuepositionhighly cited similarity 0.62
Abstract · accepted to IEEE VR 2021

Current Virtual Reality systems typically use cameras to capture user input from controllers or free-hand mid-air interaction. In this paper, we argue that this is a key impediment to productivity scenarios in VR, which require continued interaction over prolonged periods of time---a requirement that controller or free-hand input in mid-air does not satisfy. To address this challenge, we bring rapid touch interaction on surfaces to Virtual Reality---the input modality that users have grown used to on phones and tablets for continued use. We present TapID, a wrist-based inertial sensing system that complements headset-tracked hand poses to trigger input in VR. TapID embeds a pair of inertial sensors in a flexible strap, one at either side of the wrist; from the combination of registered signals, TapID reliably detects surface touch events and, more importantly, identifies the finger used for touch. We evaluated TapID in a series of userstudies on event-detection accuracy (F1 = 0.997) and hand-agnostic finger-identification accuracy (within-user: F1 = 0.93; across users: F1 = 0.91 after 10 refinement taps and F1 = 0.87 without refinement) in a seated table scenario. We conclude with a series of applications that complement hand tracking with touch input and that are uniquely enabled by TapID, including UI control, rapid keyboard typing and piano playing, as well as surface gestures.

0.63

SoTa: Soft Tactile Skins for Dexterous Manipulation

cs.RO cs.LG ·2026-10-01 ·Jingyun Yang, Baiyu Shi, Timothy Yu, Haitian Liu, Alberta Longhini +4 more
code/data similarity 0.62
Abstract · The first three authors contributed equally. Project website: https://sota-skin.github.io

A growing body of work suggests that tactile sensing gives robot policies contact information that complements vision in dexterous manipulation. However, visuo-tactile robot data remains scarce: dexterous demonstrations require teleoperating robots, which limits dataset scale. Human demonstrations are far cheaper to collect and offer a path to scale this data, but only if human and robot hands carry tactile sensors with corresponding signals. This requires sensors that conform to different hand geometries, cover the full hand, and share a common layout across embodiments. We present SoTa, a low-cost capacitive tactile skin that provides full-hand coverage on humans and robots while preserving a shared layout of 202 taxels across corresponding finger and palm regions. Our multilayer design with fabric electrodes enables in-house fabrication of thin, soft skins with customizable geometry for under $10 in materials per skin. The sensor retains over 97% of its initial response span after 10,000 loading-unloading cycles with traces retaining continuity through 1,280 tight-fist folding cycles. The shared taxel layout supports human-robot co-training with a common tactile encoder and no learned cross-sensor mapping. Across three contact-rich manipulation tasks, tactile observations improve in-distribution success over vision-only policies. With a fixed robot demonstration budget, adding human demonstrations more than doubles mean success across eight evaluation conditions, from 22.8% to 45.9%, improving success in all five out-of-distribution conditions. We plan to open-source the resources needed to fabricate and operate these skins.

0.20

FORCA: A Force-Aware Coding Agent for Contact-Rich Robot Manipulation

cs.RO ·2026-10-22 ·Xiao Tan, Yixuan Wang, Zhexi Wang, Wenzhen Yuan
similarity 0.60
Abstract · Project website: https://tanoom.github.io/forca/

Coding agents can translate language instructions into robot programs. For contact-rich manipulation, a key challenge is to generate and refine control logic that responds to physical interaction. We present FORCA, a coding agent that takes visual observations and wrist force/torque (F/T) measurements as input to write Python programs for contact probing and force-responsive motion. Between executions, the agent inspects force records alongside visual outcomes to diagnose failures and revise both motion and force-feedback control logic. FORCA requires no demonstrations, task-specific training, or simulation. Across five real-robot tasks, we demonstrate its effectiveness in three forms of contact-rich manipulation: improving manipulation accuracy, adapting to changes in object or surface properties, and regulating contact under surface variation and object motion. Compared with a vision-only variant of the same agent, FORCA improves grasp accuracy and task success and reduces contact loss in wiping and peeling.

0.35

Feelium: A Touchable Blimp Body for Aerial Telepresence

cs.HC cs.RO ·2026-08-29 ·George Xi Wang, Henghao Li, Shan Lin, Yunge Wen, Jiaqian Hu +1 more
top venue similarity 0.60
Abstract · 3 pages, 2 figures. To appear in UIST Adjunct '26: The 39th Annual ACM Symposium on User Interface Software and Technology, November 2-5, 20…

Floating things invite touch. We present Feelium, a blimp-based telepresence platform that enables visual embodiment and touch interaction through its inflatable skin. Through a VR headset, a remote person inhabits the blimp, looking out of it first-person, appearing on its skin as a face or avatar, and steering it through the room. Partners in the room pat it, press a palm against it, draw on it, or lean into it; the skin senses each contact, renders it into the wearer's view in VR spaces. Touch thus provides a physical interaction channel for remote presence, turning the skin into a shared surface between remote and co-located partners.

0.53

Evaluating Tactile Encoding Strategies for Communicating 3-D Shape

cs.HC ·2026-11-10 ·MacKenzie C. Harnett, Rebecca F. Friesen
similarity 0.59
Abstract

Two-dimensional haptic displays, such as programmable pin arrays and vibrotactile or friction-modulating touchscreens, enable tactile communication of complex shapes in a low-profile form factor. However, many shapes of interest, including CAD models, three-dimensional graphs, and map topographies, are inherently three-dimensional. This work investigates two methods for encoding 3-D information using a 2-D tactile display: (1) a single tactile height map providing coarse but quantifiable depth cues, and (2) a series of binary layers communicating sequential slices of a 3-D model, supporting more complex geometries such as overhangs and cavities. We further compare performance across two display types--a friction-modulating display and a pin array--to examine how the level of haptic abstraction affects users' ability to decode discrete 3-D information. A user study demonstrates that participants were able to effectively complete a 3-D assembly task using both strategies, achieving high accuracy across all display types. While the pin array supported faster task completion and higher-rated--though not statistically significant--user experience than the friction-modulating display for the binary-layer task, these differences did not translate to significant differences in overall usability (i.e., reassembly accuracy success). Together, these results identify design trade-offs and inform development of future surface haptic displays that support the communication and interpretation of discrete 3-D information in tactile graphics.

0.12

Bring Buttons Back: Physical Interfaces for the Age of Automation

cs.HC ·2026-09-13 ·David Goedicke, Donald Degraen, Tom Igoe, David Sirkin, Wendy Ju +1 more
similarity 0.58
Abstract · 6 Pages, 7 figures, article

As AI starts to permeate everyday life, deployment focuses on embedding intelligence in the background, abstracting away controls, and leaving automated decision-making opaque, with few obvious opportunities for human intervention. Where interfaces remain, design has drifted toward abstract, screen-based control, replacing material interaction with menu navigation. These shifts weaken the coupling between human action and system state, eroding operators' ability to understand, anticipate, and intervene in automated systems. We introduce Physically Stateful Interfaces (PSIs), a design concept that re-imagines traditional controls, such as buttons, switches, and knobs, as actuated, bidirectional interface elements. Three foundational behaviors, Self/Reset, Resist/Hide, and Assert/Unhide, unify affordance, feedforward, and feedback into a single interaction point. Rather than static inputs whose impact and consequences must be read elsewhere on a screen, PSIs position physical controls as shared mediators between automated systems and human operators. As automated systems become increasingly capable, their interfaces need to be deliberately designed to center human agency, supporting deliberate choice while keeping automation legible, contestable, and overrideable.

0.24