CS Paper Radar

Dancing Through Soundscapes: Designing a Low-Cost, Sound-Based Device for Sensing and Interpreting Movement and Dance

Swen E. Gaudl, Silvia Carderelli-Gronau

cs.HC cs.SD ·submitted 2026-08-20 09:29 UTC · v1

Abstract

When we move through space, we often rely on multiple senses beyond vision to perceive and act in that environment: we ``feel'' the presence of others; we build internal representations and models and recall them to navigate the environment. We also leave traces and impressions that others pick up on. The traces include echoes, heat, the displacement of objects such as furniture or footprints, air movement close to the face of another, smells such as perfume, but also the immediate sounds we make when we move and breathe. Movement is a spatial and temporal activity, and dance as a form of movement practice requires coordination of oneself in relation to others, the space and a potential score. When rehearsing dance, dancers have to relate to others often not just by looking but more often by feeling and imagining or remembering where others are based on experience and shared practice. So how can we approach technology-mediated movement and dance? Why should we explore it? How can we support spatial and temporal practice meaningfully and joyfully? In this work, we focus on sound traces; we present the design and rationale for a sound-based artefact that translates movement-based sound into layered, explorable, generative soundscapes. The work contributes a novel artefact for exploring movement-based activities with a audio-first approach, with a focus on the spatial performative experience. The paper further reflects on observations from workshops and public sharings, including how participants used repetition, stillness, environmental sound, and call-and-response to understand and improvise with the soundscape.

Comment
full paper on technical artifact for TEI2027 conference

Closest papers

Nearest neighbours by meaning across the whole index.

Somatosensory Activation and Attentional States in Creative Making

cs.HC ·2026-09-09 ·Katherine Rees
similarity 0.64
Abstract · In Proceedings of The First Reflection in Creative Experience (RiCE) Workshop (RiCE W1) arXiv:2607.24558

The methods for capturing the creative process come with associated tensions around memory recall, articulation, and communication during the act of making, as well as how to record these considerations. This paper has a twofold purpose: first, to offer an example of a mixed methodology, drawn from dance anthropology, sensory ethnography, and design, that applies embodied methods as an alternative for documenting creative making. Specifically, this incorporates the researcher-as-participant and the collation of fieldnotes, embodied knowledge/movement recall, with notation forms, and participant interviews. These are existing methods in dance anthropology; however, using them alongside exploratory prototyping and workshop approaches broadened this work into transdisciplinary practice. Second, it discusses the activation of somatosensory systems through wearable technology and the facilitation of heightened sensory awareness for the practitioner, leading to a subsequent ability to focus on creative decisions linked to reflection and metacognition.

0.28

Encypher: Shared Agency and Social Presence in Collaborative Music Generation for Dance Cyphers

cs.HC cs.SDeess.AS ·2026-09-16 ·Zhixing Chen, Cheng-Zhi Anna Huang
code/data similarity 0.59
Abstract · Preprint. Under review at CHI 2027. 11 figures, 1 table. Project page: https://encypher-chi.github.io/

Music and dance are social practices of expression and connection, yet most HCI work in human-AI co-creation centers the solo performer. As generative music matures, we ask not only what AI can compose but what social encounters it can organize around sound. We present Encypher, a collaborative generative music system that translates collective movement qualities into text prompts conditioning real-time music generation for dance cyphers. Through five weeks of co-design with local dancers, a user study with unacquainted participants, a public museum event, and a live performance, we found that users developed shared agency, perceiving the music as a response to the room's energy. While newcomers felt uncertain, the system fostered social presence by prompting them to look to each other for cues. By treating sociality as a design concern rather than a downstream effect, we offer a framework and design implications for AI systems for collaborative, embodied expression.

0.24

Rescuing Performance from the Demo: Co-Designing Drum Gesture Mappings with a Percussionist

cs.HC cs.SDeess.AS ·2026-09-08 ·Jordie Shier, Teresa Pelinski, Charalampos Saitis, Andrew Robertson, Andrew McPherson
published similarity 0.55
Abstract · Accepted for publication at AIMC 2026

Augmenting instruments with sensors and neural network mappings is a well-explored digital musical instrument design approach. While augmentations can create new expressive opportunities, they also exert aesthetic influence and can constrain musicians' gestural language, which, if left unchecked, can lead to technological capture. To examine this, we conducted a study with a professional percussionist, co-developing a gesture mapping toolkit and recording a ten-track album. Drawing on the concept of productive dissonance, our study aimed to hold the musician's aesthetic in tension with technological constraints. This, along with a practice-based reflective approach, supported the development of a continuous gesture recognition method for percussive mapping and surfaced insights into the design process. We identify knowing-when as a form of tacit knowledge that supported productive dissonance, and raise an open question: absent a musician's broader social context, how do we know whether a technology's influence is genuinely supporting their practice?

0.30

Making Smell Actionable in Design: Lexicon, Movement Patterns, and Gesture Cards

cs.HC ·2026-11-03 ·Anran Feng, Shengyu Gao, Xingyu Zhao, Zhenyu Qian, Jingrui An
similarity 0.53
Abstract · 28 pages, 5 figures, including appendix

Smell is powerful in design, yet it often remains vague and difficult to work with in studios and classrooms. While olfactory HCI has produced devices, toolkits, and scenario-based applications, designers still lack teachable, shareable resources that link smell to bodily action. This paper presents a three-phase research program that makes smell actionable by moving from lived smell experiences to movement patterns and, finally, to gesture resources. We study olfactory-motor mappings and package the results as teachable design representations through a progressively narrowing method: (1) extracting smell-verb-intent relations from user narratives; (2) capturing and clustering embodied enactments into standardized upper-limb trajectory families; and (3) translating these trajectories into gesture card representations for design teaching and ideation through design workshops. We present a three-phase translational framework and empirically derived smell-gesture correspondences that structure the smell-intent-trajectory-gesture pathway, together with reusable assets (lexicon, trajectory atlas, gesture tables, and cards) that support teaching, reuse, and adaptation across contexts.

0.29

Exploring Breathing-Music Coupling: Using the Breathing Mirror for Somatic Reflection in Piano Performance

cs.HC ·2026-09-02 ·Ziyue Piao, Yohei Wada, Isabelle Cossette, Marcelo M. Wanderley, Akira Maezawa
published similarity 0.53
Abstract · Proceedings of the International Conference on New Interfaces for Musical Expression (NIME)

While breathing is essential to living and for sound production in some instruments, for pianists, it is often a hidden and automatic process, making it difficult to analyze or refine. A critical gap exists between data and awareness: while sensors record precise physical metrics, they fail to capture the performer's somatic experience. Conversely, the high cognitive load of performance makes it nearly impossible for musicians to recall their internal states with temporal precision. To address this, we present a system, Breathing Mirror, and associated methodology designed to externalize the pianist's internal somatic experience through three analytical lenses: a Baseline View (synchronized signals), a First-Person View (subjective recall), and an Interpersonal View (collaborative reflection). Through a four-week longitudinal study with a skilled amateur pianist (35 years of experience), we evaluated the system's effectiveness by recording respiratory data using textile-integrated strain sensor belts. The results show that the Breathing Mirror reveals some patterns of breathing-music coupling and identifies critical blind spots where objective data diverges from subjective perception. Furthermore, we propose four somatic themes regarding the link between breathing and musical elements, offering a foundation for future large-scale validation across a broader range of pianists. This work provides a new way to study body signals, transforming breathing from an internal biological function into an articulate expressive parameter.

0.62

BODYinTRANSIT-Lab: A Multimodal Dataset for Investigating Sensory-Driven Transformations in Body Perception and Affective Experiences

cs.HC cs.LG ·2026-11-10 ·Mohammad Mahdi Dehshibi, Tomás Martínez Cortés, Amar D'Adamo, Daniel de la Prida, Luis-Antonio Azpicueta-Ruíz +9 more
benchmarkdataset similarity 0.52
Abstract · 18 Pages, 6 Figures, 7 Tables. Submitted to IEEE Transactions on Affective Computing

Multimodal datasets that combine synchronised behavioural and physiological signals with subjective affective and body perception measures are still rare. This limits both the understanding of embodied experience and the development of computational models that infer subjective states from sensor data. We introduce BODYinTRANSIT-Lab, a multimodal dataset that addresses this gap by capturing affective, perceptual, and behavioural responses to the Footsteps Illusion. In this closed-loop paradigm, participants perform walking tasks while receiving real-time manipulated auditory feedback of their own footsteps intended to alter perceived body weight. The dataset comprises synchronised recordings from 101 participants recruited through systematic stratification by eating-disorder symptomatology and physical activity level. BODYinTRANSIT-Lab combines full-body kinematics with electromyography, electrocardiography, respiration, and electrodermal activity. Its annotations capture both immediate and post-trial experience, including within-trial perceived-weight ratings, post-trial assessments of body feelings and affective experience, 3D avatar body visualisations, and quantitative body maps. Psychological, behavioural, demographic, and social measures provide further information about factors that may influence individual responses. We provide a deep learning benchmark that uses walking kinematics to classify relative affective valence and perceived weight within subjects. On within-session, segment-level splits, the baseline model achieved mean balanced accuracies of 45.9% for affective valence and 44.5% for perceived weight. The BODYinTRANSIT-Lab dataset provides an experimentally controlled resource for cognitive neuroscience and affective computing research, with potential applications in affect-aware interactive systems, health and well-being technologies, and creative embodied experiences.

0.30

The Choreographic Genome: Amplifying the Silent Structure of Text into Dance

cs.HC ·2026-09-18 ·Michael Li, Alison Ding
publishedbenchmark similarity 0.52
Abstract · Accepted at IEEE VIS 2026 Arts Program

Recent advances in generative artificial intelligence have enabled the synthesis of complex human motion with unprecedented fidelity. However, current text-to-motion systems rely strictly on linguistic semantics: if an input reads "I put my hands up", the model searches for a pose with raised hands, and every non-semantic property of the text is discarded as noise. In this work, we treat that discarded structure as the signal. We present an embodied visualization instrument that amplifies not what a text means, but how it is built. Our method first quantizes dance kinematics into a motion codebook of 256 stylistic "regions" using Principal Component Analysis and K-Means clustering, and orders those regions along the dominant axis of movement. We then map the raw byte representation of any input text directly onto this codebook, producing a deterministic sequence of regions that we call the text's "choreographic genome". A precomputed plausibility graph and a set of physics smoothing routines turn this genome into fluid, full-body movement, so that the dancing body becomes a display surface for the byte-level structure that semantic systems ignore. Through a series of artistic case studies, including a Shakespeare sonnet, a machine error log, source code, an abolitionist's question, and Indigenous and Devanagari scripts, we show that each text produces a visibly distinct dance, and that scripts marginalized by ASCII-centric computing are amplified into close to three times as much movement per character. We frame this not as a motion-synthesis benchmark, but as a critical and poetic visualization that asks what we choose to count as signal, and what we allow to go unheard.

0.30

Rhythmic Fitness Beat 'Em All using Real-Time Monocular Human Pose Estimation

cs.HC ·2026-11-06 ·Yanis Zaoui, Valentin Bégel, Mohamed Adjel
publishedhot direction similarity 0.51
Abstract

Short and frequent physical activity exercices mitigate the health risks of sedentary behavior. Yet, the lack of motivation remains a critical barrier to adoption. Exergames gamify physical activity, but most rely on dedicated hardware, and outside dance and rhythm games they rarely use the musical beat. We present a work-in-progress beat 'em up exergame played with the whole body in front of a single smartphone, which runs the entire pipeline: an off-the-shelf real-time 3D human mesh recovery model on the phone's neural processing unit, gesture recognition, and the game, shown on any TV through a USB-C to HDMI cable. Our contributions are (1) a 3D recognition of five shadow-boxing moves (guard, punch, slip, squat, jump), each defined by one metric measure in a body frame aligned with gravity and one threshold with hysteresis, personalised during a tutorial; (2) a rhythm stage whose discrete cues are placed on the song's kick drum and judged on the movement's capture time, with a delay learned for each player; and (3) a phone-only architecture, running at 30 frames per second, that turns a phone and an HDMI cable into an exergame console.

0.58