CS Paper Radar

Encypher: Shared Agency and Social Presence in Collaborative Music Generation for Dance Cyphers

Zhixing Chen, Cheng-Zhi Anna Huang

cs.HC cs.SDeess.AS ·submitted 2026-09-16 03:08 UTC · v1
code/data

Abstract

Music and dance are social practices of expression and connection, yet most HCI work in human-AI co-creation centers the solo performer. As generative music matures, we ask not only what AI can compose but what social encounters it can organize around sound. We present Encypher, a collaborative generative music system that translates collective movement qualities into text prompts conditioning real-time music generation for dance cyphers. Through five weeks of co-design with local dancers, a user study with unacquainted participants, a public museum event, and a live performance, we found that users developed shared agency, perceiving the music as a response to the room's energy. While newcomers felt uncertain, the system fostered social presence by prompting them to look to each other for cues. By treating sociality as a design concern rather than a downstream effect, we offer a framework and design implications for AI systems for collaborative, embodied expression.

Comment
Preprint. Under review at CHI 2027. 11 figures, 1 table. Project page: https://encypher-chi.github.io/

Closest papers

Nearest neighbours by meaning across the whole index.

CrowdForge: Effects of Audio and Visual Audience Participation Modalities on Agency, Collaboration, and Engagement in AI-Mediated MR Performances

cs.HC cs.AIcs.NIcs.PF+1 ·2026-11-02 ·Torin Hopkins, Shih-Yu Ma, Zander Trafton-Hancock, Ellen Yi-Luen Do, Luca Turchet
position similarity 0.69
Abstract

Technology-mediated audience participation (TMAP) enables audiences to influence live performances, yet balancing interaction with meaningful agency remains challenging. While AI-generated music and MR enable new forms of audio and visual audience participation, little is known about how these participation modalities influence audience experiences during live musical performance. In particular, it remains unclear how they shape perceived agency, collaboration, and engagement, and how they complement one another within TMAP systems. We present CrowdForge, an AI- and MR-mediated paradigm where audiences collectively control generative music and virtual environments during a live performance featuring a guitarist and DJ. Through a mixedmethods concert study evaluating three interaction conditions (audio, visual, and combined audio-visual), we demonstrate that observability is the primary driver of audience agency. While visual interaction provided easily observable feedback, audio parameters offered macro-structural control over the music. Audiences tended to use the control enabled by AI audio generation to test musical boundaries, introducing a productive adversarial tension between the crowd and the musicians. This dynamic pushed creative limits, deepened engagement, and generated moments of musical coherence when performers successfully resolved audience-induced challenges. We argue future TMAP systems should prioritize human-centered, highly observable interactions to build agency, while combining the expressiveness of music with the clarity of visual feedback.

0.19

H2H Music Improv: A Communication Model and Audio-Visual Dataset for Music Improvisation

cs.SD cs.MMeess.AS ·2026-08-14 ·Aleksandra Teng Ma, Anthony Cammarota, Jiayi Wang, Alexandria Smith, Cheng-Zhi Anna Huang +2 more
published similarity 0.67
Abstract · Published in the Proceedings of the Society for Music Information Retrieval Conference (ISMIR) 2026

Current real-time AI improvisation systems lack the communication awareness human musicians rely on: rather than treating communication as a foundational algorithm design concern, most systems layer interaction strategies post-hoc onto generative algorithms through explicit controls and predefined modes. This gap persists in part because no formalized, machine-readable communication model with musicians exists. To address this, we study how expert musicians communicate in free (non-idiomatic) improvisation, unconstrained by prior discussion or agreement. Through a collaborative co-design process with expert improvisers, we derive a communication model that (1) captures how free improvisers negotiate musical ideas and enter stable musical spaces, and (2) is formalized as a machine-readable annotation scheme. We further present the H2H (Human-to-Human) Music Improvisation dataset: six hours of audio-visual expert duo improvisations with clean per-player stems and per-player annotations of both their own intentions and their perception of their partner's intentions. To our knowledge, this is the first such dataset for free improvisation. Together, the communication model and the dataset offer a new lens and resource for studying musician communication and may in future inform the design of AI musical partners that communicate by design.

0.19

Navigating relational agency in AI feedback musicianship: a diffractive dialogue

cs.SD cs.HC ·2026-11-10 ·Yichen Wang, Charles Patrick Martin, David Gunawan, David Cooper
similarity 0.64
Abstract

Musical AI systems behave in non-linear, dynamic ways that exceed the tool-oriented framings inherited from earlier instrument design. Existing frames of control and ensemble interaction do not articulate the entangled, relational agency that emerges when performers work with AI feedback instruments over time. We examine a duet practice with two AI-enabled feedback instruments, Sù and Agentier, in which feedback operates at the inner level of the AI's recurrent processes and the outer loop of musical interaction between performer, instrument, and co-performer. Our work draws on agential realism and a diffractive dialogue methodology to read our independent reflections through one another across five performances, examining how agency is enacted between performers and AI instruments. We find that control and agency are relational and shift across performances; musical textures enact feedback moments of control with performers; reaching a productive feedback environment is itself a shared exploratory act; and performers maintain their instruments through funnelling and injection, developing the skill of navigating agency along a continuous spectrum. Feedback musicianship with AI instruments demands navigation of this relational agency, displacing the leading-versus-responding binary that initially framed our reflections. This work contributes a materially-grounded vocabulary for AI feedback musicianship and a model of human-AI creative collaboration in which mutual agency is worked for through practice rather than designed in.

0.19

Decomposing the Associative Chain: Schema-Based Mediation Supports Expression and Understanding in Human-AI Music Co-creation

cs.HC ·2026-11-14 ·Songruoyao Wu, Wanying Wu, Shuyu Li, Chongjun Zhong, Ziyi Huang +1 more
similarity 0.64
Abstract

Users cannot reliably translate their creative intents into AI-generated music. We contend that the root of this difficulty lies in associative divergence: associations between narratives and musical features are overly remote, and users lack a visible mediation structure to negotiate them. To address this, we introduce five core schema groups as cognitive mediation and propose a bidirectional schema-based framework. Its forward pathway decomposes narrative-to-music associations into staged mappings, while its reverse pathway provides structured interpretations from music back to narrative. Findings from two user studies(N = 188; N = 60) show that the framework led to clearer creative expression and greater perceived control, improved the comprehensibility of generated music, especially for unconventional pieces. Furthermore, it supported cultural and affective thinking while helping users locate associative divergence. Our work points toward better creative intent alignment in human-AI music co-creation.

0.30

Breaking the Group Size Barrier: Parameter-Efficient Group Dance Generation with Chain-of-Dancers

cs.CV cs.SD ·2026-10-08 ·Jing Xu, Cunjian Chen, Qiuhong Ke
top venue similarity 0.62
Abstract · Accepted at NeurIPS 2026

Group dance generation aims to synthesize coordinated multi-dancer choreography from music, with broad applications in animation and interactive content creation. This task requires modeling dense inter-person dependencies to ensure spatial coordination, while naturally preserving individual dancer identities. Existing approaches model all dancers jointly with end-to-end transformers, which tie the architecture to a fixed group size and entangle per-dancer identities across frames. We propose ChainDance, a scalable framework that reformulates group dance generation as a Chain-of-Dancers: a sequential decomposition over per-dancer conditional distributions, allowing a single model to scale across variable group sizes without retraining and naturally preserving per-dancer identity. Built on a frozen single-dancer diffusion backbone, ChainDance introduces two lightweight modules: a Role-Aware Text Encoder (RATE) for per-dancer semantic conditioning, and a Group-Aware Motion Encoder (GAME) that aggregates previously generated dancers via a distance-weighted graph convolutional network, and incorporates a training-free noise optimization procedure at inference time to enforce global spatial coherence. Experiments on AIOZ-GDance demonstrate that ChainDance achieves state-of-the-art motion quality and group coordination while structurally preserving per-dancer identity, with $3$-$4\times$ fewer parameters and requiring $3$-$6\times$ less training time compared to prior approaches.

0.29

Art2Song: Enhancing Visual Art Appreciation with Contextual Music Generation

cs.HC ·2026-09-05 ·Sungeun Jo, Myung Jin Kim, Chi Yoon Jeong
top venue similarity 0.62
Abstract · 3 pages, 1 figure, To appear in the Adjunct Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology (UIST Adj…

Art2Song is a conceptual framework that expresses artworks as sound by separating Non-Visual Context, which is difficult to perceive from the image alone, from Visual Evidence. Visual Evidence, such as objects, colors, and spatial composition, is transformed into Lyrics, while the Contextual Mood derived from the historical and art-historical context in the museum's artwork description is reflected in the background soundtrack. Rather than describing artworks textually, Art2Song aims to explore the possibility of a new mode of art appreciation in which viewers experience hidden stories and emotional context through music. As future interaction directions, we plan an Emotional Layer Blending Slider interface and a structured, traceable song-generation scenario, presenting the possibility that users can explore the relationship between Visual Evidence and Contextual Mood.

0.27

Reflecting on Creative-Boundaries with an AI Co-Doodler

cs.HC cs.AIcs.GR ·2026-10-04 ·Samia Menon, Samyukta Jayaram, Chetan Goenka, Shm Garanganao Almeda
similarity 0.61
Abstract · In Proceedings of The First Reflection in Creative Experience (RiCE) Workshop (RiCE W1) arXiv:2607.24558

In this pictorial, we consider how the negotiation of creative boundaries with a co-creative AI system can create moments for personal creative reflection. We ground this in our experiences with Froggi-Draw, a single-initiative co-doodling system that gives users power to decide when and how much an AI "collaborator" (Froggi) contributes to their drawing. From a 1-week pilot study where (n=8) novice and experienced artists doodled daily with the system, we report on ways participants navigated creative risk and uncertainty, and how their usage and perceptions of Froggi shifted over time. In moments of disruption, participants described the system as encroaching on their creative territory. We consider how the design of a supportive, co-creative AI "collaborator" might look like the design of a supportive power dynamic---and finding ways to offer users control to find, reflect upon, and flexibly negotiate the boundaries of that dynamic.

0.12

In Human Memories. Technology, Creativity and Self-Determination: An Ethical Proposal

cs.CL ·2026-10-16 ·Francesco Vitucci, Giuseppe Silvi, Francesco Abbrescia, Anthony Di Furia, Francesco Scagliola
published similarity 0.60
Abstract · 19 pages, 7 figures. English translation of the article originally published in Italian in Ex Chordis, vol. 3 (2024), pp. 15-38. The Italian…

This research investigates the role of artificial intelligence (AI) in modern compositional practices, with a particular focus on two case studies: Convergence (2020/21) by Alexander Schubert, for String Ensemble and AI System, and In Human Memories (2024) by Francesco Vitucci, for four cellos and one electroacoustic player, whose installation was created by the Electronic Music School of the "N. Piccinni" Conservatory of Bari. The initial historical survey traces the evolving use of technology in composition, revealing how the search for new expressive possibilities has driven the development of technological tools through experimentation. The appearance of AI marks a fundamental shift: no longer just a tool, AI now poses questions and challenges to the composer. Engaging with AI becomes an exploration of an unknown entity, whose full capabilities are still largely undiscovered. This new interaction leads composers to confront AI not merely as a passive resource but as a participant in the creative process, outlining an ethical path to reclaim technology as a creative tool. In Human Memories reflects this dynamic, demonstrating how AI can serve as both inspiration and a driving force in shaping contemporary music.

0.27