CS Paper Radar

ResonaVis: Visualizing Interactive Music Data to Support Reflective Music Composition for Therapeutic Contexts

Abhishek Karwankar, Elise Ruggiero, Daniel Stevens, Matthew Louis Mauriello

cs.HC ·submitted 2026-08-11 00:42 UTC · v1
published

Abstract

Designing music for therapeutic contexts requires navigating complex relationships between musical structure and listeners' sensory responses, yet composers often lack structured representations of these interactions, relying instead on intuition. We present ResonaVis, an interactive visualization system that helps composers analyze interaction and audio data from prior sessions with children with Autism Spectrum Disorder (ASD), informing future compositions. ResonaVis integrates audio features and interaction logs to capture how children engage with layered musical compositions, representing this engagement through coordinated visualizations of temporal transitions, layer co-occurrence, rhythmic activity, and spectral characteristics. Rather than prescribing strategies or supporting therapy sessions directly, the system surfaces patterns in past session data to support data-informed reflection during composition. We evaluated ResonaVis through a mixed-methods study with eight music students and a follow-up case study with two experienced composers. Results demonstrate good usability (SUS = 72.23), exceeding benchmarks for early prototypes, and show that participants could identify interaction patterns, reason about layer relationships, and make informed compositional decisions with high perceived performance and low frustration. Confidence in interpreting interaction and acoustic data for ASD-focused composition increased significantly across multiple dimensions (p < 0.05), with qualitative findings suggesting a shift toward more adaptive, data-informed composition. This work contributes a visualization design space for therapeutic music interaction data, an integrated system for compositional reflection, and empirical evidence that visualization tools support analytical reasoning and confidence in data-informed creative practice. (Abstract shortened for arXiv.)

Comment
9 pages core, 2 pages acknowledgements + references, 2 pages appendices. Accepted at IEEE VIS 2026, to appear in November 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Resonance and Depth: Composing Timbral Identity and Spatial Form in an Artist-in-the-Loop Co-Creative Multi-Agent Percussive Performance

cs.SD cs.MAcs.MM ·2026-10-21 ·Keon Ju Maverick Lee
published similarity 0.61
Abstract · 4 pages, 1 figure. Proceedings of Timbre 2026, 4th International Conference on Timbre, Montréal, Québec, Canada. Research-creation artistic …

This research-creation artistic report presents Resonance and Depth, a 9-minute immersive electroacoustic/acousmatic performance that treats timbre and space as primary compositional materials. Built from live electronic percussion and a personal corpus recorded, curated, and composed by the author, the work investigates how timbral identity and musical form can emerge through real-time recombination, spatialisation, and performer-system interaction rather than conventional harmonic development. The performance is realised as an artist-in-the-loop co-creative multi-agent system integrating MACAT, MACataRT, a MIDI-based musical agent, a spatialisation layer, and a conductor agent. These components recombine and transform the performer's material through concatenative synthesis, audio mosaicing, symbolic mediation, physically inspired synthesis, and object-based spatialisation. The work approaches spatialisation as an active compositional process in which position, trajectory, depth, diffusion, width, and apparent size participate in musical phrasing and form. Through this practice-based framework, Resonance and Depth examines how performer identity can be retained, transformed, and redistributed across human, algorithmic, timbral, and spatial processes within an immersive co-creative performance ecology.

0.36

Art2Song: Enhancing Visual Art Appreciation with Contextual Music Generation

cs.HC ·2026-09-05 ·Sungeun Jo, Myung Jin Kim, Chi Yoon Jeong
top venue similarity 0.59
Abstract · 3 pages, 1 figure, To appear in the Adjunct Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology (UIST Adj…

Art2Song is a conceptual framework that expresses artworks as sound by separating Non-Visual Context, which is difficult to perceive from the image alone, from Visual Evidence. Visual Evidence, such as objects, colors, and spatial composition, is transformed into Lyrics, while the Contextual Mood derived from the historical and art-historical context in the museum's artwork description is reflected in the background soundtrack. Rather than describing artworks textually, Art2Song aims to explore the possibility of a new mode of art appreciation in which viewers experience hidden stories and emotional context through music. As future interaction directions, we plan an Emotional Layer Blending Slider interface and a structured, traceable song-generation scenario, presenting the possibility that users can explore the relationship between Visual Evidence and Contextual Mood.

0.27

Exploring Breathing-Music Coupling: Using the Breathing Mirror for Somatic Reflection in Piano Performance

cs.HC ·2026-09-02 ·Ziyue Piao, Yohei Wada, Isabelle Cossette, Marcelo M. Wanderley, Akira Maezawa
published similarity 0.59
Abstract · Proceedings of the International Conference on New Interfaces for Musical Expression (NIME)

While breathing is essential to living and for sound production in some instruments, for pianists, it is often a hidden and automatic process, making it difficult to analyze or refine. A critical gap exists between data and awareness: while sensors record precise physical metrics, they fail to capture the performer's somatic experience. Conversely, the high cognitive load of performance makes it nearly impossible for musicians to recall their internal states with temporal precision. To address this, we present a system, Breathing Mirror, and associated methodology designed to externalize the pianist's internal somatic experience through three analytical lenses: a Baseline View (synchronized signals), a First-Person View (subjective recall), and an Interpersonal View (collaborative reflection). Through a four-week longitudinal study with a skilled amateur pianist (35 years of experience), we evaluated the system's effectiveness by recording respiratory data using textile-integrated strain sensor belts. The results show that the Breathing Mirror reveals some patterns of breathing-music coupling and identifies critical blind spots where objective data diverges from subjective perception. Furthermore, we propose four somatic themes regarding the link between breathing and musical elements, offering a foundation for future large-scale validation across a broader range of pianists. This work provides a new way to study body signals, transforming breathing from an internal biological function into an articulate expressive parameter.

0.62

Decomposing the Associative Chain: Schema-Based Mediation Supports Expression and Understanding in Human-AI Music Co-creation

cs.HC ·2026-11-14 ·Songruoyao Wu, Wanying Wu, Shuyu Li, Chongjun Zhong, Ziyi Huang +1 more
similarity 0.58
Abstract

Users cannot reliably translate their creative intents into AI-generated music. We contend that the root of this difficulty lies in associative divergence: associations between narratives and musical features are overly remote, and users lack a visible mediation structure to negotiate them. To address this, we introduce five core schema groups as cognitive mediation and propose a bidirectional schema-based framework. Its forward pathway decomposes narrative-to-music associations into staged mappings, while its reverse pathway provides structured interpretations from music back to narrative. Findings from two user studies(N = 188; N = 60) show that the framework led to clearer creative expression and greater perceived control, improved the comprehensibility of generated music, especially for unconventional pieces. Furthermore, it supported cultural and affective thinking while helping users locate associative divergence. Our work points toward better creative intent alignment in human-AI music co-creation.

0.30

Neuroscience of Improvisation Across Music and Other Domains: A Survey and Outlook

eess.SP ·2026-10-27 ·Levina Perzhilla, Muhammad Mahboob Ur Rahman, Tareq Y. Al-Naffouri
survey similarity 0.58
Abstract · 15 pages, 2 figures, 9 tables, submitted to IEEE for possible publication

Improvisation (or Improv), the real-time generation of novel, context-appropriate action, occurs in music, dance, theater, sport, education, therapy, and professional practice. Its neuroscience, however, has been studied almost exclusively for music, where most quantitative neuroimaging evidence lies. A PRISMA-style search done by authors quantifies this imbalance: of 111 identified studies, 70.3\% concern musical improvisation or general psychometric creativity. This explains why earlier surveys are either music-centered or confined to a single construct. This survey addresses that gap in two ways. First, for music, it organizes findings from functional magnetic resonance imaging (fMRI), electroencephalography (EEG), functional near-infrared spectroscopy (fNIRS), hyperscanning, neurofeedback, and brain stimulation, and reports transparently where the evidence agrees (e.g., experience-dependent changes in frontal engagement, increased frontal alpha and theta power) and where it is heterogeneous (e.g., the direction of lateral prefrontal activity, hemispheric lateralization, the interpretation of alpha power). Second, it extends the review to dance, theater, sport, education, therapy, and professional practice, where behavioral, qualitative, and simulation-based studies predominate. Overall, the current evidence does not point to a universal neural signature of improvisation within music domain or other domains. We conclude with an outlook on wearable EEG--fNIRS sensing and closed-loop brain--computer interfaces for musical improvisation, the domain with sufficient evidence, formulated as an expertise-gated control problem and discussed as a possible complement to deliberate practice, and we list the evidence needed for other domains.

0.38

MusGU+: Toward a Musician-Centered Evaluation Framework and Discovery Tool for Generative Music AI

cs.SD cs.AIcs.CYeess.AS ·2026-08-31 ·Laura Ibáñez-Martínez, Roser Batlle-Roca, Xavier Serra, Martín Rocamora
published similarity 0.58
Abstract · Accepted at AIMC 2026

Generative music systems are increasingly presented as tools that democratize music creation, yet their practical suitability for musicians remains underexplored. Prior work includes openness-focused evaluation frameworks, such as MusGO (Music-Generative Open AI), as well as qualitative studies of musicians' experiences with generative systems. However, these approaches do not support systematic comparison or early-stage discovery of models for creative use. Motivated by such limitations, we introduce MusGU+, a musician-centered evaluation framework organized around three dimensions: Adaptability, Usability, and Controllability. Together, these capture whether a model can be feasibly trained or fine-tuned on personal data, integrated into real-world music workflows, and controlled in musically meaningful ways. We evaluate 10 representative generative music systems and present an interactive discovery tool that enables musicians to explore and filter models according to these criteria. While MusGO remains valuable for promoting responsible research practices, MusGU+ supports informed selection and practical adoption of generative systems by musicians.

0.51

MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation

cs.SD cs.AIcs.MM ·2026-08-10 ·Shuyu Li, Kejun Zhang, Jiahe Lei, Shulei Ji, Zihao Wang +3 more
code/data similarity 0.58
Abstract

Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control. We have released the implementation as open source on GitHub at https://github.com/XaryLee/MusicLayout.

0.24

H2H Music Improv: A Communication Model and Audio-Visual Dataset for Music Improvisation

cs.SD cs.MMeess.AS ·2026-08-14 ·Aleksandra Teng Ma, Anthony Cammarota, Jiayi Wang, Alexandria Smith, Cheng-Zhi Anna Huang +2 more
published similarity 0.57
Abstract · Published in the Proceedings of the Society for Music Information Retrieval Conference (ISMIR) 2026

Current real-time AI improvisation systems lack the communication awareness human musicians rely on: rather than treating communication as a foundational algorithm design concern, most systems layer interaction strategies post-hoc onto generative algorithms through explicit controls and predefined modes. This gap persists in part because no formalized, machine-readable communication model with musicians exists. To address this, we study how expert musicians communicate in free (non-idiomatic) improvisation, unconstrained by prior discussion or agreement. Through a collaborative co-design process with expert improvisers, we derive a communication model that (1) captures how free improvisers negotiate musical ideas and enter stable musical spaces, and (2) is formalized as a machine-readable annotation scheme. We further present the H2H (Human-to-Human) Music Improvisation dataset: six hours of audio-visual expert duo improvisations with clean per-player stems and per-player annotations of both their own intentions and their perception of their partner's intentions. To our knowledge, this is the first such dataset for free improvisation. Together, the communication model and the dataset offer a new lens and resource for studying musician communication and may in future inform the design of AI musical partners that communicate by design.

0.19