CS Paper Radar

Neuroscience of Improvisation Across Music and Other Domains: A Survey and Outlook

Levina Perzhilla, Muhammad Mahboob Ur Rahman, Tareq Y. Al-Naffouri

eess.SP ·submitted 2026-10-27 20:45 UTC · v1
survey

Abstract

Improvisation (or Improv), the real-time generation of novel, context-appropriate action, occurs in music, dance, theater, sport, education, therapy, and professional practice. Its neuroscience, however, has been studied almost exclusively for music, where most quantitative neuroimaging evidence lies. A PRISMA-style search done by authors quantifies this imbalance: of 111 identified studies, 70.3\% concern musical improvisation or general psychometric creativity. This explains why earlier surveys are either music-centered or confined to a single construct. This survey addresses that gap in two ways. First, for music, it organizes findings from functional magnetic resonance imaging (fMRI), electroencephalography (EEG), functional near-infrared spectroscopy (fNIRS), hyperscanning, neurofeedback, and brain stimulation, and reports transparently where the evidence agrees (e.g., experience-dependent changes in frontal engagement, increased frontal alpha and theta power) and where it is heterogeneous (e.g., the direction of lateral prefrontal activity, hemispheric lateralization, the interpretation of alpha power). Second, it extends the review to dance, theater, sport, education, therapy, and professional practice, where behavioral, qualitative, and simulation-based studies predominate. Overall, the current evidence does not point to a universal neural signature of improvisation within music domain or other domains. We conclude with an outlook on wearable EEG--fNIRS sensing and closed-loop brain--computer interfaces for musical improvisation, the domain with sufficient evidence, formulated as an expertise-gated control problem and discussed as a possible complement to deliberate practice, and we list the evidence needed for other domains.

Comment
15 pages, 2 figures, 9 tables, submitted to IEEE for possible publication

Closest papers

Nearest neighbours by meaning across the whole index.

Evaluating AI Musical Creativity: Towards a Quantitative Multi-Dimensional Framework Based on Value, Novelty, and Surprise

cs.SD cs.AIcs.CY ·2026-11-10 ·Hindy Rossignol, Giorgio Franceschelli, Mirco Musolesi
similarity 0.61
Abstract

Despite the increasing sophistication of generative Artificial Intelligence (AI), research on the evaluation of AI-generated musical creativity remains limited. To address this gap, this paper proposes a quantitative framework for the formalization and evaluation of AI musical creativity. Building on Margaret Boden's theory, which conceptualizes creativity in terms of value, novelty, and surprise, the framework adapts and operationalizes these three dimensions for the musical domain. On the hypothesis that certain aspects of musical creativity can be assessed with a degree of objectivity, a domain-specific set of quantitative metrics for evaluating the creativity of AI-generated music is introduced. Rather than reducing musical creativity to a single unifying measure, this article proposes an ensemble of complementary metrics that capture its different dimensions, recognizing the complexity of musical artifacts and the difficulty of defining meaningful multi-objective compositional criteria. The framework has two implications. First, it gives researchers and developers a principled methodology and quantitative tools for empirically evaluating creative outputs, enabling more informed model development and reducing unnecessary computational costs. Second, it lays the groundwork for future generative models to optimize directly for creativity by adopting these metrics as learning objectives.

0.20

MEX: The Musical Expectation Benchmark Suite

cs.SD cs.AIeess.AS ·2026-11-01 ·Mathias Rose Bjare, Augustin Bouquillard, Giorgia Cantisani, Seung-Goo Kim, Stefan Lattner +1 more
publishedbenchmarkcode/data similarity 0.59
Abstract · 9 pages, 2 figures, 3 tables. Accepted at the 26th International Society for Music Information Retrieval Conference (ISMIR), Abu Dhabi, Unit…

We propose the Musical Expectation (MEX) benchmark suite for perceptually validating computational models of musical expectation, i.e., models that estimate music information dynamics (MID) measures of perceived musical expectations, such as surprisal or uncertainty from symbolic music or musical audio. MEX collects and extends tasks for evaluating such models previously proposed in music information retrieval and cognitive sciences using standard datasets, and evaluates whether: (1) proposed MID measures decrease with repetition due to in-context learning of the computational model, (2) peaks in such measures align with boundaries of music segments, (3) pitch surprisal and uncertainty estimated from audio are consistent with those modeled in event-based symbolic music, (4) proposed MID measures can predict electrophysiological responses to music. We stress that this benchmark suite does not provide an exhaustive computational characterization of music expectation. Rather, we envision this as an active, interdisciplinary initiative, with benchmarks evolving as our understanding of the underlying perceptual processes deepens. We provide open-access code https://github.com/musicperceptmodeling/mex to estimate surprisal and uncertainty from audio using baseline models, run the tasks on standard datasets, and include utilities for running the tasks with users' own data.

0.41

H2H Music Improv: A Communication Model and Audio-Visual Dataset for Music Improvisation

cs.SD cs.MMeess.AS ·2026-08-14 ·Aleksandra Teng Ma, Anthony Cammarota, Jiayi Wang, Alexandria Smith, Cheng-Zhi Anna Huang +2 more
published similarity 0.59
Abstract · Published in the Proceedings of the Society for Music Information Retrieval Conference (ISMIR) 2026

Current real-time AI improvisation systems lack the communication awareness human musicians rely on: rather than treating communication as a foundational algorithm design concern, most systems layer interaction strategies post-hoc onto generative algorithms through explicit controls and predefined modes. This gap persists in part because no formalized, machine-readable communication model with musicians exists. To address this, we study how expert musicians communicate in free (non-idiomatic) improvisation, unconstrained by prior discussion or agreement. Through a collaborative co-design process with expert improvisers, we derive a communication model that (1) captures how free improvisers negotiate musical ideas and enter stable musical spaces, and (2) is formalized as a machine-readable annotation scheme. We further present the H2H (Human-to-Human) Music Improvisation dataset: six hours of audio-visual expert duo improvisations with clean per-player stems and per-player annotations of both their own intentions and their perception of their partner's intentions. To our knowledge, this is the first such dataset for free improvisation. Together, the communication model and the dataset offer a new lens and resource for studying musician communication and may in future inform the design of AI musical partners that communicate by design.

0.19

Musical Mirrors: The LLM as Sounding Board in Songwriting

cs.HC cs.AI ·2026-08-14 ·Xiao Xiao
similarity 0.59
Abstract · In Proceedings of The First Reflection in Creative Experience (RiCE) Workshop (RiCE W1) arXiv:2607.24558

This paper examines a use of AI in creative practice as an interpretive sounding board for human-generated material, rather than the more familiar pattern of AI generation followed by human curation. Through the lens of resonance as theorized by Hartmut Rosa, I present a first-person case study of songwriting from July 2025 to March 2026, drawing on 16 original pieces in English, French, and other languages along with piano solos. I describe a configuration in which resonance is not located between user and model, but in the author's deepening contact with their own material, mediated through the model. This kind of resonance was supported rather than inhibited by AI when sounding-board behavior was cultivated through sustained calibration by the user. Two failure modes appeared when calibration was absent: sycophantic drift and magical overinterpretation. This account suggests both the potential and the risks of AI as an interpretive partner in creative practice.

0.13

ResonaVis: Visualizing Interactive Music Data to Support Reflective Music Composition for Therapeutic Contexts

cs.HC ·2026-08-11 ·Abhishek Karwankar, Elise Ruggiero, Daniel Stevens, Matthew Louis Mauriello
published similarity 0.58
Abstract · 9 pages core, 2 pages acknowledgements + references, 2 pages appendices. Accepted at IEEE VIS 2026, to appear in November 2026

Designing music for therapeutic contexts requires navigating complex relationships between musical structure and listeners' sensory responses, yet composers often lack structured representations of these interactions, relying instead on intuition. We present ResonaVis, an interactive visualization system that helps composers analyze interaction and audio data from prior sessions with children with Autism Spectrum Disorder (ASD), informing future compositions. ResonaVis integrates audio features and interaction logs to capture how children engage with layered musical compositions, representing this engagement through coordinated visualizations of temporal transitions, layer co-occurrence, rhythmic activity, and spectral characteristics. Rather than prescribing strategies or supporting therapy sessions directly, the system surfaces patterns in past session data to support data-informed reflection during composition. We evaluated ResonaVis through a mixed-methods study with eight music students and a follow-up case study with two experienced composers. Results demonstrate good usability (SUS = 72.23), exceeding benchmarks for early prototypes, and show that participants could identify interaction patterns, reason about layer relationships, and make informed compositional decisions with high perceived performance and low frustration. Confidence in interpreting interaction and acoustic data for ASD-focused composition increased significantly across multiple dimensions (p < 0.05), with qualitative findings suggesting a shift toward more adaptive, data-informed composition. This work contributes a visualization design space for therapeutic music interaction data, an integrated system for compositional reflection, and empirical evidence that visualization tools support analytical reasoning and confidence in data-informed creative practice. (Abstract shortened for arXiv.)

0.25

Resonance and Depth: Composing Timbral Identity and Spatial Form in an Artist-in-the-Loop Co-Creative Multi-Agent Percussive Performance

cs.SD cs.MAcs.MM ·2026-10-21 ·Keon Ju Maverick Lee
published similarity 0.57
Abstract · 4 pages, 1 figure. Proceedings of Timbre 2026, 4th International Conference on Timbre, Montréal, Québec, Canada. Research-creation artistic …

This research-creation artistic report presents Resonance and Depth, a 9-minute immersive electroacoustic/acousmatic performance that treats timbre and space as primary compositional materials. Built from live electronic percussion and a personal corpus recorded, curated, and composed by the author, the work investigates how timbral identity and musical form can emerge through real-time recombination, spatialisation, and performer-system interaction rather than conventional harmonic development. The performance is realised as an artist-in-the-loop co-creative multi-agent system integrating MACAT, MACataRT, a MIDI-based musical agent, a spatialisation layer, and a conductor agent. These components recombine and transform the performer's material through concatenative synthesis, audio mosaicing, symbolic mediation, physically inspired synthesis, and object-based spatialisation. The work approaches spatialisation as an active compositional process in which position, trajectory, depth, diffusion, width, and apparent size participate in musical phrasing and form. Through this practice-based framework, Resonance and Depth examines how performer identity can be retained, transformed, and redistributed across human, algorithmic, timbral, and spatial processes within an immersive co-creative performance ecology.

0.36

Decomposing the Associative Chain: Schema-Based Mediation Supports Expression and Understanding in Human-AI Music Co-creation

cs.HC ·2026-11-14 ·Songruoyao Wu, Wanying Wu, Shuyu Li, Chongjun Zhong, Ziyi Huang +1 more
similarity 0.55
Abstract

Users cannot reliably translate their creative intents into AI-generated music. We contend that the root of this difficulty lies in associative divergence: associations between narratives and musical features are overly remote, and users lack a visible mediation structure to negotiate them. To address this, we introduce five core schema groups as cognitive mediation and propose a bidirectional schema-based framework. Its forward pathway decomposes narrative-to-music associations into staged mappings, while its reverse pathway provides structured interpretations from music back to narrative. Findings from two user studies(N = 188; N = 60) show that the framework led to clearer creative expression and greater perceived control, improved the comprehensibility of generated music, especially for unconventional pieces. Furthermore, it supported cultural and affective thinking while helping users locate associative divergence. Our work points toward better creative intent alignment in human-AI music co-creation.

0.30

Navigating relational agency in AI feedback musicianship: a diffractive dialogue

cs.SD cs.HC ·2026-11-10 ·Yichen Wang, Charles Patrick Martin, David Gunawan, David Cooper
similarity 0.55
Abstract

Musical AI systems behave in non-linear, dynamic ways that exceed the tool-oriented framings inherited from earlier instrument design. Existing frames of control and ensemble interaction do not articulate the entangled, relational agency that emerges when performers work with AI feedback instruments over time. We examine a duet practice with two AI-enabled feedback instruments, Sù and Agentier, in which feedback operates at the inner level of the AI's recurrent processes and the outer loop of musical interaction between performer, instrument, and co-performer. Our work draws on agential realism and a diffractive dialogue methodology to read our independent reflections through one another across five performances, examining how agency is enacted between performers and AI instruments. We find that control and agency are relational and shift across performances; musical textures enact feedback moments of control with performers; reaching a productive feedback environment is itself a shared exploratory act; and performers maintain their instruments through funnelling and injection, developing the skill of navigating agency along a continuous spectrum. Feedback musicianship with AI instruments demands navigation of this relational agency, displacing the leading-versus-responding binary that initially framed our reflections. This work contributes a materially-grounded vocabulary for AI feedback musicianship and a model of human-AI creative collaboration in which mutual agency is worked for through practice rather than designed in.

0.19