CS Paper Radar

Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis

Rupali Rajesh, Sethuraman T, Savya Khosla, Srinidhi Sunkara, Aditya Shanmugham, Rohan Maheshwari, Derek Hoiem

cs.SD ·submitted 2026-10-21 01:39 UTC · v1

Abstract

Audio-language models remain unreliable at extracting basic musical properties from audio, motivating pipelines that place a music information retrieval (MIR) tool between the audio and the model. The tool reports measurements such as keys, chords, and structural similarity, and the model is expected to derive the answer. We test whether it can. Holding the musical information fixed, we vary derivation depth: the number of reasoning steps between the tool output and the answer. When the report states the conclusion, all five models score near 100%, and they follow it even when it is wrong. One step away, accuracy drops. On modulation, open models fall to 57-81%, although they answer that single step correctly when asked directly, and reach 0% at greater depths. Frontier models degrade less, but still degrade. Adding the audio does not recover the loss. The bottleneck in tool-augmented music analysis is not access to musical information but reasoning from it, so tool design must account for how much derivation it leaves to the model.

Closest papers

Nearest neighbours by meaning across the whole index.

HMU-Bench: Hierarchical Music Understanding Benchmark for Omni Models

eess.AS ·2026-10-11 ·Zheqi Dai, Boyang Wang, Zhiyue Wu, Yiyang Xu, Xinming Shi +4 more
benchmarkcode/data similarity 0.69
Abstract · 5 pages

Music-understanding benchmarks should identify both the evidence available to a model and the operations a correct answer requires. We introduce HMU-Bench, a bilingual benchmark with notation and audio components. ABC Q&A contains 1,029 Chinese-English question pairs across six musical domains, labeled as extraction, single-step inference, or multi-step inference. Audio Q&A contains 460 question pairs on 92 recordings, spanning five scope-difficulty configurations. All music is Suno-generated, and questions and references are human-reviewed. Reference-based evaluation of seven omni models yields high notation scores for the strongest systems, but no model averages above 1.4 on the 0-4 audio scale. Notation and audio rankings show little correlation. A transposition case study also reveals octave errors missed by LLM grading. These findings motivate separate modality reporting and event-level checks for symbolic outputs. Data, annotations, evaluator prompts, and graded responses will be released at https://github.com/stepaudiollm/hmu-bench.

0.30

PianoBench: Evaluating Musical Perception and Reasoning in Language Models

cs.SD cs.AIcs.LG ·2026-11-02 ·Louis Bradshaw, Irmak Bukey, Daniel Chenyu Lin, John Thickstun, Chris Donahue
benchmark similarity 0.68
Abstract

Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical performances across congruent acoustic and symbolic input representations. PianoBench is derived from our new multimodal, training-scale dataset comprising more than 20,000 hours of interleaved transcripts of music and spoken musical commentary from instructional piano videos. Open-weight audio-language models achieve at most 52.8% accuracy on PianoBench, and at best only modestly outperform their accuracy on a no-audio control. In contrast, frontier models using chain of thought achieve up to 93.2% accuracy when provided transcripts of music as text, exposing a gap between perceiving musical features from audio and reasoning about musical events when they are given explicitly. We then test whether mid-training on interleaved musical instruction can improve question answering without chain of thought, raising Qwen 3.5 9B Base's zero-shot accuracy on PianoBench with music transcripts from 42.8% to 56.0% after instruction tuning. These experiments suggest that instructional videos may represent an overlooked source of multimodal supervision for music. Available: https://gclef-cmu.org/pianobench.

0.17

TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

cs.SD cs.LGcs.MMeess.AS ·2026-09-16 ·Giorgia Adorni, Michela Papandrea, Battista Rimoldi, Tiziano Leidi
similarity 0.67
Abstract

Text-to-music (TTM) systems are increasingly used to generate musical audio from natural-language descriptions. Robust evaluation is therefore essential, yet reliable performance comparison remains challenging. This difficulty stems from differences in system architecture, supported conditioning information, and access mode, as well as heterogeneous and fragmented metrics that cannot be applied uniformly across systems. To address these challenges, we introduce TTM-Bench, a framework that defines a common protocol for systematic, reproducible performance benchmarking of contemporary TTM systems. It evaluates performance along two dimensions: musical-content alignment, quantified by interpretable semantic, genre, and musical-descriptor agreement scores against a common musical specification and summarized by an aggregate score; and computational efficiency, characterized by generation latency and real-time factor, alongside resource use for local models and cost for hosted services. We demonstrate the framework through a preliminary comparative case study, illustrating the complementary evidence captured by these dimensions. The results show that higher musical-content alignment does not systematically coincide with lower computational demands, highlighting the importance of assessing TTM performance through distinct, interpretable measures rather than a reductive overall indicator.

0.22

MuSP-Bench: Advanced Multimodal Benchmarking of Music Understanding across Score and Performance

cs.MM cs.SD ·2026-08-28 ·Milan Liessens Dujardin, Song-Ze Yu, Kevin Miao
publishedbenchmark similarity 0.66
Abstract · Accepted to ISMIR 2026 Late-Breaking Demo. Benchmark site: https://musp.vaclis.net/

Musicians commonly communicate musical intent through scores and performances. To investigate whether models can reason across both modalities, we introduce MuSP-Bench, a human-authored benchmark of 490 questions assessing understanding across Musical Scores and Performances, both individually and in combination. The benchmark spans interpretive and long-horizon reasoning across classical piano and orchestral works, represented through ABC notation, score images, MIDI, and audio. Our evaluation of frontier multimodal large language models reveals substantial difficulties in understanding musical scores and even greater challenges in reasoning about audio.

0.56

Note-Level Temporal Grounding of Musical Concepts in Large Audio-Language Models

cs.SD cs.IR ·2026-08-30 ·Kun Fang, Ziyu Wang, Ichiro Fujinaga
benchmark similarity 0.66
Abstract

Large audio-language models (LALMs) demonstrate growing music-understanding capabilities, but whether their responses are grounded in acoustic evidence remains unclear. Musical language often involves abstract concepts whose acoustic evidence is difficult to define and evaluate precisely. We introduce MusicGroundingBench, a controlled benchmark of algorithmically generated piano audio with exact symbolic alignment, comprising three-note and two-bar settings. We evaluate two complementary capabilities: grounding, which localizes the acoustic evidence for a musical query, and understanding, which answers questions about the same excerpts. Our experiments show that cross-modal fine-tuning enables models to learn each capability, but adding grounding supervision does not consistently improve understanding across backbones. We further test whether understanding requires listening through audio-ablation controls that remove or replace the input audio, and use attention analysis to examine whether grounding supervision shifts attention toward note boundaries. Meanwhile, the two evaluated LALMs show limited zero-shot grounding even for basic musical concepts, highlighting grounded music understanding as an important open challenge.

0.20

YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

eess.AS cs.LGcs.SD ·2026-09-27 ·Ruibin Yuan, Jiahao Pan, Junyan Jiang, Zhiyue Wu, Ziya Zhou +30 more
benchmarkcode/data similarity 0.65
Abstract · 56 pages. Technical report. Project: https://github.com/multimodal-art-projection/YuE

Symbolic models make melody, harmony, rhythm, and form explicit but typically stop before a finished recording; audio models produce complete songs while leaving composition implicit. We introduce YuE2, which unifies symbolic and audio music generation at frontier quality through symbolic planning. A single AR-NAR Mixture-of-Transformers (MoT) first writes a readable score specifying melody and harmony, expands it into semantic music tokens, and realizes it as full-song audio. In comparisons using the same checkpoint, experts prefer symbolic planning for overall quality and musicality, with 49.3% of overall preferences versus 34.6% without planning. Experts also favor the unified model over a separate language model and diffusion Transformer. On WildSongBench, YuE2 scores 6.73 on SongBench Global Avg, exceeding all evaluated public baselines. Selecting from eight candidates (best-of-8), YuE2 reaches 6.96, the highest observed mean among all evaluated systems. Expert listening further establishes its competitiveness with proprietary song generators, favoring best-of-8 over Suno v4.5 and yielding nearly balanced preferences against Suno v5. To learn this generation process from recordings without aligned scores, we introduce MERT2 and SheetSage2 to supply semantic and symbolic supervision. MERT2 sets a new state of the art in music representation learning, surpassing previous best results on 14 of 15 MARBLE metrics; SheetSage2 leads 12 of 15 benchmark-metric pairs in our lead-sheet transcription comparison. The same checkpoint follows score edits while largely preserving unedited musical content and generates zero-shot covers without cover-specific training. Its readable score also enables agentic music editing, with external language models translating user feedback into revisions of the composition.

0.59

Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals

cs.SD ·2026-09-29 ·Xiaosha Li, Chun Liu, Ziyu Wang
published similarity 0.65
Abstract · Accepted by the 27th International Society for Music Information Retrieval Conference (ISMIR 2026)

Current music generative models can produce high-quality music, but does this ability imply that they ``understand'' the musical qualities of their outputs, and is that understanding aligned with human evaluation? Previous attempts to use the likelihood of a generative model to evaluate music, an approach commonly used in text, have proven unsuccessful, leading researchers to rely on standalone supervised music evaluation models. In this paper, we answer this question affirmatively: we show that a model's intrinsic signals---derived from its hidden representations and predictions---are strongly correlated with human ratings. In particular, we study MusicGen and consider three types of features: (1) prediction loss, (2) prediction entropy, and (3) concepts extracted from the model using a sparse autoencoder (SAE). Using these features, we train a lightweight prediction model to estimate subjective ratings. We evaluate these features both individually and in combination. We hypothesize that these signals parallel the listening process: the temporal and frequency-domain structure of loss and entropy reflects listeners' expectation and surprise, while gradient directions in SAE latent space predict perceived quality. Experiments on five human-evaluation benchmarks spanning continuous ratings and pairwise preferences confirm this hypothesis, with SAE latents carrying most of the predictive signal.

0.49

A Tutorial on Generative Models for Music Audio

eess.AS cs.AIcs.ETcs.MM+1 ·2026-10-28 ·Mateusz Modrzejewski, Joanna Kurkowska, Dorien Herremans
similarity 0.64
Abstract

Music audio generation spans instrumental clips, full songs with vocals, edits of existing recordings, and systems that play alongside a musician in real time. This monograph explains the methods behind these systems for readers familiar with neural-network training who are new to music generation. It introduces neural audio codecs and autoencoders, autoregressive and masked token models, diffusion and flow matching, sampling acceleration, and preference optimization in a shared notation. Worked examples connect training objectives and sampling procedures to published systems. The application sections relate the musical requirements of instrumental generation, songs, editing, and live performance to model design. Comparison tables record representations, methods, inputs, tasks, and release information documented in papers and official sources. Symbolic composition, speech, and sound effects are included where they connect to music audio. We explain how to interpret automatic metrics and learned judges, design and report listening tests, and assess evidence about data provenance, memorization, attribution, and output detection. Selected legal decisions and disputes illustrate the implications for training and release. The reviewed systems combine established generative paradigms with new representations, planning stages, and post-training procedures. Differences in data, scale, and evaluation limit what cross-system comparisons can establish about the sources of reported gains.

0.19