CS Paper Radar

How Well Do Generative Music Models Follow Emotion Conditioning?

Morteza Heydari

cs.SD eess.AS ·submitted 2026-08-30 19:24 UTC · v1

Abstract

Recent generative music models offer increasingly fine-grained control through text and audio conditioning, yet how faithfully they follow intended emotional cues remains an open question. We address this gap with a unified evaluation pipeline for emotion-following in generated music. Using all 1000 tracks in GTZAN, we extract semantic audio descriptions with DashengLM, an audio captioning model, and estimate source-track valence and arousal with Music2Emotion, a music emotion recognition model. We construct affect-aware text prompts by combining descriptions with top-ranked emotion tags and generate 30-second outputs with three systems, Stable Audio Open, MusicGen, and InspireMusic, evaluating both text- and audio-conditioned generation. To measure emotion-following, we compute valence and arousal on generated audio and compare them with the source tracks using absolute error and Euclidean distance in valence-arousal space. Text-conditioned generation consistently outperforms audio conditioning, with MusicGen (text) and InspireMusic (text) achieving the best performance, while audio-conditioned variants prove less stable. We further find that valence is preserved more reliably than arousal and that emotion-following varies substantially across genres. These findings underscore the importance of evaluating affective controllability directly rather than relying solely on general quality or prompt-relevance metrics.

Comment
7 pages, 3 figures, 2 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Predicting Music Valence and Arousal via Textual and Multimodal In-Context Learning in Large Language Models

cs.SD cs.AIcs.CL ·2026-10-22 ·Chi-En Dai, Chih-Pin Tan, Yi-Hsuan Yang
similarity 0.79
Abstract · Submitted to ICASSP 2027

While arousal can be reliably predicted from musical audio, emotional valence remains difficult to model due to its subjective and context-dependent nature. In this work, we investigate the potential of in-context learning (ICL) with large language models to advance continuous valence and arousal prediction. Rather than predicting emotion scores directly, our framework leverages contextual demonstrations from an auxiliary dataset to refine the initial predictions produced by a small, task-specific base model. We systematically evaluate a text-based ICL variant using lyrics and generated audio captions, alongside a multimodal variant that incorporates raw audio signals. Using the entire MER60 dataset strictly as a held-out test set, our experiments demonstrate substantial performance gains, particularly for valence. Text-based ICL improves the valence $R^2$ from 0.110 to 0.427 with 80 demonstration shots, while the multimodal variant achieves 0.346 with only 10 shots. These findings suggest that training-free ICL offers an effective inference-time strategy for current language models to predict continuous music emotion.

0.19

Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals

cs.SD ·2026-09-29 ·Xiaosha Li, Chun Liu, Ziyu Wang
published similarity 0.72
Abstract · Accepted by the 27th International Society for Music Information Retrieval Conference (ISMIR 2026)

Current music generative models can produce high-quality music, but does this ability imply that they ``understand'' the musical qualities of their outputs, and is that understanding aligned with human evaluation? Previous attempts to use the likelihood of a generative model to evaluate music, an approach commonly used in text, have proven unsuccessful, leading researchers to rely on standalone supervised music evaluation models. In this paper, we answer this question affirmatively: we show that a model's intrinsic signals---derived from its hidden representations and predictions---are strongly correlated with human ratings. In particular, we study MusicGen and consider three types of features: (1) prediction loss, (2) prediction entropy, and (3) concepts extracted from the model using a sparse autoencoder (SAE). Using these features, we train a lightweight prediction model to estimate subjective ratings. We evaluate these features both individually and in combination. We hypothesize that these signals parallel the listening process: the temporal and frequency-domain structure of loss and entropy reflects listeners' expectation and surprise, while gradient directions in SAE latent space predict perceived quality. Experiments on five human-evaluation benchmarks spanning continuous ratings and pairwise preferences confirm this hypothesis, with SAE latents carrying most of the predictive signal.

0.49

Concordance-Optimal Continuous Music Emotion Estimation: Variance Decomposition, Dual-Branch Temporal Fusion, and Conformal Prediction Intervals

cs.SD cs.LG ·2026-11-06 ·Mohammad Faraz Akhtar, Aditya Parashar, Ayush Jain, Madhur Gupta, Ritesh Kumar +1 more
code/data similarity 0.71
Abstract · 28 pages, 12 figures

Continuous music emotion recognition (MER) estimates how Arousal and Valence evolve within a song, yet the field's standard metric, pooled frame-level correlation, is dominated by between-song differences and cannot distinguish a model that tracks emotional dynamics from one that only ranks songs by their average level. We decompose annotation variance into between- and within-song components and show that mean squared error (MSE) targets the former, while windowed Lin's concordance correlation coefficient (CCC = $r \cdot C_b$) targets the latter by penalising the variance compression MSE induces. A single weight $λ$ controls the trade-off: on 1,764 DEAM songs, raising $λ$ from 0 to 1 lowers pooled Arousal $r$ from 0.843 to 0.810 while raising mean within-song $r$ from 0.204 to 0.368. Building on this objective, we present MMAS (Multimodal Music Analysis System), an open-source model that predicts Arousal and Valence every 0.5 s from raw audio. It fuses frozen MERT-v1-95M embeddings with ten symbolic features from automatic transcription through a dual-branch temporal convolutional network with cross-modal attention. Because both branches are audio-derived, one model trains jointly on DEAM and PMEmo without domain conditioning. Under song-grouped 5-fold cross-validation (2,531 songs), pooled $r$ reaches 0.801/0.685 (Arousal/Valence) on DEAM and 0.742/0.746 on PMEmo. Against a ridge baseline on identical features, within-song Arousal $r$ roughly triples (0.344 vs. 0.125; $n = 353$ songs, Wilcoxon $p = 1.7 \times 10^{-21}$, $d = 0.56$). Complete transcription failure costs at most 0.04 pooled $r$. Split-conformal intervals attain 93.7% (Arousal) and 90.0% (Valence) marginal coverage at a 95% target, with the shortfall traced to within-song dependence. MMAS is released as a pip-installable package.

0.27

TASTE: Taxonomy-Aware Symbolic Piano Generation from Textual Description

cs.SD ·2026-10-11 ·Mingyang Yao, Jingyue Huang, Ke Chen, Taylor Berg-Kirkpatrick, Shlomo Dubnov
hot direction similarity 0.70
Abstract · Under Reivew

Text-conditioned symbolic music generation has attracted growing interest with the development of generative models. Yet two questions remain underexplored: what aspects of music the conditioning text should describe, and how the corresponding data should be collected and trained to improve controllability and text-music alignment. We address these challenges with TASTE, a taxonomy-aware framework for symbolic music generation with a main focus on piano pieces. First, we introduce a taxonomy of musical elements that specifies how a piece can be captured through textual description in a musically grounded way. The taxonomy comprises a set of metadata tags (genre, emotion, section plan, etc.) together with a free-form description of music imagery. This structure decomposes textual input to match the generation goal more coherently and precisely. Second, we build a data pipeline for collecting, annotating, and retrieving public symbolic piano resources, yielding a corpus of 432K MIDI files labeled at multiple levels of granularity. Finally, we design a staged training procedure that leverages this multi-granularity data across two model families: an autoregressive model and a non-autoregressive diffusion model. Both objective and subjective evaluations show that our models achieve superior musical-element control, text-description alignment, and overall music quality. We will release the full pipeline, all annotated datasets, and the model checkpoints.

0.20

Rubric-Based Optimization for Text-to-Music Generation

cs.SD ·2026-10-02 ·Ping Wang, Guang Yang, Shao-Rong Su, Junkai Wu, Pang Wei Koh +1 more
similarity 0.68
Abstract · 27 pages

Post-training text-to-music generation requires reward signals that capture multiple aspects of musical quality beyond what any single automatic metric can measure. We study structured, rubric-based rewards from pretrained audio-language models (ALMs) as training signals for both autoregressive and diffusion-based music generators. An ALM scores each generated clip against the rubric; we rank candidates generated for the same text prompt by their scores and convert these rankings into preference pairs for DPO on both MusicGen-small and ACE-Step v1, and additionally use the rubric scores directly as scalar rewards for DiffusionNFT on ACE-Step v1. On MusicCaps, rubric-based optimization improves CLAP, SongEval, and Audiobox-Aesthetics simultaneously, with the strongest gains obtained by DiffusionNFT on ACE-Step. By contrast, on MusicGen-small, building preferences from any one of these automatic evaluators produces clear cross-metric trade-offs: the targeted evaluator improves while other independent evaluators deteriorate. We further study tempo, key, and instrumentation, where precise objective rewards are available. Directly optimizing these specialized rewards reliably improves the target attributes, whereas ALM rubrics provide only partial transfer for tempo and instrumentation and no measurable improvement for key. Together, these results suggest a practical division of labor: ALM rubrics are effective for broad perceptual qualities that are difficult to formalize, while specialized objective rewards remain preferable when reliable measurements are available.

0.16

MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques

cs.SD cs.CL ·2026-08-12 ·Jiabao Zhuang, Changhao Jiang, Hanchen Wang, Jiahao Chen, Zhixiong Yang +9 more
benchmarkcode/data similarity 0.67
Abstract

Long-form song generation models continue to improve in duration, structural coherence, and acoustic complexity, increasing the need for reliable aesthetic rewards aligned with human preferences. However, reward models for complete songs remain limited, and existing evaluators typically predict scores in a single forward pass without readable explanations. To this end, we introduce MuseCritic, a semi-scalar reward model that generates a natural-language critique covering five aesthetic dimensions and uses it as an intermediate representation to predict continuous reward scores. MuseCritic follows a two-stage training pipeline: a teacher model first provides high-quality critiques for supervised fine-tuning, then the fine-tuned model generates its own critiques for reward learning, mitigating training-inference distribution shift. On an in-domain test set of 200 SongEval songs, MuseCritic reduces macro-averaged mean squared error from 0.2875 to 0.2316 and improves macro-averaged LCC, SRCC, and Kendall's tau to 0.9068, 0.8838, and 0.7178, respectively. On the out-of-domain Music Arena benchmark with 733 preference pairs, it achieves 71.35% accuracy and remains competitive with strong music-specific reward models. Using MuseCritic with GRPO also improves Muse-0.6B on all nine aesthetic metrics from SongEval and Audiobox Aesthetics. These results show that critique-conditioned reward modeling reduces scoring error and provides an effective optimization signal for song generation. The project repository is available at https://github.com/WuqnEl/MuseCritic.

0.24

EmoTrace-Bench: Diagnosing Emotional Expression in Text-to-Video Generation through Controlled Cues and Intensity

cs.CV ·2026-10-11 ·Yeheng Wang, Liudan Shi, Yi Zheng, Jingke Li, Shiyuan Liu +3 more
benchmarkcode/data similarity 0.67
Abstract · 30 pages. Code and data: https://github.com/Tonyheng6/EmoTrace-Bench

Text-to-video (T2V) models are increasingly used to generate videos with targeted emotional expressions, yet existing evaluation methods mainly rely on holistic video-text alignment and provide limited insight into whether failures arise from target-emotion expression, intensity control, or specific affective cues. We introduce EmoTrace-Bench, a fine-grained diagnostic benchmark that controls emotion intensity and selected affective cues in prompts and evaluates generated videos at multiple levels. EmoTrace-Bench contains 448 controlled prompts covering eight emotions, two intensity levels, and diverse cue conditions. Using 12 T2V models, we generate 5,376 videos and evaluate target-emotion presence, perceived intensity, and ten audiovisual dimensions of emotional expression with an automated evaluator calibrated and validated against human annotations on a subset of the data. Results reveal substantial differences across models and prompt designs. Prompts that explicitly describe a character's emotional reaction are followed more reliably than prompts that describe only events or scenes, whereas adding more affective cues or using more explicit cinematic audiovisual language does not consistently improve emotional generation. These results illustrate how aggregate alignment scores may overlook localized differences in target-emotion expression, intensity control, and cue realization. EmoTrace-Bench provides a traceable and interpretable framework for diagnosing emotional video generation without modifying existing generation pipelines, and we release the prompts and the automated evaluation rules.

0.35

FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss

eess.AS ·2026-11-05 ·Jing Yang, Haoyu Wang, Ningning Pan, Zhao Wang, Jianxuan Yang +1 more
publishedbenchmark similarity 0.67
Abstract · Accepted to ICASSP 2026

Perceptual evaluation of generated music is essential for developing text-to-music (TTM) generation systems. Existing automatic music perceptual evaluation methods rely solely on the contrastive language audio pretraining (CLAP) model to extract both audio and text features for predicting human-annotated mean opinion scores (MOS), particularly for overall musical impression (OMI) and textual alignment (TA). However, CLAP is limited in capturing fine-grained musical attributes such as timbre and expressiveness \emph{etc}. To address this, we propose FUSEMOS, a dual-encoder fusion framework that integrates CLAP with MERT, a self-supervised model pretrained on large scale music audio, to better capture both high level semantic alignment and detailed musical characteristics for more accurate OMI and TA estimation. Furthermore, to overcome the limitations of conventional regression losses in measuring human preferred relative ranking of music segments, we design a novel ranking-aware composite loss function. Experiments conducted on the MusicEval benchmark demonstrate that our approach outperforms the existing SOTA methods in both mean squared error and rank correlation metrics by a large margin.

0.38