CS Paper Radar

CoTTAGE: An Explainable Judge for Text-to-Audio Generation

Yu-Xiang Lin, Yun-Man Hsu, Bo-Ru Lu, Kuan-Po Huang, Wenze Ren, Ming-To Chuang, Chun-Yi Kuan, Jiawei Du, Byeonggeun Kim, Qingming Tang, Chieh-Chi Kao, Ankur Gandhe, Chao Wang, Hung-yi Lee

cs.SD cs.AI ·submitted 2026-11-13 01:28 UTC · v1

Abstract

Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.

Closest papers

Nearest neighbours by meaning across the whole index.

VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

cs.CV ·2026-08-19 ·Yinming Huang, Shuyuan Tu, Xi Yan, Zihan Yang, Jianhua Han +4 more
benchmarkcode/datahighly cited similarity 0.72
Abstract · 19 pages, 7 figures, 8 tables. Code: https://github.com/ShareLab-SII/VA-Judger

Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.

0.45

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

cs.SD ·2026-09-09 ·Yupei Li, Qiyang Sun, Mohamed Mady, Chenxi Wang, Zhengwei Gong +2 more
publishedcode/data similarity 0.71
Abstract · Accepted at AACL 2026

Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.

0.40

FUSEMOS: Perceptual Evaluation of Text-to-Music Generation with Dual-Encoder Fusion and Ranking-Aware Composite Loss

eess.AS ·2026-11-05 ·Jing Yang, Haoyu Wang, Ningning Pan, Zhao Wang, Jianxuan Yang +1 more
publishedbenchmark similarity 0.69
Abstract · Accepted to ICASSP 2026

Perceptual evaluation of generated music is essential for developing text-to-music (TTM) generation systems. Existing automatic music perceptual evaluation methods rely solely on the contrastive language audio pretraining (CLAP) model to extract both audio and text features for predicting human-annotated mean opinion scores (MOS), particularly for overall musical impression (OMI) and textual alignment (TA). However, CLAP is limited in capturing fine-grained musical attributes such as timbre and expressiveness \emph{etc}. To address this, we propose FUSEMOS, a dual-encoder fusion framework that integrates CLAP with MERT, a self-supervised model pretrained on large scale music audio, to better capture both high level semantic alignment and detailed musical characteristics for more accurate OMI and TA estimation. Furthermore, to overcome the limitations of conventional regression losses in measuring human preferred relative ranking of music segments, we design a novel ranking-aware composite loss function. Experiments conducted on the MusicEval benchmark demonstrate that our approach outperforms the existing SOTA methods in both mean squared error and rank correlation metrics by a large margin.

0.38

Qwen-Audio-Jev: A Unified Interface for Audio Decisions

cs.SD cs.AI ·2026-10-15 ·Peixin Chen, Zhigao Chen, Zhiyun Fan, Ying Fang, Zhifu Gao +17 more
benchmark similarity 0.68
Abstract

Large audio language models (LALMs) have made substantial progress in audio understanding. However, autoregressive text generation introduces computational overhead and latency in high-throughput applications that require frequent selection, judgment, and scoring. To address this limitation, we introduce Qwen-Audio-Jev, which augments Qwen-Audio-3.1-ASR-Flash-Next with a dedicated Jev decision head. The model unifies selection, binary judgment, and scoring through three decision primitives: CHOICE, NOUL, and SCORE. Given an audio input, a natural-language question, and an explicit answer specification, Qwen-Audio-Jev directly outputs a task-conditioned probability distribution or a numerical score through typed decision readouts. Evaluation across 18 benchmarks demonstrates the model's effectiveness: CHOICE achieves 78.40 percent accuracy on MMAU and 83.14 percent on MMSU; NOUL achieves 93.76 percent accuracy on the VoxCeleb1-O speaker verification benchmark; and SCORE achieves a normalized mean absolute error of 0.1235 on NISQA overall quality assessment. These results highlight the potential of Qwen-Audio-Jev for efficient audio decision-making in applications such as large-scale audio screening, speaker verification, and automated quality assessment.

0.34

Rubric-Based Optimization for Text-to-Music Generation

cs.SD ·2026-10-02 ·Ping Wang, Guang Yang, Shao-Rong Su, Junkai Wu, Pang Wei Koh +1 more
similarity 0.68
Abstract · 27 pages

Post-training text-to-music generation requires reward signals that capture multiple aspects of musical quality beyond what any single automatic metric can measure. We study structured, rubric-based rewards from pretrained audio-language models (ALMs) as training signals for both autoregressive and diffusion-based music generators. An ALM scores each generated clip against the rubric; we rank candidates generated for the same text prompt by their scores and convert these rankings into preference pairs for DPO on both MusicGen-small and ACE-Step v1, and additionally use the rubric scores directly as scalar rewards for DiffusionNFT on ACE-Step v1. On MusicCaps, rubric-based optimization improves CLAP, SongEval, and Audiobox-Aesthetics simultaneously, with the strongest gains obtained by DiffusionNFT on ACE-Step. By contrast, on MusicGen-small, building preferences from any one of these automatic evaluators produces clear cross-metric trade-offs: the targeted evaluator improves while other independent evaluators deteriorate. We further study tempo, key, and instrumentation, where precise objective rewards are available. Directly optimizing these specialized rewards reliably improves the target attributes, whereas ALM rubrics provide only partial transfer for tempo and instrumentation and no measurable improvement for key. Together, these results suggest a practical division of labor: ALM rubrics are effective for broad perceptual qualities that are difficult to formalize, while specialized objective rewards remain preferable when reliable measurements are available.

0.16

Listen, Reason, and Segment: Aligning LALMs with Editorial Judgment for Media Chapterization

cs.SD cs.AIcs.CLeess.AS ·2026-08-17 ·Tony Alex, Wish Suharitdamrong, Sara Atito, Armin Mustafa, Muhammad Awais +3 more
benchmarkcode/data similarity 0.68
Abstract · 19 pages, 9 figures, 8 tables

Large Audio Language Models (LALMs) have made rapid progress on standardized benchmarks, yet their deployment in practical media workflows, curation, archival indexing, and content distribution remains largely unrealized. We identify automated audio chapterization, the task of segmenting continuous audio streams into thematically coherent chapters, as a demanding and commercially consequential setting that exposes this gap. Chapterization is challenging because boundaries are defined less by objective acoustic events than by subjective editorial judgment, requiring models to reason sequentially over long acoustic contexts and approximate creator-authored boundary decisions. We present AudioChaps, a post-training framework for aligning end-to-end LALMs for this task via Group Relative Policy Optimization (GRPO) guided by Chain-of-Thought (CoT) reasoning. To support training and evaluation, we curate three datasets: AudioChaps-Alignment, derived from creator-annotated chapter boundaries on YouTube; AudioChaps-CoT, which provides structured supervision for well-formatted, high-quality, and evidence-grounded boundary reasoning; and AudioChaps-Eval, a held-out benchmark for audio chapterization. Applying GRPO directly without a Supervised Fine-Tuning (SFT) cold start, AudioChaps-R1-Zero already improves average F1 by 33 points over the state-of-the-art LALM Audio-Flamingo-3-Think. The AudioChaps framework produces our final aligned LALM, AudioChaps-R1, which improves average F1 by 49 points. These results demonstrate that GRPO-trained LALMs can reliably transform unstructured auditory streams into navigable, structured media. Our code, models, and dataset resources will be released upon acceptance at https://github.com/ta012/AudioChaps.

0.41

REAL: Reward Modeling through Evidence-Linked Analysis of Lifelikeness in Generated Speech

eess.AS cs.SD ·2026-11-09 ·Yaodong Song, Jie Lian, Xusheng Yang, Genliang Zhao, Hongjie Chen +4 more
code/data similarity 0.67
Abstract

As modern text-to-speech systems become increasingly intelligible and natural, evaluation is shifting from detecting gross synthesis failures to identifying subtle departures from lifelike human delivery. Because large-scale listening tests are costly and slow, speech reward models can serve as scalable automatic judges for comparing generated utterances, curating data, and supporting iterative TTS development. Existing speech reward models can predict scores or pairwise preferences and may generate natural-language rationales, but their supervision is typically defined at the utterance or pair level. The connection between local acoustic phenomena and the final preference therefore remains implicit. We introduce REAL, a framework for Reward modeling through Evidence-linked Analysis of Lifelikeness in generated speech. Its central idea is to supervise not only the final pairwise preference but also a structured account of its supporting observations, linking local audio spans to perceptual phenomena, affected dimensions, and the overall judgment. Specifically, REAL-Jury constructs these evidence-linked reports through timestamp alignment, multi-model analysis, and dispute-oriented deliberation. REAL-Judge learns to generate them from paired audio inputs. REAL-Eval assesses preference prediction in two tiers, from salient perceptual differences to subtle distinctions among strong TTS systems. Instantiated with MiMo-Audio, REAL-Judge improves pairwise preference accuracy over its backbone by 16.01 and 11.41 percentage points on the two tiers, respectively, and outperforms the open-source speech reward models SpeechJudge and UniSRM on both tiers.

0.19

AudioScorer: Learning Question-Aware Token Relevance for Large Audio Language Models

cs.SD cs.CLeess.AS ·2026-10-19 ·Marwan Farag, Bin Yang
similarity 0.67
Abstract

Large Audio Language Models (LALMs) reason over speech, environmental sounds, music, and other acoustic information. However, their audio encoders produce dense token sequences that make long-form audio costly to process. Existing audio pruning methods reduce this cost using heuristics such as attention, representation similarity, or temporal sampling, but do not explicitly learn which tokens are most useful for a given question. We propose AudioScorer, a lightweight model that scores audio tokens according to the user question. It is trained by distilling the ranking of audio tokens under the frozen LALM's answer-gradient saliency, together with downstream answer supervision. Compared with other pruning methods, AudioScorer achieves the best or tied-best result at 10% token retention on 7 of 10 semantic and acoustic tasks, including speech recognition, emotion recognition, and audio-scene classification.

0.20