CS Paper Radar

ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

Ashutosh Chaubey, Jinxi Guo, Minh Tran, Michael Andersland, Ozlem Kalinli

cs.SD ·submitted 2026-10-12 17:29 UTC · v1
benchmark

Abstract

Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.

Closest papers

Nearest neighbours by meaning across the whole index.

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

cs.SD cs.AIcs.LG ·2026-08-30 ·Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh, Utathya Aich, Ramani Duraiswami +1 more
top venuebenchmarkcode/data similarity 0.80
Abstract · Accepted at EMNLP 2026 Main Conference. Project page - https://kaousheik-26.github.io/tempo

Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present TEMPO (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech, and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, TEMPO outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements.

0.62

Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models

cs.SD cs.AIeess.AS ·2026-09-20 ·Jiaheng Dong, Xiaofeng Yu, Jean Honorio, Abhirup Ghosh, Hong Jia +1 more
similarity 0.77
Abstract

Large audio-language models (LALMs) are increasingly used for a broader range of audio reasoning tasks. These models typically incorporate audio representations into a large language model (LLM) backbone to enable multimodal reasoning. Recent test-time reinforcement learning (TTRL) methods further improve LLM reasoning capability by leveraging unlabelled test data after pre-training. However, the importance of the perceptual capability of LALMs remains underexplored, particularly how much acoustic evidence is integrated and relied upon during reasoning, and how this contributes to final task performance. This gap limits the development of effective post-training methods like TTRL for audio reasoning. In this work, we first analyse how audio information is integrated and utilised during reasoning process. We quantify layer-wise perceptual reliance and show that stronger acoustic reliance is associated with higher accuracy and a larger performance gain attributable to the audio input. Building on this, we propose Perception-Grounded TTRL (PG-TTRL), which aligns label-free test-time optimisation with perceptually grounded reasoning, encouraging the model to structure its reasoning more strongly on the audio input. Experiments across LALMs and benchmarks show that PG-TTRL consistently improves reasoning performance over both the base models and standard TTRL, showing the value of perceptual-grounding optimisation for test-time audio reasoning.

0.17

AudioMap: Cloze-and-Choice Reinforcement Learning for Time-Aware Dense Audio Captioning

cs.SD ·2026-08-10 ·Yan Rong, Fengji Ma, Xu Li, Jinting Wang, Chen Zhang +1 more
code/data similarity 0.76
Abstract

Time-aware dense audio captioning (TDAC) aims to generate multiple fine-grained attributes (dense) of the audio with precise time boundaries (time-aware). Existing methods struggle to achieve these two goals and mainly rely on supervised fine-tuning, yielding sub-optimal performance. While reinforcement learning (RL) shows promise, applying it to TDAC faces two main challenges: (1) existing rewards are too coarse to supervise multi-event, multi-attribute, and multi-relation descriptions in a fine-grained manner; and (2) temporal supervision is difficult for free-form captions, where flexible event-time expressions make reliable event-time correspondence challenging. To address these challenges, we propose AudioMap, a novel RL-based TDAC framework, which shifts to a unified cloze-and-choice reward paradigm. Specifically, we introduce the Evidence Sufficiency Reward (ESR) with an asymmetric hierarchical scoring mechanism to promote fine-grained accuracy and descriptive richness across diverse acoustic dimensions. Furthermore, we design the Event-Conditioned Temporal Reward (ECTR) to structurally bind timestamps to event semantics via temporal IoU, accompanied by a dual-curriculum learning strategy to facilitate the training process. Finally, to support this task, we construct the first time-aware fine-grained audio captioning dataset, AudioMapCap-44K, which contains 44K carefully annotated captions. Extensive experiments across diverse benchmarks show that AudioMap achieves state-of-the-art (SOTA) performance among open-source models and delivers competitive or superior results relative to proprietary models. Project page and release updates are available at https://github.com/ryysayhi/AudioMap.

0.60

TEMA: Evidence-Grounded Temporal Question Answering in Multi-Turn Multi-Audio Dialogs

cs.SD ·2026-09-24 ·Kaidi Yang, Hualei Wang, Zhaohui Wang, Chenxuan Wang, Hong Liu +1 more
similarity 0.75
Abstract · 5 pages, 2 figures

Multi-turn, multi-audio temporal question answering requires models to track target events across follow-up questions, recording switches, and historical references, recovering complete instances and their boundaries for temporal calculation and comparison. We propose TEMA, which connects event perception with evidence-based answering through Route, specifying the audio scope, and Span, describing all relevant intervals as conditional audio captions. We construct TEMA-Dialog with 40,704 dialogs and per-turn evidence and answer supervision, and TEMA-Bench for joint evaluation of evidence and final answers. Training combines temporal grounding initialization, full-dialog supervised fine-tuning, and completeness-first Span-only GRPO. Experiments on Qwen2.5-Omni and AF-Next show improved temporal question answering, particularly event localization and cross-audio comparison. Reinforcement learning applied solely to evidence further improves interval recovery and answer accuracy.

0.16

TiTok: Audio-Visual LLM for Multi-Segment Temporal Grounding

cs.CV ·2026-10-07 ·Eunji Shin, Dahyun Choi, Seungyeon Jo, Yejin Hong, Jiyoung Lee
code/data similarity 0.75
Abstract · ACCV'2026

Audio-visual multi-segment grounding (AV-MSG) in untrimmed videos, reasoning over audio-visual evidence and predicting multiple segments for a query, is a fundamental problem but remains challenging. Visual-only models overlook complementary acoustic cues, while audio-visual models often fail to calibrate the number of events - a phenomenon we refer to as count miscalibration. We present TiTok, an audio-visual large language model (AV-LLM) that localizes an arbitrary number of temporal event segments for each query. For precise boundary prediction, we introduce the Time Token Interleaving (TTI) method, which explicitly injects special time tokens into the audio-visual stream to align input-side temporal perception with output-side temporal prediction. We further propose decoupled, multi-segment-oriented rewards for reinforcement learning, consisting of global, local, count, precision, and format rewards, optimized with Group reward-Decoupled Normalization Policy Optimization (GDPO). To assess the performance on AV-MSG, we establish a new UnAV-100-based evaluation protocol, and propose the CountF1 metric for quantifying count miscalibration that overlap metrics fail to capture. TiTok reaches 65.7 mIoU and 0.58 CountF1, achieving state-of-the-art performance. Our code is available at this link.

0.25

Grounded in Sound: Reinforcement Learning with a Frozen Acoustic Judge to Curb ASR Insertion Hallucinations

cs.SD cs.MMeess.AS ·2026-09-13 ·Tingzhen Xiong, Rilin Chen, Weiwei Li, Wentao Zhang, Qicong Xie
similarity 0.74
Abstract · Accepted to IEEE Spoken Language Technology Workshop (SLT) 2026

When reinforcement learning (RL) is used for post-training automatic speech recognition (ASR), the reward almost always lives in the text space: it compares a hypothesis with the reference and never checks whether the hypothesis is supported by the audio. On highly regular speech this licenses a shortcut - guessing from a strong language prior rather than listening. Once the acoustics degrade, the shortcut runs unchecked and emits fluent but ungrounded words, i.e., insertion errors. We propose an acoustic-fidelity reward: a GRPO reward augmented with a separately pretrained, permanently frozen, non-autoregressive character-level wav2vec2-CTC acoustic judge, used strictly at training and absent at inference, where a single model decodes greedily. Trained on LibriSpeech and evaluated across a six-tier difficulty gradient including real AMI meeting speech (33,282 utterance-condition instances), the method reduces insertion errors by 28.3% on close-talking AMI-IHM and 22.3% on far-field AMI-SDM, while lowering WER on AMI-SDM from 35.89% to 34.71% and showing no detectable WER difference on the other five tiers, against a schedule-matched WER-GRPO baseline. The insertion reduction holds under a meeting-level clustered bootstrap. Four prespecified analyses support content-conditioned insertion calibration: output collapses 85-90% on unintelligible audio that preserves energy and voice activity; the gain is not recovered by the evaluated 32-best CTC rescoring configuration, yet RL internalizes it into a single greedy decoding run; and policy-only confidence yields lower insertion-AURC in all four evaluated settings. We frame this as a mechanism paper, demonstrated in one instantiation: a 7B speech LLM with a 0.3B CTC judge.

0.56

RefTAG: Reflection-aware Temporal Audio Grounding in Large Audio Language Models

eess.AS ·2026-10-12 ·Yuhang Dai, Xin Shu, Zhennan Lin, Mingchen Shao, Chengyou Wang +3 more
code/data similarity 0.74
Abstract · 5pages, 2 figures, 3 tables

Temporal audio grounding requires locating every occurrence of a queried target, making both occurrence counts and interval boundaries essential. We propose RefTAG, a post-training framework for verifiable temporal self-correction. Its Reflection Chain-of-Thought (Re-CoT) protocol combines draft intervals, discrete count and boundary actions, and revised intervals in one response. Supervised fine-tuning on these trajectories (Re-CoT-SFT) teaches the protocol; reinforcement learning (RL) jointly rewards final localization and revision outcomes. Across 8,228 queries, Re-CoT-SFT improves query-weighted mIoU over final-answer SFT by 1.74 percentage points. RL further improves TAG-Bench multi-target mIoU by 0.40 points and R@0.5 by 0.86 points. The final model achieves 38.50% TAG-Bench mIoU and 51.01% mean mIoU across three datasets, outperforming the evaluated open-source MOSS-Audio and FireRedAudio models and the commercial Gemini-3.1-Pro on both measures. These results demonstrate that explicitly coupling count- and boundary-aware revision with joint reward optimization provides an effective and verifiable path to temporal audio grounding.

0.20

Audio-R1: Audio Reasoning through Reinforcement Learning on Self-Supervised Tasks

cs.SD cs.AIeess.AS ·2026-10-25 ·Jihoon Jeong, Mirco Ravanelli, Zhepei Wang, Cem Subakan
similarity 0.74
Abstract

Reasoning is increasingly important for large audio-language models (LALMs), yet remains relatively underexplored compared with text-only and vision-language models. We introduce Audio-R1, a reinforcement learning (RL) post-training framework that revisits audio self-supervised learning as a source of verifiable rewards for improving audio understanding and reasoning task performance. Audio-R1 uses answers automatically determined during the construction of self-supervised tasks to verify model responses and compute rewards. Experiments across multiple LALMs and diverse audio benchmarks show consistent performance gains, while ablations demonstrate the benefits of each self-supervised task and additional gains from joint training. While both supervised fine-tuning and Audio-R1 improve self-supervised task accuracy, Audio-R1 further improves performance on downstream tasks distinct from the training tasks. These results highlight the potential of RL post-training with self-supervised audio tasks to improve downstream performance and reasoning-trace quality.

0.05