CS Paper Radar

On Temporal Binding in Large Audio Language Models

Paul Primus, Gerhard Widmer

cs.SD cs.LG ·submitted 2026-09-28 10:10 UTC · v1
code/data

Abstract

Reasoning about temporal structure of audio recordings requires Large Audio Language Models (LALMs) to associate sound events with their temporal position. Understanding the underlying mechanisms is a first step toward diagnosing failures and identifying model components that may need improvement. Using mechanistic interpretability, we investigate how temporal information is represented and bound to sound events in three open-source LALMs. We find that across all three, event-specific location becomes concentrated in event name representations at intermediate modality integration layers. These representations encode coarse event position along a low-dimensional, curved relative time trajectory. Steering event name representations along this trajectory systematically shifts before/after beliefs, providing evidence that these representations contribute to coarse temporal reasoning. In contrast, the same interventions do not reliably shift predicted onset timestamps, suggesting that coarse temporal reasoning and precise metric event localization rely on distinct mechanisms.

Comment
Repository: https://github.com/OptimusPrimus/icassp2027_temporal_binding This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Closest papers

Nearest neighbours by meaning across the whole index.

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception

cs.SD cs.AI ·2026-09-14 ·Yanfeng Shi, Yan Song, Junhui Li, Tinggan Huang, Wu Guo +2 more
similarity 0.79
Abstract · Submitted to ICASSP 2027

Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.

0.37

TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence

cs.SD eess.AS ·2026-10-15 ·Giovana Morais, Yujin Kim, Atilay Kucukoglu, Magdalena Fuentes
benchmark similarity 0.73
Abstract · currently under review

We introduce TRACE (Trustworthy Reasoning from Acoustic Cues and Evidence), a benchmark for diagnosing the reliability of large audio-language models (LALMs) by revealing when, and under what acoustic conditions these models behave untrustworthily. Existing benchmarks primarily measure whether a model produces the correct answer, but correctness alone cannot determine whether the answer is grounded in acoustic evidence or instead reflects language and contextual priors. TRACE evaluates model behavior along three complementary dimensions: (1) task performance, whether the answer is correct; (2) evidence alignment, whether the model is attending to the audio regions required to answer the question; and (3) robustness, whether answers and attributed evidence respond coherently to controlled changes such as event order, overlap, signal-to-noise ratio, or background content. TRACE targets temporal ordering, counting, and event existence, three tasks that remain challenging for current models. We synthesize contextually informed acoustic scenes that allow us to test whether models hallucinate events associated with the context but absent from the audio. This synthesis also allows us to obtain a scalable benchmark for which we know the answers by design. For each scene, we create controlled variants in a programmatic way to stress-test model robustness. We introduce attribution-based metrics that quantify model attention at scale on target events and non-target events. Evaluation across five state-of-the-art models shows that current LALMs often detect where relevant sound evidence occurs, but they do not reliably interpret it, integrate it into the answer, or respond to controlled changes in that evidence. Our benchmark also shows that models still struggle the most with ordering and counting questions.

0.20

Logbook: Extremely Long-form Audio Event Understanding

eess.AS cs.AIcs.CLcs.LG+1 ·2026-10-05 ·Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk, Guitang Lan, Chao-Wei Huang +5 more
benchmarkcode/data similarity 0.73
Abstract · Submitted to ICASSP 2027. Source code available at https://github.com/facebookresearch/logbook

Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we introduce Logbook, a benchmark for hour-scale audio understanding, with recordings ranging from ten minutes to six days. Given a continuous audio recording and an event label vocabulary, a system must predict a gap-free segmentation with an event label and a description per segment. We compare 52 systems, end-to-end and cascaded, and ablate fine-tuning, context length, and reasoning budget. We find the task tractable, though the best systems remain below the human reference. Also, over-segmentation is pervasive, and fine-tuning partially mitigates it. Finally, end-to-end are often better than cascaded systems, but degrades with longer context.

0.25

Probing Large Audio-Language Models for Compositional Understanding of Sounding Actions

cs.SD ·2026-09-28 ·Michel Olvera, Paraskevas Stamatiadis, Changhong Wang, Ga{ë}l Richard
top venuebenchmarkcode/data similarity 0.72
Abstract

Large audio-language models (LALMs) excel at understanding and reasoning tasks over atomic sound events, yet their ability to infer higher-level human activities from such fine-grained events remains largely unexamined. Everyday human actions and activities, such as setting a table, cleaning the house, or preparing a breakfast emerge compositionally from temporally distributed sound events, requiring abstraction beyond the event-centric granularity that dominates current training and evaluation paradigms. Our benchmark evaluates a wide set of LALMs under a principled framework that tests how language-based reasoning, grounded in acoustic perception, structures sound abstractions into higher-level understanding. By systematically varying exemplar typicality and distractor similarity, our evaluation exposes \added{that current models do not reliably perform compositional inference from atomic acoustic events to higher-level human activities solely from audio.} All data, taxonomies, and evaluation scripts are publicly available on our companion website: https://alm-sounding-actions.onrender.com/

0.51

What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability

cs.SD cs.CVcs.LG ·2026-09-11 ·Lucia Cascone, Valeria Fraenza, Michele Nappi, Fabio Narducci, Benedetto Simone
similarity 0.72
Abstract

Audio-based Multimodal Large Language Models (MLLMs) can generate detailed natural-language descriptions of complex acoustic scenes, yet it remains unclear which parts of the input audio support each generated token. This is particularly challenging because acoustic evidence is distributed across time and frequency, and concurrent sound events may overlap temporally while occupying different spectral regions. We introduce STAG, to our knowledge the first post-hoc framework for token-level spectro-temporal grounding of captions generated by audio-based MLLMs. STAG estimates the temporal support for each generated token using target-token-specific vocabulary projections of the encoded audio representations, measures frequency-band relevance through controlled spectral occlusion, and combines the two signals into a spectro-temporal relevance map. We evaluate STAG against ten post-hoc explanation methods across four grounding benchmarks, where it achieves the best event-localization performance on every dataset, and apply it to eight audio-language backbones without parameter updates. Counterfactual deletion further shows that removing the identified evidence selectively reduces confidence in the corresponding event and frequently removes it from the regenerated caption. These results provide behavioral support for the faithfulness and selectivity of the explanations.

0.22

VocalGrad: Evaluating Acoustic Perception in Audio Language Models

cs.SD cs.CL ·2026-10-16 ·Koki Ryu, Hitomi Yanaka
publishedbenchmarkcode/data similarity 0.70
Abstract · NeurIPS 2026 (Evaluations & Datasets Track). Dataset page: https://huggingface.co/datasets/ynklab/vocalgrad

Recent large audio language models (LALMs) have extended their capabilities beyond lexical speech understanding to tasks requiring paralinguistic and non-linguistic audio understanding, such as emotion recognition. However, these models still struggle to reliably utilize acoustic information, particularly when it co-occurs with spoken linguistic content. Existing benchmarks do not clearly distinguish acoustic perception from higher-level reasoning, making it difficult to identify the source of these failures. To address this issue, we introduce VocalGrad, a benchmark designed to evaluate LALMs' ability to perceive low-level acoustic variations. VocalGrad formulates this problem as a simple binary classification of temporal change direction, enabling evaluation largely independent of higher-level reasoning or semantic understanding. Our benchmark results reveal a substantial gap in acoustic perception between human annotators and state-of-the-art LALMs: while human annotators achieve high accuracy, LALMs perform at near-chance levels. We further conduct linear probing to identify structural bottlenecks in these models. Our analysis reveals two distinct failure modes: for some categories, task-relevant information is preserved in internal representations but not utilized by the language model head; for others, the information becomes progressively less linearly decodable across model layers. Overall, our results reveal substantial limitations of LALMs in modeling acoustic variation under controlled settings and expose representation differences not observable from text outputs.

0.39

CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning

cs.SD ·2026-08-25 ·Peiwei Ren, Jinbo Hu, Fang Kang, Shan Liang, Yin Cao
top venuehot direction similarity 0.69
Abstract

Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.

0.60

TAC2: Timestamped Audio Captioning as composable functions

cs.SD ·2026-10-27 ·Swapnil Bhosale, Prem Seetharaman, Ke Chen, Haonan Chen, Justin Salamon +1 more
similarity 0.68
Abstract · 22 pages

Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.

0.09