CS Paper Radar

Auditory Illusion Benchmark for Large Audio Language Models

Hayoon Kim, Eunice Hong, Kyogu Lee

cs.SD cs.AI ·submitted 2026-09-02 08:23 UTC · v1
publishedbenchmarkcode/data

Abstract

Perceptual illusions have long served as crucial probes into human cognition, revealing biases and limitations of perception. In the auditory domain, such illusions provide a unique lens for testing whether Large Audio Language Models (LALMs) replicate human perceptual tendencies. Despite their importance, most benchmarks focus on visual illusions or general audio tasks, leaving auditory illusions underexplored. To this end, we present AIB, the first auditory illusion benchmark for LALMs, covering ten representative illusions across music, sound, and speech, each annotated for the presence of knowledge-based priors. Our methodology pairs model evaluation with controlled human listening studies, enabling direct comparison of responses. Results show systematic differences: while most LALMs remain signal-faithful on low-level acoustic illusions, several exhibit more human-like responses when linguistic or musical priors are involved, although no model matches the human perceptual profile. These findings highlight the current limitations of LALMs as cognitive models. By establishing auditory illusions as a rigorous testbed, our work offers a new perspective for probing neural black-box models and advancing understanding of auditory cognition. AIB is publicly available at https://github.com/gillosae/aib.

Comment
Accepted to the 2026 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2026)
DOI
10.1109/icassp55912.2026.11462423

Closest papers

Nearest neighbours by meaning across the whole index.

PADP: Perceptual Audio Data Perturbation for Probing Perception Awareness in Audio Quality Models

eess.AS ·2026-10-01 ·Guanxin Jiang, Andreas Brendel, Pablo M. Delgado, Jürgen Herre
similarity 0.66
Abstract · 5 pages, 3 figures

This paper presents a collection of audio transforms that introduce perceptually irrelevant distortions and demonstrates their use as perceptual stress tests for audio quality models. We refer to these methods as Perceptual Audio Data Perturbation (PADP). PADP exploits the insensitivity of the human auditory system to certain fine-grained signal variations to substantially alter the waveform while preserving perceived audio quality and content. The audibility of PADP and the selection of its parameters are evaluated through controlled listening tests, ensuring that the transformations achieve transparent or near-transparent quality for both non-critical and critical items. We further probe the robustness of state-of-the-art (SOTA) perception-motivated objective audio quality models and foundation models. The results reveal a misalignment between model responses and human auditory perception, highlighting the limited perceptual awareness of these models for certain proposed transforms.

0.25

AnchorPrompt: Self-Distilled Soft Prompts for Robust Audio-Language Models

cs.SD cs.LG ·2026-09-30 ·Pooneh Mousavi, Amir Ivry, Mirco Ravanelli, Cem Subakan
benchmark similarity 0.65
Abstract

Large audio-language models (LALMs) are sensitive to input perturbations, such as noise, waveform corruption, and adversarial injections. We propose AnchorPrompt, an efficient adaptation method that keeps the model frozen and learns a single block of prompt vectors inserted at the decoder input, between the audio and question embeddings. We train these vectors through self-distillation over diverse audio and text perturbations. To improve answer consistency and mitigate hallucination, we use the model's prediction on the clean recording as the target for answerable inputs, and assign a refusal target when the audio lacks sufficient evidence to answer. Furthermore, AnchorPrompt is perturbation-agnostic at inference, requiring no prior detection of perturbations and enabling zero-shot transfer to unseen distortions. We evaluate three LALMs across three benchmarks and show that AnchorPrompt improves answer consistency in most tested conditions. Clean accuracy improves in six of nine model-benchmark pairs, with minimal impact on the remainder of 1.2% at most. Crucially, AnchorPrompt reduces hallucinations under severe audio corruption while keeping false refusals on clean audio rare. Finally, these consistency gains transfer to unseen perturbations, such as choice permutations and reverberation.

0.15

TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence

cs.SD eess.AS ·2026-10-15 ·Giovana Morais, Yujin Kim, Atilay Kucukoglu, Magdalena Fuentes
benchmark similarity 0.65
Abstract · currently under review

We introduce TRACE (Trustworthy Reasoning from Acoustic Cues and Evidence), a benchmark for diagnosing the reliability of large audio-language models (LALMs) by revealing when, and under what acoustic conditions these models behave untrustworthily. Existing benchmarks primarily measure whether a model produces the correct answer, but correctness alone cannot determine whether the answer is grounded in acoustic evidence or instead reflects language and contextual priors. TRACE evaluates model behavior along three complementary dimensions: (1) task performance, whether the answer is correct; (2) evidence alignment, whether the model is attending to the audio regions required to answer the question; and (3) robustness, whether answers and attributed evidence respond coherently to controlled changes such as event order, overlap, signal-to-noise ratio, or background content. TRACE targets temporal ordering, counting, and event existence, three tasks that remain challenging for current models. We synthesize contextually informed acoustic scenes that allow us to test whether models hallucinate events associated with the context but absent from the audio. This synthesis also allows us to obtain a scalable benchmark for which we know the answers by design. For each scene, we create controlled variants in a programmatic way to stress-test model robustness. We introduce attribution-based metrics that quantify model attention at scale on target events and non-target events. Evaluation across five state-of-the-art models shows that current LALMs often detect where relevant sound evidence occurs, but they do not reliably interpret it, integrate it into the answer, or respond to controlled changes in that evidence. Our benchmark also shows that models still struggle the most with ordering and counting questions.

0.20

MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

cs.LG ·2026-09-27 ·Zhi Wen Soi, Giulio Segalini, Jian-Jia Chen, Lydia Chen
benchmark similarity 0.64
Abstract

Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims. Existing audio hallucination benchmarks mainly measure response correctness, leaving it unclear whether an LALM hallucinates or simply fails to understand the audio. We challenge correctness-based evaluation by defining two hallucination categories: (i) context, where claims are not grounded in the audio; and (ii) knowledge, where claims about audio-related topics lack support from externally verifiable facts. We introduce MISHAP-Bench, a comprehensive benchmark with 12,000 challenging open-ended question-audio pairs and a rigorous evaluation pipeline covering both categories. To evaluate open-ended responses, we propose a groundedness judge that uses reference rubrics and judge prompts guided by human annotations. We extensively evaluate ten state-of-the-art LALMs and show that hallucination remains substantial. Even a frontier model such as Gemini 3.7 Flash reaches a hallucination rate of 36.5%. We further adapt and benchmark four mitigation methods from multiple domains for LALMs. Despite some improvements, effective hallucination mitigation remains an open challenge. Finally, we call on the community to evaluate hallucination and benchmark mitigation methods with MISHAP-Bench.

0.12

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

cs.AI ·2026-10-02 ·Hyunjae Ra, Aecheon Jung, Jungin Park, Sungeun Hong
similarity 0.64
Abstract

Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions require different perceptual evidence, including audio, video, or their interaction. Notably, we observe that joint audio-visual inference can weaken the prediction even when a model can recover the correct answer from a single informative modality. For example, when asked which instrument is heard, a model may correctly predict violin from the audio alone. Once a video showing a guitar is added, its confidence in violin may drop. In this paper, we introduce Relevant Evidence Decoding (RED), a training-free method that identifies question-relevant evidence and selectively strengthens its contribution. RED uses pointwise mutual information to quantify the predictive support provided by audio and video beyond the question alone. It decomposes their joint contribution into audio, video, and residual interaction components. A question-only inference pass determines the required evidence type, after which the model augments the original audio-visual prediction with the corresponding PMI contribution. Across three audio-visual hallucination benchmarks and three AV-LLMs, RED improves accuracy over standard decoding by up to 7% on CMM, 6.3% on AVHBench, and 3.8% on SVHalluc, with an average relative time to first token of 1.5x standard decoding.

0.36

The Machine's Internal Clock: Do LLMs Share Human Temporal Illusions?

cs.CL ·2026-08-15 ·Catherine Bao, Vivek Srikumar
benchmark similarity 0.64
Abstract · 27 pages, 19 figures

Human perception of time is subjective. Temporal illusions show that the brain relies on context and relational cues when judging duration. Prior studies established these effects with visual and auditory stimuli. We ask whether written narratives alone can cause these illusions, and whether LLMs respond as human participants do. We introduce a benchmark of 6,684 paired scenarios spanning five illusions, each judged from the character's and the reader's perspective. In our human study (60 participants), participants' judgments matched the predicted directions for all five illusions in at least one perspective (7 of 10 conditions). Across 14 LLMs, models reproduce every effect humans show in the same direction, but on average more strongly. Most effects persisted even after cue words were removed and after controlling for 15 lexical and semantic features. These results show that models and humans agree in direction but not in strength, which cautions against treating LLM judgments as substitutes for human responses for subjective temporal reasoning.

0.10

Probing and Mitigating Hallucinations in Speech-augmented Language Models for Automatic Speech Recognition via Small Language Models

eess.AS ·2026-10-18 ·Bi-Cheng Yan, Jhih-Rong Guo, Fu-An Chao, Berlin Chen
published similarity 0.63
Abstract · 5 pages, 5 figures, accepted by INTERSPEECH 2026

The widespread success of large language models in natural language processing has spurred the development of speech-augmented language models (SLMs) to extend their reasoning capabilities from textual inputs to spoken ones. Despite impressive performance, SLMs remain susceptible to hallucinations, posing challenges in practical utility. To this end, we first probe hallucinations in SLMs for automatic speech recognition (ASR) via the lens of causal mediation and behavioral analysis. To mitigate fabricated ASR hypotheses, we propose AudioSLM, a compact yet effective ASR framework built upon a small language model, which leverages alignment cues and cross-attention layers to synergistically bolster acoustic modeling and cross-modal interactions. Experiments on LibriSpeech reveal that hallucinations arise from multi-head self-attention modules with an excessive attention bias towards textual tokens. AudioSLM considerably eases ASR hallucinations and outperforms some LLM-based ASR models.

0.32

Do Audio LLMs HEAR Pain? Tracing Acoustic Evidence from Speech to Language Reasoning

cs.CL cs.AIcs.SD ·2026-10-28 ·Wenqi Li, Xiangjue Dong, Sulaiman Vesal
similarity 0.63
Abstract · 26 pages, 9 figures

Large audio language models (LALMs) can process speech directly, but it remains unclear how well they use the non-lexical acoustic information that speech carries. We study this question through pain, using TAME-Pain, which contains pain-neutral Harvard sentences and spoken pain ratings recorded at different self-reported pain levels. We introduce HEAR, a four-stage diagnostic framework that traces pain-related information from the speech signal to the model's prediction, and apply it to seven closed-source and open-weight LALMs. On the Harvard sentences, where pain can only be inferred from the voice, all models perform at chance level, and same-speaker references or few-shot examples improve balanced accuracy by no more than six points. On the spoken ratings, the same models reach 87.5-99.7% balanced accuracy. Layer-wise probing of open-weight LALMs shows that pain can be decoded from early audio representations (74.1% balanced accuracy for binary prediction and 45.0% for three-class severity), but that this information, along with most pain-associated acoustic cues, decreases in deeper layers. In an acoustic perception test, weaker models cannot distinguish even large changes in these cues, whereas stronger models can but still fail to infer pain. Finally, we recover pain-related acoustic cues and directly reinject them as structured text into the prompt, which improves pairwise pain prediction by up to 32 points. These results indicate that current LALMs encode paralinguistic information in early layers that is largely unavailable when the model produces its answer.

0.18