CS Paper Radar

AcoustiClaim: A Numeric Claim Benchmark with Instrument Ground Truth

Sheng-Tse Lin, Siyuan Zhai, Chien-Liang Kuo, Massa Baali, Bhiksha Raj

cs.SD cs.CLcs.LGeess.AS ·submitted 2026-09-24 19:19 UTC · v1
benchmarkcode/datahot direction

Abstract

Audio language models state numbers for acoustic quantities, and neither human opinion nor a judge model says whether such a number is true of the signal. AcoustiClaim extracts each numeric claim from free text, scores it against the instrument that defines the quantity, and classes each quantity by where its reference can be read. Four open-weight systems and one closed model, asked for ten quantities five ways on two corpora, fill 207 cells. Of these, 49 emit fewer than five distinct values, and eight of the 158 cells that can be ranked exceed a rank correlation of 0.3, the bar we set, three with an interval clear of it, five of them one closed model reading pitch. Error sits at or above a constant-predictor floor in every ranked cell but three. The reference decoder we train declines the five voice quantities in prose on 95% of mixtures, with nothing withheld, and states them on the clean twins, reproducing its targets' rule from audio alone. With a calibrated threshold, withholding lowers error on all ten quantities on the mixtures in the mean and on eight at every split, against at most 0.6% from a random selector. A linear baseline orders errors at least as well as ours. F0 s.d. and shimmer stay above the constant floor.

Comment
5 pages, 3 figures, 2 tables. Submitted to ICASSP 2027. Siyuan Zhai and Chien-Liang Kuo contributed equally. Code and outputs: https://github.com/sheng-tse/acousticlaim

Closest papers

Nearest neighbours by meaning across the whole index.

Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

cs.SD eess.AS ·2026-10-21 ·Hardik B. Sailor, Qiongqiong Wang, Dheemant Rastogi, Nancy F. Chen, Ai Ti Aw
similarity 0.66
Abstract · Submitted to IEEE ICASSP 2027

Correct transcription does not establish that a word depended on its local audio. We introduce the Acoustic Evidence Contribution (AEC), a word-level counterfactual measure of acoustic grounding. It is obtained from the change in reference-token log likelihood when the word's aligned acoustic span is replaced. We combine decoding outcomes on unedited audio with thresholded AEC in an outcome-contribution matrix, where correct targets are categorized according to their local acoustic contribution as high or low. Depending on the system, 33-49% of correct single-word targets fall below the twofold contribution threshold when averaged across corpora, despite no evident rank association with WER. Retention of reference words after span removal decreases monotonically across AEC quintiles, from 76-83% in the lowest quintile to approximately 1% in the highest. AEC therefore separates transcription correctness from dependence on the aligned local acoustic span while remaining an oracle-aligned diagnostic method.

0.24

CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification

cs.CL cs.SDeess.AS ·2026-09-27 ·Massa Baali, Sarthak Bisht, Ziyue Qiu, Joseph Konan, Rita Singh +1 more
similarity 0.66
Abstract

Speaker verification systems achieve high accuracy but provide little account of the acoustic evidence behind their judgments. Making these systems inspectable requires exposing interpretable evidence while retaining the richer information on which their decisions depend. We present \textbf{CoLMbo-SV}, a speaker language model that combines strong speaker discrimination with structured, acoustically grounded comparison reports. By connecting a pretrained speaker encoder to a language model and supplying explicit acoustic measurements, CoLMbo-SV makes voice comparisons inspectable without restricting verification to the evidence verbalized in its reports. We additionally introduce \textbf{VoxReason}, paired recordings with measured acoustic properties and comparison reports filtered through numerical and qualitative checks, providing supervision for this combined capability. We also develop an evaluation framework that separates what acoustic information a speaker representation encodes, what influences the verification score, and what the generated report discusses. On VoxCeleb1-O, CoLMbo-SV achieves 0.99\% EER, reducing verification error by approximately 80\% relative to the strongest audio-language baseline fine-tuned on VoxReason, while attaining a numerical-grounding score of 0.82. Our analysis further demonstrates that acoustic correctness and decision relevance are distinct properties of an explanation, exposing a gap that numerical-grounding metrics miss. Together, these contributions substantially advance audio-language speaker verification, bring its accuracy toward that of dedicated speaker encoders while adding checkable acoustic reporting, and establish an empirical framework for connecting natural-language explanations to the decisions they explain.

0.28

Do Audio LLMs HEAR Pain? Tracing Acoustic Evidence from Speech to Language Reasoning

cs.CL cs.AIcs.SD ·2026-10-28 ·Wenqi Li, Xiangjue Dong, Sulaiman Vesal
similarity 0.65
Abstract · 26 pages, 9 figures

Large audio language models (LALMs) can process speech directly, but it remains unclear how well they use the non-lexical acoustic information that speech carries. We study this question through pain, using TAME-Pain, which contains pain-neutral Harvard sentences and spoken pain ratings recorded at different self-reported pain levels. We introduce HEAR, a four-stage diagnostic framework that traces pain-related information from the speech signal to the model's prediction, and apply it to seven closed-source and open-weight LALMs. On the Harvard sentences, where pain can only be inferred from the voice, all models perform at chance level, and same-speaker references or few-shot examples improve balanced accuracy by no more than six points. On the spoken ratings, the same models reach 87.5-99.7% balanced accuracy. Layer-wise probing of open-weight LALMs shows that pain can be decoded from early audio representations (74.1% balanced accuracy for binary prediction and 45.0% for three-class severity), but that this information, along with most pain-associated acoustic cues, decreases in deeper layers. In an acoustic perception test, weaker models cannot distinguish even large changes in these cues, whereas stronger models can but still fail to infer pain. Finally, we recover pain-related acoustic cues and directly reinject them as structured text into the prompt, which improves pairwise pain prediction by up to 32 points. These results indicate that current LALMs encode paralinguistic information in early layers that is largely unavailable when the model produces its answer.

0.18

Right Answers, Inconsistent Evidence: Source-Level Counterfactual Auditing of Large Audio-Language Models

cs.SD cs.AI ·2026-11-11 ·Yi Zhang, Yi Li, Hongwei Du
code/datahot direction similarity 0.65
Abstract · 5 pages, 3 figures, 1 table. Submitted to ICASSP 2027

A large audio-language model (LALM) can answer "is a cat present?" about a mixture correctly, and its confidence can drop when the cat is removed, yet this does not show that it can tell the cat's presence from its absence. We introduce TRACE. It deletes the queried source and each rival in turn and asks whether every view still containing that sound outscores the view with it removed. Across 8 LALMs and 3,456 controlled mixtures, 6.3% of correct answers fail this ordering: the target-absent view scores at or above some target-present view, a conflict no shared threshold repairs. A target-only score check misses 42.5% of them. The share rises from 4.9% to 8.4% as target SNR falls from +5 to -5 dB, even on units correct at every level, and survives loudness and re-pairing controls. Accuracy certifies the answer; TRACE audits whether it orders presence above absence. Code and data: https://github.com/yikise/TRACE.

0.41

VeriSpoof: Beyond Real or Fake with One Audio LLM

eess.AS cs.CRcs.SD ·2026-11-03 ·Yang Xiao, Ting Dang
similarity 0.64
Abstract · Tech Report

A speech deepfake detector estimates whether a recording is synthetic, but a forensic analyst also asks which system produced it, whether recordings share a source, and what evidence supports each conclusion. We present VeriSpoof, an audio language model that unifies five tasks with shared weights: deepfake detection, generator attribution (generator, acoustic model and vocoder), recording-condition analysis, synthesized-span localisation, and source verification. Each decision is supported by grounded rationales whose statements can be independently verified. Answers are scored by sentence likelihood, allowing detection and source verification to be thresholded without separate heads. Rationale statements can be checked against waveform measurements, span labels, or generator metadata. Counterfactual examples train decisions to follow the stated evidence. Across the 14 Speech-DF-Arena test sets, VeriSpoof-Plus achieves 2.14% pooled EER, ranking third of 28 systems and first among open-weight systems. VeriSpoof-Base, trained only on official training splits of eight corpora, achieves 4.43% EER, less than half that of the next open-weight detector. With the same weights, VeriSpoof-Plus verifies whether recordings from generators unseen in training share a source at 6.2% EER. Controlled studies show that the audio representation governs detection, the LLM governs recording comparison, and source supervision is necessary for neural-codec fakes. Statement-checkable rationales catch partial fakes missed by one-step answers (99.6% vs. 78.4% on PartialEdit) and determine decisions under edits unseen in training (99.5%). Training updates only 29.7M of 2.85B parameters, fitting a single 24 GB GPU with gradient checkpointing. We release both models, inference code, a demo, and VeriSpoof-Bench, the evaluation suite covering these questions.

0.33

One Waveform, Two Stories: Benchmarking Speech-Acoustic Conflict in Audio-Language Models

cs.SD cs.CLeess.AS ·2026-10-30 ·Wenbin Wang, Yuheng Li, Koichiro Yoshino
benchmark similarity 0.64
Abstract · 26 pages, 7 figures, 5 tables

Audio-language models (ALMs) are evaluated on their understanding of both speech and environmental sounds. What remains underexplored is how they respond when spoken claims contradict acoustic evidence within the same recording. We introduce SAC-Bench, a benchmark of intra-audio speech--acoustic conflict comprising 616,031 unique clips from 188 sound-event classes across seven semantic, temporal, and spatial tasks. It pairs environmental recordings with compatible or contradictory synthesized spoken claims and includes audio-only controls where applicable. Across five open-weight ALMs evaluated on semantic and temporal tasks, high audio-only accuracy often coexists with large losses under conflicting claims; Event Identity accuracy is 24.8--46.5 percentage points lower in conflict than compatible conditions. Temporal Order failures persist even when speech follows the complete event sequence, so simultaneous speech--event masking cannot fully explain them. A spatial case study with Spatial-Omni finds lower motion accuracy associated with both spoken claims and the speech source's physical trajectory. Prompt interventions sometimes restore acoustic accuracy, but can also reduce claim-aligned answers without improving grounding or worsen performance. SAC-Bench thus separates acoustic competence, conflict robustness, and intervention effects.

0.14

SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMs

eess.AS cs.CLcs.SD ·2026-08-27 ·Mengzhe Geng
benchmark similarity 0.63
Abstract

Speech language models (speech LLMs) can generate plausible outputs from audio that contains no usable speech evidence. We study this failure as a pre-generation support-estimation problem and present SURE-Voice, a training-free front end that decides whether an audio prompt contains intelligible speech evidence before calling a speech LLM. We build SURE-Challenge with a 640-example SURE-Core split and a 1,920-example SURE-Extended split derived from 120 LibriSpeech source utterances. Using one fixed operating point, an energy screen plus Whisper token confidence raises unsupported accuracy on the held-out Extended test from 0.000--0.133 to 0.919 for six non-degenerate speech LLM backbones, while supported accuracy remains 0.919--0.970 and downstream calls fall from 480 to 287. A 500-clip ESC-50 sanity set shows the same pattern on real environmental audio, with vocal non-speech as a residual failure mode. An overlap diagnostic shows that source attribution remains separate from speech-evidence filtering. The evidence supports a controlled benchmark baseline and a deployment-oriented analysis; it does not establish universal robustness to semantic answerability, gain variation or natural conversations.

0.17

Beyond the Transcript: Measuring and Improving the Use of Acoustic Information in Audio Language Models

cs.CL ·2026-11-08 ·Linkai Peng, Baorian Nuchged, Yuyang Yao
benchmark similarity 0.63
Abstract

Audio language model benchmarks provide comprehensive evaluation across diverse tasks. However, some tasks can be answered from the transcript or even the question alone, and benchmark scores can therefore confound acoustic understanding with language modeling ability. We distinguish lexical tasks, for which the transcript usually suffices, from global-attribute and within-recording-structure tasks, which require the model to analyse the recording beyond its words, either as a whole, for properties such as the speaker and emotion, or within it, for patterns such as stress and intonation. We evaluated 19 open models and Gemini 3.1 Pro on 92 tasks from 6 benchmarks and estimated the recording gain, the accuracy advantage of the original recording over references that retain its words, by comparing the recording with synthetic readings of its transcript and by adding it to the transcript. On attribute and structure tasks, the open models fall behind Gemini by 28.6 and 41.9 percentage points, and about half or more of each deficit lies in the recording gain. Nevertheless, linear probes in Gemma-4 12B and Qwen2.5-Omni-7B show that the acoustic information needed for answering is already readable in early language-model layers, while the readout of many cues declines in the layers that follow. We therefore constructed training data and trained LoRA adapters in these layers, with a content anchor, which raises attribute and structure accuracy by 12.8 to 44.4 points in the two models and keeps mean lexical accuracy within 1 point of the base model. Most of the improvement appears as larger recording gains. These results suggest that training improves the use of acoustic information that the models already encode.

0.10