CS Paper Radar

ASR-Protected Fusion and Reference-Anchored Policy Optimization for Joint ASR and Word-Level Speech Anti-Spoofing

Xueping Zhang, Xiaoxiao Miao, Tiantian Feng, Ming Li

cs.SD ·submitted 2026-10-25 08:40 UTC · v1
code/data

Abstract

Existing anti-spoofing systems report either an utterance-level score or a time-based spoof interval: the former cannot show which words are spoofed, while the latter does not directly match the transcript and must be post-aligned with words, so alignment errors can lead to wrong word-level labels. Standard ASR model outputs only text; spoof information in the audio is lost after recognition and is not passed with the transcript. Yet changing only a few words can change the meaning of a sentence, leaving downstream systems to use incorrect text without knowing which words are spoofed. We therefore treat partial anti-spoofing as tagged ASR: the model transcribes speech and marks spoofed words at the same time, directly linking spoof labels to output words. We adopt the audio LLM and use ASR-Protected Subspace Gated Fusion (APSGF) to fuse spoof cues with ASR features. Spoof cues are added only outside an ASR-protected subspace, so they do not disturb recognition features. After cross-entropy training, we fine-tune the model with Reference-Anchored Policy Optimization (RAPO). Since the SFT policy is already concentrated, multiple samples from the same input tend to receive similar rewards, resulting in weak group-wise advantages. RAPO instead generates a single sample with low sampling randomness and anchors its advantage to the reward of a reference model. This allows the model to reinforce responses that outperform the reference while suppressing those that underperform it. Experiments on five datasets demonstrate the effectiveness of our approach in jointly detecting spoofed words and preserving ASR accuracy. The code is publicly available.

Closest papers

Nearest neighbours by meaning across the whole index.

Collapse, Not Invariance: Diagnosing Auxiliary Objectives in Speech Anti-Spoofing

cs.SD ·2026-09-30 ·Ksenia Lysikova, Kirill Borodin, Maxim Maslov, Grach Mkrtchian
similarity 0.72
Abstract · Submitted to IEEE ICASSP 2027

Speech anti-spoofing countermeasures degrade when the generator, codec or channel changes, and a common remedy is an auxiliary objective that shapes the embedding space; whether it does is invisible to EER, a pure ranking metric. We compare seven such objectives with cross-entropy over 113 runs on five corpora, AASIST3 at three seeds plus four pre-trained detectors, and measure the embedding space of the 24 AASIST3 runs directly. Raw augmentation displacement makes cosine consistency look effective, but the gain is a smaller space, not a more stable one: normalised by the spread, no configuration consistently improves on cross-entropy. Every trained space is dominated by the single decision axis expected for two classes, whose training-set structure does not transfer, and four runs collapse to a near-constant output that displacement rewards and EER reports as poor accuracy. No auxiliary objective keeps an advantage over cross-entropy across architectures, corpora and seeds.

0.14

Spooftral: Can Voxtral Audio-Language Model Detect Speech Spoofing?

eess.AS cs.CL ·2026-09-23 ·Avishai Weizman, Yehuda Ben-Shimol, Itshak Lapidot
published similarity 0.72
Abstract · 8 pages, 3 figures, 5 tables. Accepted to the Spoken Language Technology (SLT) 2026

Self-supervised learning (SSL) countermeasures (CMs) have shown strong performance in recent years. However, they often show degraded performance while facing unseen spoofing attacks and mismatched conditions. This study examines the Voxtral audio-language model (ALM) framework for spoofing detection, as a step toward combining CM capabilities within the ALM framework. We analyze how Voxtral captures spoofing cues through audio-text processing and propose an instruction-guided approach that uses label-sequence likelihoods to evaluate bonafide and spoofed speech. Experiments on the ASVspoof databases show that without task-specific adaptation, the LLM layers emphasize semantic representations, reducing the separability of spoof-discriminative acoustic cues compared to the Whisper-based audio encoder. Consequently, spoofing-related information becomes less separable after language-model processing. We also applied lightweight adaptation using weight-decomposed low-rank adaptation (DoRA) to the Voxtral model and propose the Spooftral model, achieving an equal error rate (EER) of 4.25% on the ASVspoof5 evaluation set.

0.38

Temporal Anchors and Editing Sensitivity in Partial Speech Spoofing: A Controlled Study

cs.SD ·2026-10-03 ·Xiaosu Su, Yun Cao, Yiping Ni, Xiaowei Yi
code/data similarity 0.69
Abstract

Partial-spoof detectors must reject synthetic content while accepting benign edits. We diagnose temporal anchors and boundary-consistency training using frozen WavLM features, nonlinguistic unit controls, and source-locked evaluation. Genuine-genuine and genuine-fake splices contrast editing false alarms with synthetic-content misses; they do not isolate a unique causal artifact. With layer-6 features, phone units have higher PartialSpoof evaluation frame EER than soft frames (11.18% versus 9.59%). In 500 retrospective PS-eval cases, augmentation reduces genuine-splice false alarms; synthetic-core misses increase at 5% PS-development FPR but not demonstrably at 1%. A 200-case Llama A follow-up retains the false-alarm reduction but does not establish a miss-rate increase. Constructed-case ranking can improve while fixed-threshold misses rise. Consistency gives no uniform gain, and encoder-layer rankings change across corpora. We report frame and event localization with three seeds, including word units, and treat PS evaluation as retrospective. Audit code, selected training scripts, and result summaries are available at https://github.com/mysxs/partial-spoof-diagnostics.

0.24

CRAF: Cross-View Residual-Aware Fusion for Deepfake Speech Detection

cs.SD cs.AIeess.AS ·2026-09-12 ·Minh-Xuan Phan, Khalid Zaman, Candy Olivia Mawalim, Masashi Unoki
similarity 0.68
Abstract

Recent advances in speech synthesis and voice conversion have made deepfake speech increasingly realistic, making generalization to unseen spoofing attacks a critical challenge. Pretrained speech and audio models offer a promising direction for improving robustness to such unseen attacks. Self-supervised learning (SSL) models capture fine-grained, low-level acoustic characteristics, whereas Auditory Large Language Models (ALLMs) provide higher-level contextual representations. These complementary views can provide useful cues for improving generalization to unseen attacks. However, direct fusion does not explicitly disentangle information shared across the two views from view-specific complementary information, limiting effective cross-view integration. To address this, we propose CRAF, a cross-view residual-aware fusion framework that uses ALLM-guided cross-view attention to enrich SSL representations and adopts ALLM as a high-level reference to separate ALLM-explainable information from complementary SSL residual information. The residual is selectively refined through adaptive gating and integrated through SSL-primary fusion. Experiments on ASVspoof 5 show that CRAF with Kimi-Audio achieves an EER of 5.96% and a minDCF of 0.1192, demonstrating robustness to unseen spoofing attacks.

0.26

GhostWord: A Fine-Grained Backdoor Attack on Automatic Speech Recognition

eess.AS cs.SD ·2026-09-02 ·Mojtaba Nafez, Mobina Poulaei, Kiarash Kiani Feriz, Aref Mousavi, Mohammad Ebrahim Mahdavi +2 more
publishedcode/data similarity 0.67
Abstract

Automatic Speech Recognition (ASR) systems are widely deployed in safety-critical settings but remain vulnerable to data-poisoning backdoor attacks. Existing ASR backdoors typically use phrase-level triggers paired with a fixed target sentence, creating strong artifacts (e.g., repeated transcripts or triggers placed in non-speech regions) that simple preprocessing can mitigate. We propose GhostWord, a word-level, time-localized ASR backdoor that uses codebooks mapping short ($\approx$400\,ms) acoustic triggers to target words. During poisoning, we inject a trigger into the forced-aligned time span of a chosen source word in the audio and replace only that word in the transcript, enabling precise semantic flips and composable sentence manipulation while avoiding many-to-one label artifacts. Across Common Voice (v23 English, v24 Lithuanian) and multiple backbones (Whisper-Small/Medium, MMS, SpeechT5), GhostWord achieves an average attack success rate of 89.3\% and transfers across languages and models. Adapting optimization-based defenses (ABL, ANP, SAU, I-BAU) reveals a sharp robustness--accuracy trade-off: attack success drops from 89.3\% to 29.1\% while clean WER rises from 21.5\% to 45.0\%, consistent with our theoretical analysis showing that, in high-vocabulary models, backdoor suppression structurally tends to degrade clean performance. The source code is publicly available at https://github.com/rohban-lab/GhostWord

0.31

Towards Quantifying Benchmark Optimization in ASR Models

cs.SD cs.AI ·2026-08-20 ·Theo Lebryk, David Ayllon, Alice Baird, Jakub Piotr Cłapa, Jens Madsen +1 more
benchmarkcode/data similarity 0.67
Abstract

Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for these benchmarks in ways that do not generalize well to real-world data. We present a methodology for quantifying benchmark optimization, focusing on cases where the audio underdetermines the reference transcript. We identify three families of behavioral probes that reveal models' capabilities of reproducing benchmark reference spans despite underdetermined audio: reference disagreement, masked-number recovery, and orthographic switching. We find that the highest-scoring open source models output verbatim reference transcript spans even when the relevant audio is contradictory, masked, or ambiguous. Using a variety of mechanistic probes, we show that models respond to narrow acoustic cues to override the faithful representation of the audio in favor of a benchmark-optimized policy. We show the benchmark-optimized behavior can be causally manipulated via low-rank linear steering or simply appending audio to the end of a segment in some cases. Overall, our results indicate that high-performing models exhibit benchmark-conditioned behaviors that can inflate benchmark performance without reflecting improved general-purpose transcription ability.

0.42

Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

cs.CL cs.AI ·2026-08-31 ·Chan-Jan Hsu, Jaeyeon Kim, Chao-Han Huck Yang, Shinji Watanabe, Hung-yi Lee +1 more
top venue similarity 0.66
Abstract · 24 pages, 8 figures, 2 tables. Accepted to Findings of EMNLP 2026

Recent automatic speech recognition (ASR) systems increasingly integrate large language models (LLMs) to leverage their semantic knowledge, either externally through logit fusion or internally through warm initialization. However, how to effectively combine these two strategies remains underexplored. In this work, we refine warm-initialized LLM-based ASR models by leveraging their own pre-adaptation base LLMs, focusing on LoRA-adapted settings where the base LLM is preserved. To achieve this, we propose Hybrid Search, a targeted correction strategy motivated by two observations. First, interaction features that characterize the relationship between LLM-based ASR hidden states and base-LLM hidden states provide informative signals about a token's degree of semantic dependence. Second, selectively refining targeted tokens with high semantic dependence improves ASR performance far beyond naive global LLM-correction methods including rescoring and late fusion. Our analysis suggests that, even after semantic knowledge transfer through warm initialization, LLM-based ASR models can still leverage their base LLM to further improve inference-time performance.

0.41

From Reliable Text to Real Voices: Trust-Aware Progressive Adaptation for Low-Resource TTS

cs.SD ·2026-09-22 ·Jiayi Lu, Yizhong Geng, Jinghan Yang, Tianhan Jiang, Boxun An +2 more
hot direction similarity 0.66
Abstract · 5 pages, 2 figures, 3 tables. Jiayi Lu and Yizhong Geng contributed equally. Corresponding author: Ya Li

Low-resource text-to-speech (TTS) adaptation is constrained by scarce paired data and costly manual transcription. Existing fixed-voice TTS systems can provide relatively accurate pronunciation, but their synthetic speech offers limited speaker diversity and may exhibit flat prosody. Real recordings provide natural prosody and diverse voices, yet their automatic speech recognition (ASR) pseudo-labels may contain transcription errors. We find that supervision order affects content accuracy and speaker similarity. We propose trust-aware progressive adaptation: synthetic-to-real adaptation first establishes text-speech correspondences, then restores reference-speaker control using real speech. Transcript-agreement weighting uses agreement between two fixed ASR systems as a proxy for pseudo-label reliability to limit noisy supervision. Experiments with FireRedTTS3 on Burmese and Lao and OmniVoice on Burmese show improved content accuracy with high naturalness and competitive speaker similarity. Jointly considering supervision order and pseudo-label reliability when combining synthetic and real speech offers a practical path to zero-shot voice cloning in low-resource languages with less manual transcription. Audio demos are available at https://insiderx-pro.github.io/S2R-Adaptation-TTS/

0.35