CS Paper Radar

BranchShine-CR: Compact Multilingual IPA Transcription with Self-Conditioned CTC and Consistency Regularization

Nikhil Navas, Sergio Chevtchenko, Talisson Damiao, Saeed Afshar

cs.LG ·submitted 2026-09-24 05:58 UTC · v1

Abstract

We introduce BranchShine-CR, a 25M-parameter model for multilingual transcription into the International Phonetic Alphabet (IPA). It combines log-mel features, a rotary-position E-Branchformer encoder, intermediate self-conditioned connectionist temporal classification (CTC), and consistency regularization across augmented views. On 16,646 shared IPApack++ test utterances, it achieves 4.47% IPA character error rate, a 22.3% relative reduction from ZIPA-CTC-NS, with approximately one-twelfth as many parameters while being trained from scratch. BranchShine-CR also outperforms a similarly sized NeMo Conformer baseline across all 41 dataset language labels. Ablation studies indicate the individual components synergetically acting in model performance contribution. These findings support compact IPA recognition capabilities under limited compute budget, for applications in low-resource on-device pronunciation assessment.

Comment
5 pages, 3 figures, 3 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Deep Learning Techniques for Phoneme Recognition in Italian Children' s Speech

cs.LG ·2026-09-27 ·Nicola Barbaro, Cristina Gena, Francesco Petriglia, Andrea Meirone, Alessandro Mazzei +1 more
hot direction similarity 0.64
Abstract

Speech therapists often face difficulties diagnosing impairments due to the lack of efficient tools for transcribing speech into the International Phonetic Alphabet (IPA). This work addresses this challenge with Broca, a Conformer-based deep learning system pretrained on 8 days of adult speech and fine-tuned on a 165-minute dataset of Italian child speech collected through a range of standardized diagnostic tests for children aged 3.5-6.5. Broca was optimized to handle phonetic variability in children's speech, including tone, accent, and speech errors, and achieved a state-of-the-art weighted Phoneme Error Rate of 13.36% on Italian speech. Remarkably, this performance was obtained using less than three hours of child-specific data, underscoring the model's efficiency and robustness in low-resource clinical settings. This work demonstrates that accurate, vocabulary-independent speech-to-IPA transcription can be achieved with minimal data, paving the way for more accessible, data-efficient tools to support speech assessment and diagnosis.

0.33

Nepali-English Code-Mixed Speech Recognition: Data Preparation and Adaptation of a Speech Foundation Model

cs.AI ·2026-11-10 ·Jeevan Neupane, Janardan Bhetwal, Kshitiz Paudel, Anup Aryal, Bipin Bashyal
code/data similarity 0.64
Abstract · 9 pages, 5 figures, 13 tables. Code and model checkpoints: https://github.com/AI-Lab-Pvt-Ltd/nepali-codemixed-asr

Nepali speakers regularly mix English words into conversation, but multilingual speech recognisers often struggle with language switches and the writing script used for English words. We study how to adapt SraVaani, a 449M-parameter hybrid TDT-CTC model, using a 172.8-hour corpus covering six Nepali speech domains. Its pretrained tokenizer does not support digits and several common symbols. Rather than replace it or discard these recordings, we generate Nepali and English readings for numbers and use zero-shot ASR output to select a plausible form. Full fine-tuning reduces test word error rate (WER) from 48.6% to 17.1%; a 12-epoch run reaches 16.3%, compared with 16.4% for fine-tuned Whisper large-v3-turbo. Freezing the encoder increases WER to 25.5%, while excluding numeric clips raises WER on numeric test speech from 14.3% to 21.3%. We also evaluate on FLEURS using both its original references and mixed-script alternatives. The comparison shows that transcription conventions explain part, but not all, of the out-of-domain performance loss.

0.25

Multilingual TASU: Extending Text-Only Alignment to Multilingual Speech Recognition via Simulated CTC Posteriors

eess.AS ·2026-10-31 ·Chenghao Wang, Yi Yang, Jing Peng, Yu Xi, Shuai Wang +1 more
publishedcode/data similarity 0.64
Abstract · Accepted at NCMMSC 2026, to appear in Springer CCIS

Speech Large Language Models (Speech LLMs) achieve strong multilingual ASR when large-scale paired speech-text data is available, yet such resources remain scarce and highly imbalanced across languages. Text-Only Alignment for Speech Understanding (TASU) alleviates this bottleneck by training cross-modal alignment from simulated CTC posteriors without paired audio, but prior work has been limited to English, leaving open whether text-driven alignment can scale to the multilingual setting where data sparsity and cross-lingual phonetic interference amplify the difficulty of learning a shared alignment interface. In this work, we investigate TASU-style posterior alignment as a means of mitigating paired-data scarcity in multilingual ASR. We adopt the native 50,002-way posterior space of a frozen OWSM-CTC frontend as a shared multilingual interface to Qwen2.5-1.5B-Instruct, and systematically compare three supervision sources on eight Multilingual LibriSpeech languages: real audio-derived posteriors, heuristic CTC Posterior Simulation (CPS), and posteriors generated by a learned multilingual simulator. With real posteriors, LoRA adaptation reduces pooled WER from 10.08% (standalone OWSM-CTC) to 8.21%, improving all eight languages. For text-only alignment, the learned simulator substantially outperforms heuristic CPS and attains 10.51% WER with LoRA. Notably, a CPS-initialized model reaches 9.19% WER after only one real-posterior adaptation epoch using merely one-ninth of the complete MLS paired-speech training set, demonstrating that text-first initialization can drastically reduce the amount of paired audio required for competitive multilingual ASR. These findings establish TASU as a viable multilingual alignment framework and highlight text-only simulation as an effective remedy for paired-data scarcity and imbalance across languages.

0.28

FuseAlign: Forced Alignment in the Wild

cs.LG ·2026-09-27 ·Mithilesh Vaidya, Stephen Bailey, Sumukh Badam, Matthew Bendel, Xingzhe He
benchmark similarity 0.64
Abstract · Preprint. Under review

Word-level forced alignment estimates when each transcript word occurs in an audio recording. It underpins text-based media editing, subtitling, speech-data curation, and phonetic analysis. Existing evaluations understate the difficulty of forced alignment by relying on short, clean speech, perfect transcripts, and metrics that obscure consequential alignment errors. In contrast, real-world media and data-processing pipelines operate on long and diverse recordings. Additionally, forced aligners often operate on error-prone automatic speech recognition (ASR) output. We address these gaps with improved evaluation metrics, a scoring protocol for real ASR transcripts, and AlignBench, a benchmark spanning diverse speaker, acoustic, and text conditions. We further introduce FuseAlign, a transformer-based aligner trained on large-scale pseudo-labeled speech with online label correction. FuseAlign performs joint contextualization of audio and text for the localization of coarse words. The model then refines boundaries at millisecond resolution and detects missing transcript words in the audio without lexicon-based or Viterbi decoding. On AlignBench, FuseAlign substantially outperforms all baselines and remains robust under real ASR transcripts. Ablations show that convolutional upsampling and EMA-snapshot label correction matter more than model properties such as parameter count.

0.18

A Training Criterion with Token-Level Tolerance to Transcription Ambiguity for Automatic Speech Recognition

cs.CL eess.AS ·2026-09-24 ·Saurabh Kumar, Diptiman Mohanta, Prasanta Kumar Ghosh
similarity 0.63
Abstract · 5 pages, 2 figures, 4 tables; submitted to ICASSP 2027

Automatic speech recognition is typically trained assuming that the reference transcript is the only valid labeling of an utterance, yet even nominally verbatim transcripts contain localized differences in pronunciation, spelling, or lexical realization that the acoustics do not uniquely determine. Omni-temporal Classification (OTC) tolerates such noise by adding wildcard paths to the connectionist temporal classification (CTC) alignment graph, but its word-level arcs are too coarse, since bypassing one unsupported token discards supervision for the whole word. We move wildcard arcs to token granularity so unsupported tokens can be bypassed while the rest of the word stays supervised, and we combine token- and word-level arcs as complementary escape paths. Across 19 languages and three corpora, token-level OTC improves over CTC on all 25 tasks. We also replace epoch-indexed relaxation of the wildcard weights with a predictive-entropy-indexed schedule, which performs comparably while reducing dependence on training length. Combining this schedule with the hybrid graph gives the lowest mean word error rate (WER) on every corpus and a 9.45% average relative WER reduction over CTC. Independent validator transcriptions show that token-level models place significantly more wildcard-bypass probability than CTC on disputed characters, indicating that token-level tolerance targets localized transcript ambiguity.

0.29

Subphonetic Acoustic Modeling via Optimal Transport for Pronunciation Assessment

eess.AS cs.SD ·2026-09-12 ·Haopeng Geng, Jiun-Ting Li, Daisuke Saito, Nobuaki Minematsu
publishedhot direction similarity 0.63
Abstract · Accepted to SLT 2026

Pronunciation assessment requires acoustic evidence that is temporally precise, diagnostically meaningful, and faithful to the learner's actual production. However, existing acoustic models often struggle to provide recognition and segmentation evidence simultaneously. CTC-based phone recognizers can predict phone sequences flexibly, but their sparse and peaky posteriors often miss phone boundaries and fine-grained pronunciation cues. In contrast, text-dependent forced aligners provide reliable temporal information when transcripts are available, but are not directly applicable to reference-free pronunciation analysis. In this work, we propose a topology-aware frame-wise acoustic model that learns dense ordered state posteriors within each phone. The key idea is to recover phone-internal state structure in a neural acoustic model by combining ordered subphonetic states with optimal temporal transport classification (OTTC). This combination encourages dense monotonic frame-level state discrimination while preserving phone recognition ability. Experiments on read, spontaneous, and L2 speech show improved segmentation over neural baselines with competitive recognition performance. Downstream evaluations further show gains in mispronunciation detection and automatic pronunciation assessment. Probing analysis suggests that the learned states capture phoneme-dependent acoustic structure rather than arbitrary frame-level distributions.

0.38

On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin

eess.AS ·2026-08-12 ·Shuiyuan Wang, Bingshen Mu, Pengshen Zhang, Chengyou Wang, Yujie Liao +4 more
similarity 0.63
Abstract

Recent large-scale ASR models already achieve strong Mandarin recognition accuracy and have some ability to recognize Chinese dialects. However, their dialect recognition accuracy is still limited in real-world speech. Direct dialect adaptation can lower dialect CER, but it may also raise Mandarin CER. We therefore study how to adapt a capable ASR model to improve multi-dialect recognition without degrading Mandarin recognition. We adopt an adaptation pipeline where continual pre-training (CPT) and dialect supervised fine-tuning (SFT) provide a strong foundation, and On-Policy Self-Distillation (OPSD) serves as the final refinement. OPSD addresses the train--test mismatch in autoregressive ASR by training the student model on its own decoded prefixes while a frozen teacher, conditioned on the reference transcript as privileged context, provides soft token-level targets. This replaces hard cross-entropy updates on dialect data with distillation, preserving Mandarin ability while refining dialect recognition. We instantiate the framework with Qwen3-ASR-1.7B and evaluate it on public and internal Mandarin and dialect test sets. Under matched refinement data and schedule, OPSD improves dialect recognition without raising Mandarin CER, whereas continued teacher-forced fine-tuning increases Mandarin CER. We will release the model weights and evaluation scripts.

0.39

CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition

cs.CL cs.AI ·2026-10-13 ·Tianchi Liu, Nicholas Moratelli, Khurram Azeem Hashmi, Mohammadreza Zolfaghari, Rishabh Jain +3 more
benchmark similarity 0.63
Abstract

Feature distillation is usually expected to add to logit distillation. We find that this expectation does not hold for two common feature losses in compact multilingual automatic speech recognition (ASR), and analyze why through their gradients. The gradient of a projection-based loss is orthogonal to the task gradient, while that of Centered Kernel Alignment (CKA), a similarity-based loss, is weakly aligned with it. CKA lowers the word error rate (WER) on non-English languages, but at the expense of data-rich English. Three loss-level remedies (one trained, two probed) all fall short, and the CKA gradient overlaps with the subspace English uses in the shared encoder by 97.5% to 99.5% (about 81% by chance). Guided by this diagnosis, we propose CAFE (Confined-Adapter Feature distillation), which confines the CKA gradient to adapters that English bypasses, addressing the trade-off in the parameters rather than in the loss. On the Open ASR Leaderboard, CAFE lowers the average WER from 8.55% to 8.02%, beats an adapter-only capacity control at 8.51%, and makes English more stable across checkpoints. At 254M parameters, it trails the 0.6B teacher by 2 WER points and is competitive with much larger models across 48 test sets in 24 languages. We use 564k hours of training data and state-of-the-art models to keep the findings relevant to practice.

0.20