CS Paper Radar

Turning Speech Language Models into Multilingual Listeners

Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning, Ahmet Üstün, Martijn Bartelds

cs.CL ·submitted 2026-09-27 04:43 UTC · v1
top venuebenchmark

Abstract

Speech Language Models (SLMs) that understand spoken language questions support only a few high-resource languages, limiting access to millions of people worldwide. This gap stems from the scarcity of multilingual speech instruction-tuning datasets. We present MULTISPEECHQA, a large-scale, synthetically generated and human-verified dataset comprising 9200 hours of 10.8 million spoken question-answer pairs in 23 typologically diverse languages. Using MULTISPEECHQA, we also introduce MULTISPEECH-BENCH, a multi-task benchmark for evaluating SLM performance on 23 languages. We compare the performance of a cascading system to open-weight and closed SLMs on MULTISPEECH-BENCH and find that the cascading system outperforms open-weight SLMs but not all closed SLMs. We use MULTISPEECHQA to finetune Qwen 2.5-Omni, which improves its performance on our benchmark. Our findings show that high-quality synthetic datasets offer a cheap solution to improving the multilingual capabilities of SLMs.

Comment
Interspeech 2026 Long

Closest papers

Nearest neighbours by meaning across the whole index.

Nuha-Speech: Building General-Purpose Arabic Speech-LLMs

cs.CL ·2026-09-10 ·Yingzhi Wang, Reem Alhazzani, Muhammad Alqurishi
similarity 0.75
Abstract

As Speech Large Language Models (speech-LLMs) become increasingly multilingual, Arabic remains significantly underrepresented, highlighting the need for dedicated infrastructure to train and evaluate Arabic speech-LLMs. To address this gap, we introduce Nuha-Speech, a comprehensive initiative to develop general-purpose Arabic speech-LLMs spanning dataset construction, model training, and systematic evaluation. Specifically, we constructed a large-scale Arabic Speech Question-Answering (SQA) corpus comprising over 1.5 million training samples to allow instruction tuning over a broad range of core speech tasks. Then, the corpus was used for supervised fine-tuning based on Qwen-Omni model variants at different scales. Finally, we designed an evaluation framework featuring diverse tasks and tailored metrics. Through this work, we aim to establish foundational infrastructures for Arabic Speech-LLMs under constraints imposed by limited Arabic speech resources.

0.30

Monsoon: Scalable Spontaneous Speech Collection for Low-Resource ASR in 50 Languages

cs.CL ·2026-10-19 ·Shobhit Banga, Manas Dhir, Aaditya Pareek, Amritansh Walecha, Bhaskar Singh +6 more
similarity 0.73
Abstract · Submitted to ICASSP 2027

For most languages, automatic speech recognition (ASR) is limited less by model architecture than by a lack of accurately transcribed spontaneous speech. We present Monsoon, a framework for collecting such data at scale. It elicits unscripted conversations between paired contributors, screens recordings with automated classifiers, corrects transcripts over five levels of human review, and selects segments for lexical and acoustic coverage. It has collected speech in 50 languages; we analyse seven 3,500-hour corpora. At equal volume, its quality stages reduce semantic word error rate (WER) by 10-39% relative to a conventional protocol. Compared with an architecture-matched baseline trained on public corpora, Whisper fine-tuned on Monsoon performs comparably on the baseline's own test sets and reduces semantic WER by 20-44 points on conversational benchmarks. The gains transfer to a second architecture and grow with data volume. We release the framework, its quality-control models, and the fine-tuned models.

0.11

SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

cs.CL ·2026-09-09 ·Jingyi Liao, Wenyu Zhang, Zhuohan Liu, Yingxu He, Geyu Lin +4 more
top venuebenchmarkcode/data similarity 0.73
Abstract · Accepted to EMNLP 2026

The rapid advancement of audio and multimodal large language models has unlocked transformative speech understanding capabilities, yet evaluation frameworks remain predominantly English-centric, leaving Southeast Asian (SEA) languages critically underrepresented. We introduce SEA-SpeechBench, to the best of our knowledge, the first large-scale multitask benchmark that evaluates speech understanding in 11 SEA languages through 97,194 samples across 99 evaluation sets and 597 hours of curated audio data. Our benchmark comprises 9 diverse tasks across 3 categories: speech processing (automatic speech recognition, speech translation, spoken question answering), paralinguistic analysis (emotion, gender, age, speaker recognition), and temporal understanding, a novel dimension featuring timestamped content queries and temporal localization within extended audio sequences up to 3 minutes. We implement multilingual prompting in both native SEA languages and English to reflect user interactions with audio-language models. Evaluation of leading open-source and proprietary systems reveals marked performance gaps. Across all models, performance remains underwhelming on temporal understanding, emotion recognition, and speech translation. Prompting in low-resource languages such as Burmese and Tamil lags behind English by up to 41 percentage points. Our findings expose critical model limitations and underscore the need for inclusive model development. The SEA-SpeechBench benchmark is available at https://zwenyu.github.io/SEA-SpeechBench/.

0.76

Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text

cs.CL cs.AI ·2026-08-24 ·Hyeonyu Kim, Hwayeon Kim, Youngwon Choi, Myeongkyun Cho, Huu-Kim Nguyen
top venuecode/data similarity 0.73
Abstract · Accepted to EMNLP 2026 Findings

Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.

0.75

Multilingual TASU: Extending Text-Only Alignment to Multilingual Speech Recognition via Simulated CTC Posteriors

eess.AS ·2026-10-31 ·Chenghao Wang, Yi Yang, Jing Peng, Yu Xi, Shuai Wang +1 more
publishedcode/data similarity 0.72
Abstract · Accepted at NCMMSC 2026, to appear in Springer CCIS

Speech Large Language Models (Speech LLMs) achieve strong multilingual ASR when large-scale paired speech-text data is available, yet such resources remain scarce and highly imbalanced across languages. Text-Only Alignment for Speech Understanding (TASU) alleviates this bottleneck by training cross-modal alignment from simulated CTC posteriors without paired audio, but prior work has been limited to English, leaving open whether text-driven alignment can scale to the multilingual setting where data sparsity and cross-lingual phonetic interference amplify the difficulty of learning a shared alignment interface. In this work, we investigate TASU-style posterior alignment as a means of mitigating paired-data scarcity in multilingual ASR. We adopt the native 50,002-way posterior space of a frozen OWSM-CTC frontend as a shared multilingual interface to Qwen2.5-1.5B-Instruct, and systematically compare three supervision sources on eight Multilingual LibriSpeech languages: real audio-derived posteriors, heuristic CTC Posterior Simulation (CPS), and posteriors generated by a learned multilingual simulator. With real posteriors, LoRA adaptation reduces pooled WER from 10.08% (standalone OWSM-CTC) to 8.21%, improving all eight languages. For text-only alignment, the learned simulator substantially outperforms heuristic CPS and attains 10.51% WER with LoRA. Notably, a CPS-initialized model reaches 9.19% WER after only one real-posterior adaptation epoch using merely one-ninth of the complete MLS paired-speech training set, demonstrating that text-first initialization can drastically reduce the amount of paired audio required for competitive multilingual ASR. These findings establish TASU as a viable multilingual alignment framework and highlight text-only simulation as an effective remedy for paired-data scarcity and imbalance across languages.

0.28

HINTT Submission to the 2nd MLC-SLM Challenge: Comparing Cascaded and Unified Approaches to Diarization and ASR

eess.AS cs.CL ·2026-10-06 ·Takanori Ashihara, Kohei Matsuura, Masato Mimura
similarity 0.72
Abstract

This paper presents the HINTT system submitted to the 2nd Challenge and Workshop on Multilingual Conversational Speech Language Model (MLC-SLM). We address multilingual speaker-attributed ASR, where systems must determine who spoke when and what was spoken. We investigate two modeling strategies for this problem: a cascaded pipeline that combines speaker diarization with speech-LLM-based ASR, and a unified speech LLM that directly generates speaker labels, timestamps, and transcriptions. Our final submission is based on the cascaded pipeline, consisting of a fine-tuned DiariZen diarization model, a fine-tuned Qwen3-ASR model, and LLM-based generative error correction. For comparison, we also fine-tune VibeVoice-ASR as a unified model using the same official training data. All task-specific fine-tuning and model selection are performed using only the official MLC-SLM data, without external data or pseudo-labels. Experimental results demonstrate that the cascaded system remains more reliable under the MLC-SLM Task 1 conditions, while unified speech LLMs offer a promising direction for future speaker-attributed ASR.

0.13

sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak

cs.CL cs.AI ·2026-10-06 ·Marek Šuppa, Ivan Vykopal, Andrej Ridzik, Kristián Sopkovič, Natália Kňažeková +7 more
top venuebenchmarkcode/data similarity 0.71
Abstract · Accepted to EMNLP 2026 Main

Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation. We present sk-bench, a native-first Slovak benchmark with 30 datasets (33 scored task variants) across ten skill categories. Eleven resources are introduced or first packaged for generative-LLM evaluation, including IFEval-SK with Slovak-adapted instruction checkers and native Chiby/SKJ1 resources for Slovak grammar and morphology. We evaluate 55 open- and closed-weights models under one harness. The best open model trails proprietary APIs by 12.6 points. Model rankings are similar for native and translated closed-form data ($ρ\geq0.98$), though translation separates the strongest models less well. By contrast, human-authored and LLM-generated QA questions rank models differently ($ρ=0.72$). For Qwen3-14B, continued Slovak pretraining lowers the overall score by 13.9 points. A small instruction set restores three quarters of that loss. Test-time reasoning improves scores by 8.5 to 12.5 points for models of 9B and above. Together, these findings suggest four design lessons for other under-resourced languages: use native data where translation fails, plan instruction repair after language adaptation, enable test-time reasoning before scaling up, and avoid overinvesting in target-language prompts. We release the data and code at https://github.com/slovak-nlp/sk-bench

0.41

Benchmarking Automatic Speech Recognition Tools for Iberian Languages

cs.CL ·2026-09-29 ·Fernando López, Pablo Gómez, David Solans, Paulo Villegas, Jordi Luque
publishedbenchmark similarity 0.71
Abstract · Accepted in IberSPEECH 2026

Comprehensive evaluations of automatic speech recognition (ASR) for Iberian languages remain limited, and low-resource languages, biases, and efficiency trade-offs are underexplored. We benchmark eleven systems, ten open-weight models and one commercial API, across five Iberian languages (Basque, Catalan, Galician, Portuguese, Spanish), with German and Turkish as controls. Evaluation uses an 85-hour dataset covering read speech, broadcast media, and audiobooks, assessing accuracy and efficiency via word error rate (WER) and real-time factors (RTF/RTFx). Results show no single model dominates: accuracy, efficiency, and language coverage present clear trade-offs. Low-resource languages, especially Basque, degrade significantly, highlighting the role of training coverage. We observe consistent sex disparities across most systems, highlighting fairness challenges in multilingual ASR. Overall, the benchmark provides practical guidance for real-world model selection.

0.43