CS Paper Radar

The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge

Jordi Luque, Lorenzo Concina, Marco Matassoni, Alessio Brutti, Filippo Vella

cs.CL ·submitted 2026-09-10 15:41 UTC · v1

Abstract

This paper details the Eloquence team's approach to Task 2 of the 2nd MLC-SLM challenge at Interspeech 2026, which involves multilingual Multiple-Choice Question Answering (MCQA) across 21 languages. Three approaches are explored. First, we fine-tune Voxtral-Mini-3B via LoRA with cross-lingual data augmentation, ASR transcript augmentation and timestamp-aware audio cropping, achieving 0.72 macro-accuracy on evaluation Phase 2. Second, we apply multimodal in-context learning (ICL) to the frozen Voxtral-24B model to correct a strong label bias, reaching 0.81, our best result. Third, a training-free retrieval system based on a three-layer voice-anchored memory combining acoustic identity, semantic content, and a knowledge graph achieves 0.68. All three systems substantially outperform the official baseline.

Closest papers

Nearest neighbours by meaning across the whole index.

Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

cs.CL cs.HC ·2026-09-18 ·Xize Cheng, Wenxu Jia, Chenyuhao Wen, Dongjie Fu, Zehan Wang +2 more
benchmarkcode/datahighly cited similarity 0.73
Abstract

Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient information. This issue is even more pronounced in the speech domain, where audio, as a low-compression modality, requires substantially more embeddings than text to preserve both semantic content and acoustic cues. To address this challenge, we introduce \textbf{Vox-Infinity}, the first benchmark specifically designed to evaluate long-context understanding in spoken language models. Vox-Infinity systematically extends audio history along two dimensions: turn count and turn duration. It covers a diverse range of representative scenarios with varying interaction structures and semantic complexity. Crucially, Vox-Infinity provides explicit answer-provenance annotations and organizes samples according to the amount of historical context required to resolve each query, enabling precise and length-aware evaluation. Extensive evaluations of seven representative spoken language models reveal a clear overall recency effect: models generally achieve higher accuracy when answer-supporting evidence is closer to the query, but struggle to retrieve and use evidence located farther back in the dialogue history. Cases and datasets are available at https://vox-infinity.github.io.

0.51

Leading-Silence Augmentation and Multi-Stage Synthetic Supervision for the Second MLC-SLM Challenge

cs.CL ·2026-08-14 ·Kexin Shi, Renhe Sun, Yuge Huang, Ximeng Wang, Jiayi Zhou +2 more
similarity 0.72
Abstract

The second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge evaluates two tasks over complete, unsegmented multilingual conversations: speaker diarization and recognition (Task 1) and conversational speech understanding (Task 2). Neither task provides oracle utterance boundaries or speaker labels at evaluation, and Task 2 provides no question-answer training set. For Task 1, we fine-tune VibeVoice-ASR-7B with random leading-silence cropping, consistent timestamp correction, and an exponential moving average (EMA) training strategy. For Task 2, we construct synthetic question-answer pairs through multimodal candidate generation, silent-audio filtering, and distribution-matched augmentation, and fine-tune Qwen3-Omni-30B-A3B-Instruct for tagged direct answering. On the Task 1 evaluation set, cropping reduces tcpMER from 18.30% to 17.27%, and EMA further reduces it to 16.73%. On the Task 2 evaluation set, jointly applying distribution-matched augmentation and tagged direct answering raises accuracy from 83.0% to 86.0%.

0.45

What Do Multilingual Self-Supervised Speech Encoders Know About Sentence Meaning?

cs.CL ·2026-11-13 ·Kevin Krahn, Eric Fosler-Lussier
published similarity 0.72
Abstract · Accepted to EACL 2027

Self-supervised speech models (S3Ms) have been scaled up to thousands of languages and millions of hours of audio, yet little is known about how these models structure semantic knowledge across languages. In this work, we probe sentence-level semantics in massively multilingual S3Ms across 76 languages, using linear maps from frozen speech representations into a frozen text embedding space, evaluated by cross-modal retrieval. Sentence meaning is linearly accessible but highly uneven across languages and resource tiers. Cross-lingual probes can match language-specific probes on languages never seen in probe training, but per-model monolingual decodability does not predict transfer. Within the OmniASR wav2vec family, scaling from 300M--7B disproportionately benefits high-resource languages, and ASR tuning sharply improves semantic decodability while making representations more language-specific. However, semantic content is entangled with acoustics, inflating apparent performance on single-speaker corpora.

0.28

The Bairong System for MLC-SLM 2026: Dynamic Question-Aware Evidence Routing for Multilingual Conversational Speech Understanding

cs.CL cs.SD ·2026-09-02 ·Shangkun Huang, Junchao Hu, Huan Shen, Guoji Wang, Yingao Wang +3 more
similarity 0.72
Abstract · Accepted at the 2nd MLC-SLM Challenge and Workshop, INTERSPEECH 2026

Long multilingual conversational spoken question answering requires systems to balance long-range transcript semantics with sparse acoustic and speaker-sensitive cues. We present the Bairong system for the MLC-SLM 2026 Challenge, where a diarization-ASR front-end produces speaker-attributed transcripts and a dynamic evidence router constructs question-specific inputs for answer prediction. Instead of applying a fixed transcript-only or audio-only policy, the router infers the required evidence type and context scope from the question and answer options, and selects among full transcript context, local audio-text fusion, speaker-linked evidence, and compact global acoustic samples. This transcript-backbone design keeps discourse context available while activating audio only when it provides complementary evidence. Our Task 1 system achieves 25.70% and 18.44% tcpMER on the development and evaluation sets. For Task 2, the final system obtains 94.84% devel?opment accuracy, outperforming the full-transcript baseline by 1.68 points and the best audio-centric diagnostic system by 2.77 points. These results support dynamic question-aware routing as an effective evidence allocation strategy for conversational spoken QA.

0.46

EXAM2: Extending Audio Understanding in Multilingual and Multimodal Analysis

cs.SD cs.AI ·2026-08-24 ·Jiawen Wang, Xiaoxue Gao, Zi Haur Pang, Nancy F. Chen
publishedbenchmarkcode/data similarity 0.70
Abstract · 9 pages, 2 figures

Recent large audio language models (LALMs) have achieved impressive progress in audio understanding. However, existing evaluations remain largely constrained to English and narrow audio domains. Prior benchmarks typically focus on a single audio modality, i.e., speech, sound, or music, limiting the systematic investigation into how these models generalize across diverse visual scenarios. In this paper, we introduce EXAM$^2$, a benchmark for multilingual and multimodal audio understanding spanning six languages and multiple modalities, including speech, sound, music, mixed-audio settings, and visual images. By incorporating visual information alongside heterogeneous audio inputs, EXAM$^2$ enables more realistic evaluation of scene-aware audio reasoning and cross-modal comprehension. EXAM$^2$ comprises $5,667$ multiple-choice questions, $22,614$ image instances, and $135,684$ multilingual translations. We evaluate state-of-the-art open-source and proprietary LALMs as well as multimodal LLMs, revealing substantial performance gaps in multilingual and cross-modal understanding. Furthermore, we propose Gemma3n-EXAM$^2$, a lightweight fusion-model fine-tuned on EXAM$^2$-train, achieves up to $15.8\%$ improvement in multilingual settings and $16.5\%$ gains in multimodal evaluation over a strong baseline. Empirical results establish EXAM$^2$ as a challenging benchmark and pioneer future multilingual and multimodal audio intelligence research.

0.43

The Second MLC-SLM Challenge: Multilingual Conversational Speech Diarization, Recognition, and Understanding

eess.AS cs.SD ·2026-09-23 ·Bingshen Mu, Mingchen Shao, Zhennan Lin, Liumeng Xue, Hexin Liu +5 more
benchmark similarity 0.69
Abstract

This paper summarizes the Interspeech2026 second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge, which aims to advance the development of effective multilingual conversational speech language models. We describe the two challenge tasks: multilingual conversational speech diarization and recognition, and multilingual conversational speech understanding, together with the released real-world conversational speech dataset, evaluation protocols, and baseline systems. The challenge attracted 91 teams worldwide, with 704 valid leaderboard results and 14 technical reports across the two tasks. Based on the participating systems, we summarize representative approaches and distill practical insights into multilingual conversational speech recognition and understanding to support future research in the community.

0.22

Improving Large Audio-Language Model Embedding Baselines for Instruction-Aware Speech Retrieval

cs.SD cs.CLeess.AS ·2026-11-06 ·Chen-An Li, Cheng-Han Chiang, Hung-yi Lee
benchmarkcode/data similarity 0.68
Abstract · Code and artifacts will be uploaded to https://github.com/lca0503/IRIS

Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text instruction. Large audio-language models (LALMs) effectively process speech and text, and our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We introduce IRIS-Embed, which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. We construct about 900K training samples from real data and synthetic data, with instructions expressing requirements for one or more speech attributes. On INSPIRE, an instruction-aware speech retrieval benchmark, IRIS-Embed consistently outperforms the corresponding LALM baselines. The best model achieves 62.92% macro-average Recall@50, surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness and the benefits of audio layer weighting. On MAEB, a massive audio embedding benchmark, macro-average gains of 9.4-19.8 points over the corresponding baselines show generalization beyond speech retrieval.

0.19

PhonoVoxtLM: Exploring Single-Codebook Phonological Tokens for Speech Understanding and Generation

eess.AS ·2026-10-13 ·Mingrui Liang, Kentaro Onda, Shinji Watanabe
similarity 0.68
Abstract

We investigate how speech token representations can support both understanding and generation within a unified speech language model. We present PhonoVoxtLM, built on the existing VoxtLM framework using phonological tokens designed to retain linguistic content and prosodic information. The existing tokenizer is jointly trained for speech recognition and resynthesis. Its single-codebook units serve as both speech inputs and generation targets, without additional speech-token prediction modules. Experiments show improvements over HuBERT-based baselines across understanding and generation under different training-data scales. Further comparisons with WavLM-based tokens reveal trade-offs between recognition and synthesis, highlighting the importance of evaluating shared representations across both capabilities. We further extend the framework to spoken question answering through fine-tuning with replay. The adapted model improves answering performance over similarly adapted HuBERT baselines while retaining advantages on the original speech tasks. Our findings support single-codebook representations designed for both content and prosody as a practical choice for unified multitask speech modeling.

0.19