CS Paper Radar

Latent Mechanisms of Language Control in Multilingual Language Models

Ryo Mitsuhashi, Sabri Boughorbel, Majd Hawasly

cs.CL ·submitted 2026-08-31 20:21 UTC · v1
top venuecode/data

Abstract

Multilingual large language models can exhibit unintended code-switching -- unnecessarily alternating between languages during generation. We present a comparative study of three methods that identify language-controlling latents in cross-layer transcoders: activation value-based selection (ValSel), activation frequency-based selection (FreqSel), and LLM-generated latent annotation-based selection (AnnSel). To evaluate the efficacy of these methods in identifying language-controlling latents, we introduce two multilingual benchmarks that exhibit code-switching for fine-grained analysis of language steering across seven languages. Through targeted intervention experiments on Gemma-2-2B and Qwen3-4B, we find that all three methods effectively manipulate generation language, with FreqSel achieving the strongest overall performance, while AnnSel offering interpretable latent selection through explicit language annotations. A knock-out analysis suggests the methods select non-overlapping but each-functional latent subsets, indicating redundancy rather than a single canonical language direction. Code and data can be found at https://github.com/rm-3284/Latent-Mechanism-Multilingual.

Comment
23 pages, accepted for presentation at EMNLP 2026 main track

Closest papers

Nearest neighbours by meaning across the whole index.

Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergence

cs.CL ·2026-09-24 ·Dries Rooryck, Alex Cai, Yonatan Belinkov, David Alvarez-Melis, Kianté Brantley
code/data similarity 0.72
Abstract · 17 pages, 8 figures. Accepted to the BabyLM Workshop at EMNLP 2026

Children in multilingual communities often code-switch, using multiple languages in a single utterance. Can we induce cross-lingual alignment in language models by training on code-switched text? We pretrain small decoder-only transformers on two 100M-word multilingual corpora: a base corpus formed by mixing the English, Dutch, and Chinese BabyBabelLM datasets, and a corpus generated from it by inserting word- and sentence-level code-switching using an LLM. We find that training on code-switched data aligns the representations of parallel text, particularly across different scripts, and that this alignment persists through training on monolingual documents. Under a learning curriculum that progresses from word-level code-switching, to sentence-level code-switching, to monolingual documents, models trained on code-switched data outperform baselines trained without it on the BabyLM evaluation suite. Our work characterizes code-switching curriculum learning as an effective data augmentation method for multilingual pretraining. We release our code, data, and models at https://github.com/drooryck/multilingual-macaroni.

0.53

Language Defaults Before Content: A Causal Layer Map of Multilingual Generation

cs.CL ·2026-11-05 ·Shenbin Qian, Yves Scherrer
similarity 0.69
Abstract

Multilingual generation requires large language models (LLMs) to determine both what to say and in what language to say it, yet these two commitments are difficult to disentangle from model outputs alone. We use translation as a controlled probe: for multi-parallel segments that a model translates well in one language direction but poorly in another, content is held fixed while language varies. Using activation patching, we write residual-stream states from successful runs into failing ones and compare content-matched states with content-independent controls, allowing us to trace content-specific and language-related signals across layers. Across three LLMs of different scales and architectures, we find a consistent three-stage pattern of multilingual generation. Intermediate layers first express a default English signal; content information becomes causally available one layer later; and the final target language is resolved only at later layers. The ordering of the English default before content also appears in a reading-comprehension task. Our sparse autoencoder analyses reveal a further dissociation: the intermediate language signal is only weakly represented by sparse features, whereas a later language distinction is clearly sparse, but resists intervention at the patched position. These results provide a mechanistic account of how language selection and content information emerge and interact during multilingual generation.

0.17

Recurrence Is Not Enough: Causally Validating Multilingual SAE Translation Features in Gemma 2 and 3

cs.CL cs.AI ·2026-09-04 ·Giang Son Nguyen, Nhi Ngoc-Yen Nguyen, Wray Buntine, Dung D. Le
similarity 0.69
Abstract · Accepted to BlackboxNLP 2026 Special Track

Sparse autoencoder (SAE) features are increasingly used to explain and steer language-model behavior, but it remains unclear whether a feature found in one language context plays the same causal role when processing prompts in another language. We study this question using translation-initiation features (Wu et al., 2026). We reproduce the SAE feature discovery method from Wu et al. in Gemma 2 and extend it to multilingual settings that vary prompt language, source language, and target language. We then test whether features that recur across settings affect translation behavior by amplifying or ablating their activations during inference. We also examine whether the method can be applied to Gemma 3. In both models, we observe an identical finding: although we can find more than 20 features that activate frequently across all discovery settings, causal validation shows that nearly all have small or inconsistent effects. In contrast, one feature -- Gemma 2's (L10, 5717) and Gemma 3's (L20, 2456) -- consistently improves COMET scores when amplified and degrades them when ablated across 23 language settings. These results show that feature recurrence can overstate cross-lingual transfer, while identifying a language-agnostic translation-initiation direction in Gemma 2 and Gemma 3.

0.31

Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture

cs.CL cs.AI ·2026-08-11 ·Yunsu Kim, Kaden Uhlig, Ashwin Purohit, Milind Agarwal, Patrick Simianer +8 more
benchmarkcode/data similarity 0.69
Abstract

Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Terminal-Bench-LILT, a suite of 300 authentic coding tasks in ten languages: Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese. Each task targets issues specific to non-English software development that have no direct English equivalent, e.g., internationalization, encoding, text normalization, and cultural conventions. All tasks are authored by native-speaker programmers and validated through a multi-stage quality control pipeline. Evaluation of six frontier models reveals that even the strongest model reaches only 63.1\% pass rate, with many tasks unsolved by any model. Performance varies substantially by language and does not track general coding benchmark rankings, highlighting that multilingual coding competence is a distinct and underexplored capability axis. Sample tasks are available at https://github.com/lilt/terminal-bench-lilt

0.24

LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR

cs.SD cs.LG ·2026-10-12 ·Yuxuan He, Genshun Wan, Pengcheng Li, Gongping Huang, Jianqing Gao +1 more
top venue similarity 0.69
Abstract · 5 pages, 2 figures, 2 tables

Code-switching (CS) speech, characterized by language alternation within a single utterance, is prevalent in multilingual communities and poses challenges to automatic speech recognition (ASR) due to rapid language transitions. We propose LLM-HB, a language-aware large language model-guided hotword biasing framework for code-switching ASR (CS-ASR). The framework incorporates a mixture-of-experts (MoE) adaptor and an auxiliary language prediction head to better capture and differentiate multilingual representations, while hotwords are injected through LLM-based prompting. Experiments on the ASRU2019 code-switching challenge test set demonstrate that LLM-HB achieves a 20.30% relative reduction in mixed error rate (MER) over the baseline, reaching 5.85% MER under a 15-distractor hotword setting. To the best of our knowledge, this is the first work to integrate LLM-guided biasing into code-switching ASR. We further release the extracted ASRU2019 hotword list to support future research.

0.28

IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing

cs.CL ·2026-09-10 ·Pruthwik Mishra, Rudra Trivedi, Avi Patel, Ashok Urlana, Shrikant Malviya
benchmark similarity 0.67
Abstract · 9 pages, 9 tables

Language identification in code-mixed text, largely observed in social media, is highly essential when users frequently switch between multiple languages within a single utterance. Accurately identifying the languages of code-mixed tokens becomes an urgent necessity. Traditional language identification models, designed for monolingual text, are not well suited for token-level language identification in code-mixed settings. We formulate the task as a sequence labeling problem and fine-tune contextual transformer-based models MuRIL and XLM-RoBERTa best suited for Indian languages. We evaluate these systems on three different data configurations (Hindi, Gujarati, and Bengali) to predict language labels for individual tokens. We release a benchmark for language identification in code-mixed tokens with manually annotated test sets. We propose two approaches of code-mixed generation using parallel sentences of three languages. The trained models demonstrate the effectiveness of contextual embeddings for token-level language identification in multilingual social media text. For reproducibility and to facilitate future research, we publicly release our fine-tuned models.

0.23

Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation

cs.CL ·2026-08-25 ·Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett
top venue similarity 0.67
Abstract · EMNLP 2026 CR

Crosslingual evaluation of language models that enables fair comparisons remains a fundamental challenge in multilingual NLP. Existing studies adopt a variety of downstream tasks and intrinsic metrics with different theoretical justifications, yet there has been little empirical investigation into whether these approaches yield meaningful crosslingual conclusions. We systematically examine crosslingual evaluation approaches using controlled monolingual language models trained on parallel data with varying tokenizer vocabulary sizes and model sizes, and further validate our findings on multilingual LLMs. We further discuss challenges in achieving comparable downstream evaluation across languages. Our results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences. In contrast, sentence-level negative log-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons.

0.63

PRISM: Transfer and Limits of Activation Interventions in Tool-Calling Language Models

cs.CL ·2026-10-14 ·Giovana Kerche Bonás, Thales Sales Almeida, Marcelo da Silva Reis, Rodrigo Nogueira, Marcos Meideiros Raimundo
code/data similarity 0.67
Abstract

Recent literature shows that unsafe tool calls can often accompany textual refusals in language models. To investigate this hidden vulnerability, we introduce PRISM, a framework for evaluating the transfer of activation interventions across precision, languages, and scales. Our findings demonstrate that leveraging lower-resource settings (specifically quantized NF4 labels and cross-lingual transfer) offers a robust pathway to expose and improve model alignment. Notably, using behavioral labels from an NF4 copy enables stable intervention estimation even with very few scenarios. Furthermore, these alignment improvements transfer effectively across languages: five-language NF4-supervised directions successfully transfer to held-out Arabic, reducing unsafe violations in Llama-3.1-8B by 21.95 percentage points without target-language tuning. By mapping paired benign activations, we also successfully transfer these safety interventions to much larger models (up to 70B parameters). Ultimately, PRISM highlights how auxiliary supervision from compressed models and multilingual data can uncover and mitigate hazardous tool-calling behaviors across different model families. We release all experimental code and supporting artifacts at https://github.com/GiKerche/prism, alongside an interactive explorer of recorded model responses and experimental results at https://gikerche.github.io/prism/.

0.28