CS Paper Radar

Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam

Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani, Sujoy Sarkar

cs.CL ·submitted 2026-08-30 18:41 UTC · v1
benchmark

Abstract

Language is humanity's most consequential technology, yet for over a billion speakers across India's twenty-two constitutionally recognised languages, its digital layer remains structurally incomplete. Named Entity Recognition (NER), the foundational step in transforming raw text into machine-interpretable knowledge, has been studied exhaustively for English but remains largely unsolved across most Indic languages. This paper presents a rigorous comparative study of generative and encoder-based neural architectures for NER on all eleven languages of the Naamapadam benchmark. We evaluate five classic model families spanning sequence-to-sequence transformers and multilingual encoders; four decoder-only large language models (LLMs) fine-tuned with LoRA and 4-bit NF4 quantisation; and nine generative models in zero-to-5-shot inference. Under strict CoNLL span-level evaluation, encoder-based models (mBERT and XLM-R, both F1=0.675 on Hindi) substantially outperform every generative architecture in ten of eleven languages, with gaps of 7.5-40 percentage points against the strongest competitor (Gemma-2-2B: avg F1=0.427). The best few-shot result reaches only 28% of the encoder baseline. We identify three language clusters--encoder-dominant, partial-coverage, and failure-zone; and provide actionable deployment guidelines grounded in transfer learning and low-resource NLP principles.

Closest papers

Nearest neighbours by meaning across the whole index.

Cross-Dialect NER for Bangla Regional Dialects Using Leave-One-Dialect-Out Cross-Validation and Explainable AI

cs.CL ·2026-09-18 ·Shamim Rahim Refat, Faika Fairuj Preotee, Shuvashis Sarker, Shifat Islam, Bidyarthi Paul +1 more
publishedbenchmark similarity 0.71
Abstract · Accepted for Publication in IEEE 3rd INTERNATIONAL CONFERENCE ON COMPUTING, APPLICATIONS AND SYSTEMS (COMPAS 2026)

Bangla, the seventh most spoken language in the world, exhibits significant regional dialectal diversity, with dialects such as Barishal, Chattogram, Sylhet, Noakhali, and Mymensingh differing in lexical, morphological, and syntactic characteristics. These variations pose substantial challenges for Named Entity Recognition (NER), limiting the generalization of models trained on Standard Bangla or a single regional dialect. This paper presents a cross-dialect Bangla NER framework using the publicly available ANCHOLIK-NER dataset, comprising 17,405 annotated sentences and 101,817 tokens across five major Bangla regional dialects. A Leave-One-Dialect-Out Cross-Validation (LODOCV) strategy is adopted, training models on four dialects and evaluating on the remaining unseen dialect. Eight pretrained transformer-based models, including BanglaBERT, MuRIL, XLM-RoBERTa, and Multilingual-E5, are evaluated under identical experimental settings. Multilingual-E5 Large achieves the highest F1-score in every fold, peaking at 97.26% on Mymensingh and reaching its lowest, 82.38%, on Chattogram, the most challenging target dialect. To improve interpretability, Local Interpretable Model-agnostic Explanations (LIME) are applied to word-level predictions, revealing that the model's decisions are driven primarily by the surface form of the target entity word itself rather than by surrounding sentence context. These findings establish a benchmark for cross-dialect Bangla NER and demonstrate the effectiveness of transformer-based transfer learning for low-resource regional dialects, while highlighting the surface-form dependence of current models as a direction for future work.

0.26

Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER

cs.LG ·2026-09-24 ·Rakib Abdullah, Md. Maruful Islam Maruf
publishedbenchmark similarity 0.68
Abstract · 6 pages, 2 figures. Accepted at the 2026 IEEE International Conference on Biomedical Engineering, Computer and Information Technology for He…

Medical Named Entity Recognition (NER) for low-resource languages remains a challenging task due to high linguistic variability and a scarcity of domain-specific annotated corpora. This work presents a comprehensive empirical benchmark evaluating three fine-tuned transformer encoders-BanglaBERT, multilingual BERT (mBERT), and XLM-RoBERTa-against GPT-4o mini under zero-shot and few-shot prompting configurations for Bangla medical NER. In contrast to prior studies that evaluated large language models on limited subsets of only 50 samples, we conduct a large-scale evaluation across the full test set of 3,179 samples, providing statistically robust and reproducible baselines. Our fine-tuned XLM-RoBERTa model achieves an F1- score of 0.5959, establishing a new state-of-the-art and surpassing the previously reported best result of 0.5848. Crucially, we demonstrate that the language-specific BanglaBERT model consistently underperforms its multilingual counterparts with an F1-score of 0.4937, indicating that pretraining domain diversity can outweigh language specificity in highly specialized clinical settings. Furthermore, we present a detailed per-entity-type analysis for this task, revealing that Medicine and Specialist categories are recognized with high reliability, achieving F1- scores above 0.83, while the Symptom category remains the most challenging with an F1-score of 0.4367 despite being the most frequent training class. Finally, fine-tuned transformer models outperform the optimal prompting configuration by a factor of 3.76, confirming that prompt-only pipelines remain inadequate for structured clinical entity extraction in low-resource language environments.

0.19

RunyaNER: Auxiliary Language Selection for Runyankore NER

cs.CL ·2026-09-29 ·Prosper Arineitwe Asiimwe, Francois Meyer, Jan Buys
benchmark similarity 0.68
Abstract · Accepted to the 6th Workshop on Multilingual Representation Learning (MRL 2026) at EMNLP 2026. Camera-ready version. 4 figures

Cross-lingual zero-shot transfer and multilingual fine-tuning are promising approaches for NLP tasks such as Named Entity Recognition (NER) in low-resource languages, but in the absence of target language benchmarks, it is unclear which auxiliary language selection strategy leads to the best transfer. We introduce RunyaNER, the first publicly available NER benchmark for the East African language Runyankore, and use it to investigate the choice of which languages to use for transfer. Created with a semi-automated pipeline and fully manually verified, RunyaNER contains over 237k annotated words across 30k sentences. We benchmark pretrained models on RunyaNER, establishing that our dataset is of sufficient quality and size to produce effective Runyankore NER models. We then use RunyaNER to investigate auxiliary language selection in cross-lingual zero-shot and multilingual fine-tuning settings. Our experiments show that while transfer performance is highly sensitive to auxiliary language selection, embedding-based measures computed from labelled training spans correlate more strongly with downstream transfer performance than traditional linguistic features based on metadata or typology. By releasing RunyaNER and providing a systematic analysis of auxiliary language selection strategies, this work contributes both a new benchmark resource and practical insights for multilingual transfer in low-resource settings.

0.17

Quality over Quantity: Rethinking Data and Model Assumptions for Bengali Coreference Resolution

cs.CL ·2026-10-26 ·Zenith Biswas, Andrew Thomas Dyer
publishedcode/data similarity 0.67
Abstract · Accepted at KONVENS 2026, Hamburg, Germany. 11 pages, 1 figure, 5 tables. Code: https://github.com/zenith19/bengali-coref-low-resource

Multilingual transformer models are often assumed to generalise across languages, but their effectiveness for coreference resolution in low-resource settings remains unverified. We investigate this gap in Bengali, a language with over 250 million speakers but limited annotated coreference resources. We evaluate transformer-based pre-trained encoders under multiple settings, including zero-shot, fine-tuning, data augmentation, and cross-lingual transfer. Our results challenge several common assumptions about model generalisation and data efficiency in low-resource NLP. We demonstrate that a small amount of high-quality annotated data significantly outperforms larger mixed-quality datasets (p < 0.01). We find that data augmentation, particularly back-translation, does not improve performance and can degrade results in coreference resolution. Cross-lingual transfer is also inconsistent: transfer from a distant language like English reduces performance, and transfer from the related language Hindi provides no consistent benefit across models. Several widely used multilingual models exhibit collapsed embedding spaces, rendering them unsuitable for similarity-based clustering tasks. These findings highlight the importance of annotation quality over data quantity and provide empirical guidance for developing coreference resolution systems in low-resource languages.

0.41

COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages

cs.CL ·2026-09-23 ·Kshetrimayum Boynao Singh, Nitin Kumar Mishra, Palash Pratim Dutta, Atai Waris Khan, Aparna Kaushik +20 more
benchmark similarity 0.65
Abstract · 17 pages, including references and appendices

Machine translation (MT) for Indian languages remains constrained by the limited availability of high-quality, Indic-centric parallel corpora and evaluation benchmarks. Existing multilingual resources are largely constructed from English-pivot content and often fail to capture the linguistic diversity, cultural complexity, and domain-specific characteristics of Indian languages. We present COILD, an Indic-centric parallel corpus comprising over 1.16 million human-translated and human-verified sentence pairs, covering 20 Indian language pairs across the Indo-Aryan, Dravidian, Tibeto-Burman, and Austro-Asiatic language families. The corpus is built entirely from original Indian language sources collected from licensed repositories spanning eight domains with direct real-world applicability. Furthermore, we introduce a domain-centric benchmark comprising 2,000 expert-verified sentences to enable consistent multilingual and cross-lingual evaluation across Indian language pairs. To validate the effectiveness of COILD, we fine-tune two representative multilingual neural machine translation models, IndicTrans2-Distilled and NLLB-200. Experimental results demonstrate consistent improvements across language pairs, domains, automatic evaluation metrics, and human evaluation, highlighting the effectiveness of high-quality Indic-centric supervision. COILD provides a valuable training and evaluation resource for advancing multilingual machine translation and future multilingual language models for Indian languages.

0.09

Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

cs.CL ·2026-10-19 ·Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker, Anusha Kamath, Brandon Norick +22 more
benchmark similarity 0.65
Abstract · 44 pages, 5 figures. Includes appendices and references

Building capable language models for India requires scalable multilingual data and robust evaluation. We introduce Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form. It is the largest openly released Indic pretraining corpus. Using our GPU-accelerated pipeline, we translate curated English documents, assess translation quality, and repair low-quality outputs while preserving technical content and document structure and maintaining source traceability. Through continued-pretraining experiments across twelve languages, we show that combining our synthetic corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline. Additional experiments show how data volume, script coverage, and Indic training share affect English performance. We also introduce the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation. The suite combines existing Indic benchmarks with task-aware translations of English benchmarks, using automated quality assessment, targeted correction, and native-speaker review in a human-in-the-loop pipeline. We use the suite to comprehensively benchmark contemporary open-weight Indic and multilingual models across capabilities and languages, revealing substantial variation across tasks and languages. Together, these resources provide a scalable foundation for developing and evaluating more capable Indic language models.

0.16

A Scalable Framework for Automated NER Annotation Correction in Low-Resource Languages

cs.CL cs.AI ·2026-09-16 ·Toqeer Ehsan, Thamar Solorio
top venue similarity 0.65
Abstract · Accepted to Findings of EACL 2026

Poor quality or noisy annotations in Named Entity Recognition (NER), as in any other NLP task, make it challenging to achieve state-of-the-art performance. In this paper, we present a multi-step framework to enhance the annotation quality of NER datasets by employing automated techniques. We propose a frequency-based iterative approach that leverages self-training and a dual-threshold mechanism to enhance inference confidence. Experimental evaluations on different NER datasets demonstrate significant improvements in NER performance with respect to the original datasets. This work further explores the potential of generative Large Language Models (LLMs) to perform NER for low-resource languages.

0.48

Unified Multi-Dialectal Neural Machine Translation for Bangla Using the Dwadash Benchmark Corpus

cs.CL ·2026-08-12 ·Rakib Ullah, Md. Ruhul Islam, Tanbir Ahmed, Nayan Kumar Nath
benchmarkcode/data similarity 0.65
Abstract · changed to thesis to journal format, add a co-author

Neural Machine Translation (NMT) and Large Language Models (LLMs) excel at cross-lingual tasks but often fail to capture intra-lingual morphological variation, marginalizing dialectal speakers. In Bangla, existing translation frameworks commonly rely on Standard Colloquial Bangla (SCB) as an intermediate pivot, which can compound errors and reduce cross-dialectal nuance. To address this gap, we introduce a unified, multi-directional NMT system for direct translation between SCB and eleven regional variants. We first review prior dialectal NLP resources to identify existing technological gaps. As a foundational contribution, we construct and release a large multi-dialect parallel corpus for Bangla, comprising 14,562 aligned rows and 51,541 non-null sentence pairs through the integration of seven prior datasets and native-speaker-verified manual augmentation. Using this corpus, we benchmark state-of-the-art sequence-to-sequence architectures with parameter-efficient Weight-Decomposed Low-Rank Adaptation (DoRA). Results show that deep monolingual pre-training is more effective than large multilingual capacity for this task. The compact BanglaT5 model outperforms NLLB-200 and mBART-50 by up to 13.96 BLEU, achieving 29.26 BLEU, 57.26 chrF++, and 49.68 METEOR. A dataset scaling study shows diminishing returns beyond 3,000 parallel pairs and indicates that linguistic proximity to Standard Bangla is more important than raw data volume for translation quality. Finally, we deploy the optimized model as an INT8-quantized web application, providing a scalable, open-source framework for inclusive language technology and equitable digital access.

0.20