CS Paper Radar

Last Translation Benchmark

Vilém Zouhar, Niyati Bafna, Mukund Choudhary, Maike Züfle, Sara Rajaee, Pinzhen Chen, Jannis Vamvas, Sara Papi, Ona de Gibert, Bhavitvya Malik, Eliya Habba, Orfeas Menis Mastromichalakis, Patrícia Schmidtová, Michelle Wastl, Sheriff Issaka, Leshem Choshen, Stella Biderman, Antonis Anastasopoulos, Jan Niehues, Rico Sennrich, Mrinmaya Sachan, Ondřej Bojar, Kenton Murray, Jörg Tiedemann, Alham Fikri Aji, Philipp Koehn, Christof Monz, Alexandra Birch, Sowmya Vajjala, Chalamalasetti Kranti, Cristina España-Bonet, Nobin Sarwar, David Kaczér, Sourajit Saha, Jonathan Tonglet, Shunta Asano, Malik Marmonier, Daban Q. Jaff, Vaisakhi Mishra, Hend Al-Khalifa, Gabriele Sarti, Nils Rehlinger, Juan Daniel Cuervo Villa, Dominik Macháček, Saugata Purkayastha, Jagannathan Ramanujam, Shubhashis Roy Dipta, Aviral Nigam, Shuaib Shuaib Yusuf, Heejin Do, Jonathan Yahav, Johannes-Rudolf David, Maria Carmen Staiano, Zuzana Nadova, Fred Philippy, Ron Keinan, Maria Lymperaiou, Silvia Casola, Fabian Retkowski, Andrés Jerez, Hanna Yukhy- menko, Sangwon Ryu, Avantica Vempati, Sukannya Purkayastha, Adrian Cosma, Erivan Inan, Vitalii Babenko, Wafa Aissa, Valentin Scourneau, Fatima Haouari, Venkata Prasanth Kumar Gummadi, Mehdi Jafarzadeh, Manon Reusens, Kaiser Sun, Lukas Edman, Shaomu Tan, Giuseppe Gallipoli, Pawan Sasanka Ammanamanchi, Manar Ali, Mohammad Sadegh Gholizadeh, Dipankar Srirag, Marek Šuppa, Javier García Gilabert, Ruta Binkyte, Ana-Maria Bucur, Sabry E. Farrag, Youssef Saber, Yihong Liu, Theresia Veronika Rampisela, Christian Hoang, Nicoleta Cojocaru, Jan Kocoń, Jean Maillard, Xiaochuang Yuan, Sina Ahmadi, Daryna Dementieva, Philipp Mondorf, Kaustubh Dhole, Valmik Nahata, Roman Wixinger, Amir Hossein Yari, Shenbin Qian, Manuel Tuor, Fida Mohammad Thoker, Sergey Troshin, Lance Calvin Lim Gamboa, Amir Arsalan Rezapour, Kätriin Kukk, Koel Dutta Chowdhury, Shaswati Saha, Seth Aycock, Bo Chen, Linh Vu, Vatsal Venkatkrishna, Shayan Bali, Arafat Ahsan, Luan Thanh Nguyen, Hassan Soliman, Ngoc Quynh Tram Do, Azmine Toushik Wasi, R. Damanhuri, Marius Huber, Kazuki Egashira, Jimson Paulo Layacan, David Africa, Vladislav Poritski, Mike Zhang, Deep Shah, Abdulaziz Nura Kani, Luis Frentzen Salim, Paul Gavrikov, Bello Umar Bello, Ayush Sunil Munot, Anumit Garg, Yolanda Xavier, Qiaoyuan Zheng, Kawsar Ahmed, Debanshu Das, Zimu Wang, Gengyu Rao, Kamile Dementaviciute, Farhan Farsi, L D M S Sai Teja, Dawei Zhu, Yi Fan, Wei Liu, Marco Gaido, Elias Herranen, Sankalan Pal Chowdhury, Karen Sanchez, Guy Kaplan, Farzad Shami, Ashok Urlana, Amir Hossein Kargaran, Sofie Goethals, Priyaranjan Pattnayak, Oksana Volchek, Marii Ojastu, Hongbin Na, Emilian Radoi, Chenyi Zhao, Carlos Hinojosa, Andrei Niculae, Andrea Gregor de Varda, Zaid Alyafeai, Tomasz Limisiewicz, Reem Alzahrani, Pouya Sadeghi, Nehal Kathrotia, Mateusz Lango, Enzo Doyen, Alex Flückiger, Ulysses Sekai Tully Carr, Samuel Simko, Ritwik Tiwari, Rishit Dagli, Isaac R Caswell, Bowen Yi, Aicha Chorana, Selja Keränen, Sadiksha Chitrakar, Muhammad Ravi Shulthan Habibi, Joy Olusanya, Bishal Shrestha, Zhengxiang Wang, Vivek Harsha Lakkamaneni, Sophia Conrad, Panayiotis Panayiotou, Nazia Tasnim, Marta Punsola Munárriz, Marko Culjak, Luis Lara, Jenny Chim, Jannatul Nayem, Fidel Rodríguez Velásquez, Eran Yahav, Blanka Kövér, Beatrice Savoldi, Anmol Goel, Aishik Mandal, Tosin Adewumi, Raoyuan Zhao, Mykola Haltiuk, Antonia Karamolegkou, Yuxing Lu, Thura Aung, Naser Almousa, Tommaso Cerruti, Raia Abu Ahmad, Beni Egressy, Alireza Pakniat, Stéphane J. P. S. Thunus, Rachel Bawden, Lena Libon, Samridhya Biswas, Prakhar Gupta, Nusrat Jahan Lia, Nguyen Tai, Natchapon Jongwiriyanurak, Minh Ngoc Do, Ivan Barać, Dzmitry Kuzmin, Badal Nyalang, Antoine Taroni, Andy Catruna, Rushikesh Zawar, Roland Aydin, Pavel Stepachev, Ilai Yaron Levy, Andreas Simons, Rayyan Merchant, Ziyi Yang, Samuel Frontull, Kenneth Enevoldsen, Harris Abdul Majid, Tim Graf, Tatiana Bielakova, Sharifa Djurabaeva, Shaoxiong Ji, Jirui Qi, Ayla Rigouts Terryn, Yurii Paniv, Xiyan Fu, Sunisth Kumar, Shree Harsha Bokkahalli Satish, Papa Abdou Karim Karou Diallo, Mengyu Ye, Maximilian Vieweg, Matija Akrap, Kristýna Onderková, Joseph Attieh, Ivan Yuri De Leon, Ibrahim Baroud, Esrael Teferi Tensay, Elisabeth Fittschen, David Dukić, Benoît Sagot, Jingwei Ni, Yu Fan, Juri Opitz

cs.CL ·submitted 2026-09-03 17:54 UTC · v2
benchmark

Abstract

For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, opaque, and vulnerable to reward-hacking. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.

Comment
typeset in Typst

Closest papers

Nearest neighbours by meaning across the whole index.

Augmenting Text to Increase Translation Difficulty

cs.AI ·2026-08-16 ·William Kalikman, Šimon Sukup, Michal Tešnar, Vilém Zouhar
publishedbenchmarkcode/data similarity 0.77
Abstract · 18 pages, 8 figures, 10 tables. William Kalikman and Šimon Sukup contributed equally. Published in EAMT 2026. Code: https://github.com/Break…

As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models of varying quality. We propose augmenting existing benchmarks to increase translation difficulty by combining adversarial optimization with a differentiable translation difficulty estimator. Our Adversarial Translation Optimization (ATO) uses gradients from a combined difficulty and fluency objective to iteratively replace tokens. Because each step branches over candidate substitutions at every position, optimization becomes a tree search problem, which we address with Beam Search. ATO offers a gradient-based alternative to LLM-based dataset creation without LLM prompting, expensive human curation, or task-specific model training. Our ATO-modified benchmark lowers average translation quality (xCOMET) from 0.93 to 0.82, compared to 0.88 for paraphrasing and 0.86 for a zero-shot baseline. Human evaluation shows the modified texts are somewhat less natural than the baselines but remain reasonably grammatical and plausible while being substantially harder to translate. We release two datasets of 350 English texts each, generated by our methods, as well as the code.

0.78

TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs

cs.CL ·2026-09-10 ·Shenbin Qian, Yves Scherrer
publishedbenchmark similarity 0.77
Abstract · Accepted to the Eleventh Conference on Machine Translation (WMT26)

Large language models (LLMs) are increasingly used for machine translation, yet their outputs often contain additional text beyond the translation itself, such as language labels, explanations or bilingual repetitions, which we term translation noise. Despite its prevalence, this problem lacks dedicated benchmarks and systematic study. We analyze over 790,000 translation outputs from 12 LLMs across 22 language pairs (LPs) and identify 12 recurring noise patterns, which we group into formatting and content noise. Building on the observed patterns, we construct TransClean, a controlled benchmark of 9,900 pairs of noisy and clean translation outputs, comprising 8,800 synthetically generated instances and 1,100 manually curated authentic instances. We evaluate two extraction approaches on the TransClean benchmark: 1) a span-based extraction method leveraging translation quality estimation models for span detection, and 2) an LLM-based extraction method that prompts an LLM to isolate the translation. Our benchmark and analysis provide the first systematic framework to evaluate and improve the cleanliness of LLM translation outputs.

0.31

Dataset Collection and Fine-Tuning of NLLB for German--English Translation

cs.CL ·2026-11-05 ·SK Deeraj, Yash Ingle, Pruthwik Mishra
code/data similarity 0.75
Abstract · 6 pages, 5 tables

This paper describes the curation of a high-quality parallel corpus from OPUS Books and the fine-tuning of Meta's NLLB-200 models for German--English translation. We detail a dataset extraction pipeline that filters 45k raw OPUS sentence pairs to 25k high-quality examples through encoding validation, length filtering, language identification, deduplication, and quality scoring. We present a reproducible fine-tuning methodology optimized for consumer GPUs using mixed precision, gradient accumulation, and gradient checkpointing. Our fine-tuned NLLB-600M model achieves statistically significant improvements (+3.4 BLEU, +2.6 chrF++, -3.2 TER) over the base model on literary and technical German-English translation. We demonstrate practical deployment by integrating the fine-tuned model into LinguaHaru, an existing open-source document translation system, achieving format-preserving translation across PDF and Office documents. All dataset preparation scripts, training code, and evaluation tools are provided for reproducibility. The fine-tuned model is available at HuggingFace.

0.25

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

cs.CL ·2026-10-07 ·Hala Almaghout, Christian Federmann, Qin Gao
top venue similarity 0.75
Abstract · Accepted at EMNLP 2026

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.

0.41

Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems

cs.CL ·2026-10-24 ·Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev +1 more
top venuedatasetcode/datahighly cited similarity 0.74
Abstract · Published in Findings of EMNLP 2025. 17 pages. Dataset: https://huggingface.co/datasets/yandex/wmt24-en-ru-rate

As machine translation systems approach human-level quality, traditional evaluation methodologies struggle to detect subtle translation errors. We critically examine limitations in current gold-standard approaches (MQM and ESA), including inconsistencies from variable annotator expertise, excessive categorization complexity, coarse severity granularity, accuracy bias over fluency, and time constraints. To address this issue, we introduce a high-quality dataset consisting of human evaluations for English-Russian translations from WMT24, created by professional linguists. We show that expert assessments without time pressure yield substantially different results from standard evaluations. To enable consistent and rich annotation by these experts, we developed the RATE (Refined Assessment for Translation Evaluation) protocol. RATE provides a streamlined error taxonomy, expanded severity ratings, and multidimensional scoring balancing accuracy and fluency, facilitating deeper analysis of MT outputs. Our analysis, powered by this expert dataset, reveals that state-of-the-art MT systems may have surpassed human translations in accuracy while still lagging in fluency - a critical distinction obscured by existing accuracy-biased metrics. Our findings highlight that advancing MT evaluation requires not only better protocols but crucially, high-quality annotations from skilled linguists.

0.64

Generating Adversarial Texts for Machine Translation via GRPO

cs.CL ·2026-09-05 ·Florian Zogaj, Jakob Hütteneder, Giovanni De Muri, Federico Villa, Aryan Sood +1 more
published similarity 0.73
Abstract · Accepted at KONVENS 2026

As machine translation (MT) systems continue to improve, standard benchmarks become less informative for exposing remaining weaknesses. Traditional methods for creating challenging test sets rely on expensive manual creation or curation, while automated approaches struggle to produce sets with the necessary translation difficulty and linguistic diversity. We propose a scalable reinforcement-learning-based approach for rewriting existing source texts into instances that are more difficult to translate for MT systems. We fine-tune a large language model with Group Relative Policy Optimization (GRPO), using reward signals based on translation difficulty together with constraints for semantic similarity, grammaticality, and approximate length preservation. On WMT25, our approach substantially reduces average COMET translation quality from 0.63 to 0.48, while preserving grammaticality and readability, whereas the base model remains at 0.64. Evaluations on the unseen WMT19-WMT24 benchmarks confirm that this behavior generalizes beyond the training data, and human evaluation further shows that the rewrites substantially lower translation quality while incurring a moderate drop in naturalness and only a small change in grammaticality. We release our code to support reproducibility.

0.47

UvA-MT's Participation in the WMT26 General Translation Shared Task

cs.CL ·2026-10-20 ·Di Wu, Sergey Troshin, Wafaa Mohammed, Seth Aycock, Evgeniia Tokarchuk +2 more
publishedbenchmark similarity 0.73
Abstract · Accepted to WMT26

This paper presents UvA-MT's submission to the WMT 2026 General Machine Translation shared task, competing in the unconstrained track across all 23 translation directions. This year, we fully leverage the test-time methods of Large Language Models (LLMs) for machine translation. Specifically, (1) we use sequential sampling with a refinement prompt to generate a pool of translation candidates for each source segment, a strategy shown to outperform traditional parallel sampling; and (2) we employ LLMs as pairwise judges to select the best candidate via a round-robin voting mechanism, with offline evaluation on the WMT25 benchmark showing a very clear improvement over point-wise best-of-N selection. Combined, these two strategies form our WMT26 submission.

0.21

FACET at WMT 2026 Automated Translation Quality Evaluation Task

cs.CL ·2026-09-08 ·Ahrii Kim, Chanjun Park, Seong-heum Kim
published similarity 0.73
Abstract · Accepted at WMT 2026 (shared task system paper)

Different error types in machine translation require different evidence. Whether meaning is preserved can be judged only against the source, while whether the target is well-formed, or whether it names one entity consistently, can be judged from the target alone. We present FACET, our reference-free submission to the WMT26 Automated Translation Quality Evaluation Task, which decomposes evaluation into Fluency, Accuracy, and Consistency passes and gives each pass only the context its error type requires. A single fixed model is prompted three times, and the merged error spans yield the three task outputs, error spans, quality scores, and error-free labels, with no trained components. We also submit FACET-C, which omits the Consistency pass. Without gold labels, we characterize the predictions of FACET. Its system rankings place post-edited human translation first, and the Consistency pass changes about a tenth of segment scores while leaving the ranking nearly unchanged.

0.33