CS Paper Radar

QMUL_AK at AISA-ArabicFC: Schema-Constrained Function Calling with ALLaM

Ali Khairallah, Arkaitz Zubiaga

cs.CL ·submitted 2026-10-27 21:43 UTC · v1
published

Abstract

We describe the QMUL_AK submission to AISA-ArabicFC 2026. The system adapts the Arabic-English ALLaM-7B-Instruct model to function calling with low-rank adaptation (LoRA). It combines an Arabic schema-aware prompt, dialect and no-call resampling, a token-weighted training objective, and deterministic validation of generated calls against the candidate tool schema. The same output supports Track A function calling and Track B reasoning-augmented calling. Training uses only the official task data. On the development set, the system achieves 0.9982 function accuracy and 0.8920 argument exact match, corresponding to overall A and B scores of 0.9345 and 0.9454. On the blind test set, the system reaches overall A and B scores of 0.9253 and 0.9380, ranking 3rd on both submission tracks and 1st of 21 teams on the Track C dialect-robustness diagnostic with a macro-averaged score of 0.9075 and the smallest dialect gap in the shared task (0.077). The results show that function selection is nearly saturated, while argument extraction remains the main source of error.

Comment
Published at ArabicNLP 2026 (Shared Tasks), co-located with EMNLP 2026. ACL Anthology: https://aclanthology.org/2026.arabicnlp-sharedtasks.203/
Journal ref
Proceedings of The Fourth Arabic Natural Language Processing Conference: Shared Tasks, pages 1580-1584 (2026)

Closest papers

Nearest neighbours by meaning across the whole index.

AK_QMUL at ArGuard Shared Tasks: A Single Fine-Tuned JAIS for Arabic Harmful-Prompt Detection

cs.CL ·2026-10-26 ·Ali Khairallah, Arkaitz Zubiaga
publishedhot direction similarity 0.70
Abstract · Published at ArabicNLP 2026 (Shared Tasks), co-located with EMNLP 2026. ACL Anthology: https://aclanthology.org/2026.arabicnlp-sharedtasks.4…

We present AK_QMUL's submissions to the ArGuard-2026 shared tasks on Arabic harmful-prompt detection: a binary safe/unsafe task (B1) and a seven-way harm-domain task (B2). Both submissions use one backbone -- JAIS-6.7B fine-tuned with 4-bit QLoRA and a linear classification head over its last-token representation. For B1 we train a four-way head that factors the label by dialect register {safe, unsafe} x {MSA, other}, we augment the training fold with MSA prompts from AraSafe, and decode with a plain argmax to score safe/unsafe. For B2 we fine-tune the same backbone as a seven-way classifier and average the softmax outputs of five random seeds. Both submissions ranked first on their track, with macro-F1 0.984 on B1 and 0.790 on B2. We report results by dialect and by class: on B1 the model is accurate across all six dialects (93.6--99.8%) and its predicted unsafe rate tracks the true rate closely; on B2 six of the seven classes reach F1 between 0.73 and 0.98, with the residual error concentrated in the Bully class, which overlaps semantically with harassment and harm-to-others.

0.70

Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects

cs.CL cs.AI ·2026-11-07 ·Hossam Elsafty, Tobias Deußer, Lorenz Sparrenberg, Rajkumar Ramamurthy, Rafet Sifa
similarity 0.66
Abstract · 28 pages, including references and appendices

How does Jev compare with other systems on Arabic tasks, and how does its performance change between Modern Standard Arabic (MSA) and dialects? We evaluate 67 systems, including 47 decision models and 20 large language models (LLMs), on 43 tasks across 9 task families using a shared zero-shot protocol. The Arabic-focused frozen suite includes English controls in three parallel resources, which we also remove in a separate descriptive comparison. Jev 1.14.0 achieves a mean task-primary metric of 69.24 on a scale multiplied by 100, compared with 68.77 for Jev 1.13.0 and 71.81 for Gemma-4-31B, the highest-scoring system on this aggregate. Removing the recorded English controls, repeated cross-task items or both leaves the full system order unchanged. Relative performance and gains between Jev versions vary across task families. In content-matched comparisons within Jev 1.14.0, pooled dialect-minus-MSA accuracy gaps range from -7.94 to -0.35 percentage points across four datasets; dialect-specific results show larger variation, and SIB-200's source-clustered interval includes zero. Paired contrasts with Jev 1.13.0, Gemma-4-31B and Jais-2-8B show a larger Palestinian banking penalty for Jev 1.14.0 than for either LLM under simultaneous uncertainty; the other gap contrasts remain inconclusive. These descriptive findings show why aggregate scores should be read alongside task-family results, coverage and matched language comparisons. They do not establish a general model ranking for dialect robustness or explain the causes of the observed gaps.

0.18

alkhder at AlexandriaX-2026: Context-Aware English-to-Dialectal Arabic Dialogue Translation with QLoRA-Fine-Tuned NileChat-3B

cs.CL ·2026-10-15 ·Hasan Alkhder, Maher Pasa, Maria Hanini, Amro Najjar, Ahmet Zengin
published similarity 0.64
Abstract

This paper describes team Alkhder's submission to Subtask 1 of the AlexandriaX-2026 shared task, which asks systems to translate English dialogue turns into thirteen Arabic dialects. We fine-tune NileChat-3B-Base with QLoRA on the Alexandria parallel dialogue corpus, training one set of adapter weights for all dialects. Each turn is translated conditioned on the target country and dialect, the domain, the participant roles, the speaker-addressee gender direction, and the preceding dialogue context. At inference, conversations are translated autoregressively and the model is fed its own previous outputs back as context without using any gold target-side test information. Under the constrained track, using only Alexandria data, it trains in about one hour and translates the private test set in under two hours. The system reaches a country-macro-averaged spBLEU of 20.36 and chrF++ of 36.64 on both tracks.

0.38

TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification

cs.CL cs.AIcs.LG ·2026-09-24 ·Ali Abusaleh, Bhuvanesh Verma, Alexander Mehler
publishedcode/data similarity 0.60
Abstract · Accepted at ArabicNLP 2026, shared task AlexandriaX-2026

We present TTLab's submission to the AlexandriaX-2026 Subtask~3 on Arabic MT error span detection and classification. Our system frames the task as token-level classification over surface forms, preserving character offsets to ensure exact alignment with the evaluation metric. To handle severe label imbalance, we employ a focal loss with class weighting and dialect-specific decoding thresholds. Among six Arabic pre-trained encoders, MARBERTv2 achieves the best overall performance of 40.8 and 40.91 on the development and test set, respectively, ranking $\nth{3}$ out of all participating teams. While our system localizes error spans effectively, classification of rare error types remains challenging, highlighting the need for data augmentation for tail categories. The code is available at ${\href{https://github.com/ENTAILab/arabic-dialectal-mt-error-span-detection}{\faGithub~ TTLab at AlexandriaX-2026}$

0.77

Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context

cs.CL cs.AI ·2026-09-12 ·Nawar S. Alseelawi, Mustafa S. Aljumaily
benchmark similarity 0.60
Abstract · 11 pages, 3 figures, 1 table. Live leaderboard: https://mizan-bench.onrender.com

Arabic large-language-model (LLM) evaluation has matured around Modern Standard Arabic (MSA): aggregated leaderboards such as the Open Arabic LLM Leaderboard (OALL), HELM Arabic, and BALSAM rank models across dozens of MSA tasks, and frontier systems increasingly saturate them. Dialectal Arabic, the language Iraqis actually speak, remains nearly invisible to this infrastructure. We introduce Mizan ("the balance"), Iraq's national benchmark for evaluating LLMs on Iraqi Arabic and the Iraqi civic context: an MSA baseline track paired with an Iraqi track across six axes (dialect comprehension, dialect generation, bidirectional MSA-Iraqi translation, Iraq-specific knowledge, official-document field extraction, and safety), built from 340 originally authored, dually reviewed items with statistically audited answer positions and Wilson intervals on every published score. A pilot evaluation of 27 systems, spanning closed frontier models three days after release, open weights across size tiers, and an Arabic trio of commercial, open-specialized, and sovereign systems, yields four findings. The MSA track saturates while the Iraqi track discriminates, with a consistent 14-18-point per-model gap and statistically tied leaders. Official-document extraction confines every system to 32-56. Arabic-focused specialization behaves as MSA specialization: two dedicated Arabic models score below a size-matched generalist on the Iraqi track. And the safety-hardened tier of the newest model family deterministically refuses innocuous dialect-comprehension items as policy violations, an over-refusal mode invisible to MSA benchmarks. The platform enforces an integrity protocol of immutable snapshots, verification certificates, a human publication gate, and public retraction, all exercised during this study. Code and the public development set accompany the paper.

0.54

MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents

cs.AI ·2026-09-17 ·Pritish Mishra, Ishaan Kumar, Akshat Mandoli, Sudarshan Kamath
benchmark similarity 0.60
Abstract

Generally, most voice agents are cascaded systems, i.e., an ASR model transcribes the caller's audio, a language model reads the transcript and decides what to say and which backend tools to call, and a TTS model speaks the reply. Nearly all of the decision making happens in the language model, but existing evaluations measure it either too broadly or too narrowly. End-to-end voice benchmarks score the full pipeline, so recognition errors and model errors mix into a single number. LLM benchmarks isolate the model but they do not evaluate what makes real phone calls hard, such as transcription issues, caller's voice being split across messages and the requirement that replies follow the language and script specified. We introduce the Multi-Turn Voice Agent Benchmark (MTVA-Bench), which evaluates the language model on the same conditions it faces inside a cascaded system. The caller is played by an LLM following a set of rubrics and tool calls are answered by a mock backend which responds to the arguments the model actually sent. The benchmark contains 49 agents working across 490 reviewed scenarios and supports 7 languages. Scoring is a combination of deterministic checks on tool calls with two LLM judges, one that scores scenario specific rules and one that grades conversation quality without access to the task. Both judges must cite specific messages from the transcript. Task and conversation scores are weighted equally, since a call can complete its task and still go badly for the caller. In a seven-model study, six of the models select the correct tool within 6.4 points of one another, but their overall scores span 24.4 points. Most of the gap comes from argument values, action ordering, rule compliance, and what the model says around its tool calls.

0.48

Itgan at NADI 2026 shared task: Parameter-Efficient Whisper Adaptation for Robust, Mixed-Dialect and Code-Switched Arabic ASR

cs.CL cs.AIcs.SD ·2026-10-07 ·Ibrahim Almajai
published similarity 0.60
Abstract · 12 pages, Arabic NLP 2026 Shared Task

We describe the Itgan systems for the three ASR subtasks of NADI 2026, namely robust country-level ASR (1.1), mixed-dialect ASR (1.2), and Tunisian code-switched ASR (1.3). All three share one recipe, Whisper adapted with LoRA on consumer GPUs, and each was carried by a different addition to it. On 1.1, where the dialect label is given at test time, per-dialect specialists continued from a pooled adapter gave the largest gain, and the submitted system reached 57.1% country-average WER. A post-evaluation linear probe on frozen encoder features routes utterances without the label and recovers 44% of what oracle routing gives. On 1.2 the choice of base model mattered more than adapter capacity, and system combination helped only once we added a decorrelated member, reaching 46.7% WER. On 1.3 our system placed second at 14.49% WER with the lowest CER among the leading submissions, 5.38%. Its last 0.60 WER points came without further training, mostly from an exact weight-space average of independently trained runs, with ROVER voting adding the remainder. Every comparison carries a paired-bootstrap test, and we report eight directions that did not work.

0.58

Task-Preserving Multilingual Instruction Tuning for Open-Ended QA in WMT 2026 MIST

cs.CL ·2026-10-21 ·Aleix Sant, Jordi Luque, Carlos Escolano
published similarity 0.59
Abstract

This paper describes our submission to the WMT 2026 Multilingual Instruction Shared Task. We adapt Qwen3-8B with LoRA using organiser-provided data, EuroAlpaca, a multilingual instruction dataset created through task-preserving localisation, and CrossEuroAlpaca, a cross-lingual augmentation that assigns different source and target languages to instructions, contexts and responses. Our system targets open-ended question answering (QA-OEG) in ten shared-task languages represented in the auxiliary data. Under matched decoding, adaptation improves the mean Gemma-4 judge score on target-language QA-OEG by 7.8% relative to the base model. On an additional evaluation using the Aya Evaluation Suite across all shared-task languages, it yields relative gains of 41.5% in macro-averaged ROUGE-L and 2.9% in BERTScore-F1, with larger improvements in target languages. However, these gains are task- and language-specific. Gemma-4 scores improve for QA-OEG and summarisation in the target languages but decline for context-grounded QA there and across all three tasks in the non-target languages.

0.51