CS Paper Radar

AK_QMUL at ArGuard Shared Tasks: A Single Fine-Tuned JAIS for Arabic Harmful-Prompt Detection

Ali Khairallah, Arkaitz Zubiaga

cs.CL ·submitted 2026-10-26 22:25 UTC · v1
publishedhot direction

Abstract

We present AK_QMUL's submissions to the ArGuard-2026 shared tasks on Arabic harmful-prompt detection: a binary safe/unsafe task (B1) and a seven-way harm-domain task (B2). Both submissions use one backbone -- JAIS-6.7B fine-tuned with 4-bit QLoRA and a linear classification head over its last-token representation. For B1 we train a four-way head that factors the label by dialect register {safe, unsafe} x {MSA, other}, we augment the training fold with MSA prompts from AraSafe, and decode with a plain argmax to score safe/unsafe. For B2 we fine-tune the same backbone as a seven-way classifier and average the softmax outputs of five random seeds. Both submissions ranked first on their track, with macro-F1 0.984 on B1 and 0.790 on B2. We report results by dialect and by class: on B1 the model is accurate across all six dialects (93.6--99.8%) and its predicted unsafe rate tracks the true rate closely; on B2 six of the seven classes reach F1 between 0.73 and 0.98, with the residual error concentrated in the Bully class, which overlaps semantically with harassment and harm-to-others.

Comment
Published at ArabicNLP 2026 (Shared Tasks), co-located with EMNLP 2026. ACL Anthology: https://aclanthology.org/2026.arabicnlp-sharedtasks.43/
Journal ref
Proceedings of The Fourth Arabic Natural Language Processing Conference: Shared Tasks, pages 347-351 (2026)

Closest papers

Nearest neighbours by meaning across the whole index.

ArGuard Shared Task: Harmful Content Detection in Arabic Memes and LLM Prompts

cs.CL cs.AI ·2026-09-24 ·Firoj Alam, Md. Rafiul Biswas, Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Hamdy Mubarak +3 more
publishedhighly cited similarity 0.70
Abstract

ArGuard is a shared task on harmful content detection in Arabic memes and LLM prompts. It includes two tracks: Track A focuses on multimodal hate detection in Arabic memes, while Track B addresses harmful prompt detection for Arabic LLM safety evaluation. In total, 58 teams registered, 35 participated in the final evaluation, and 27 submitted system-description papers. Participating teams explored models such as AraBERT, Jais, and Qwen3-VL. The best systems achieved macro-F1 scores of 0.823 on A1, 0.419 on A2, 0.984 on B1, and 0.790 on B2. Fine-grained meme classification in A2 was the most challenging setting, partly due to sparse labels and train-test distribution shifts.

0.57

QMUL_AK at AISA-ArabicFC: Schema-Constrained Function Calling with ALLaM

cs.CL ·2026-10-27 ·Ali Khairallah, Arkaitz Zubiaga
published similarity 0.70
Abstract · Published at ArabicNLP 2026 (Shared Tasks), co-located with EMNLP 2026. ACL Anthology: https://aclanthology.org/2026.arabicnlp-sharedtasks.2…

We describe the QMUL_AK submission to AISA-ArabicFC 2026. The system adapts the Arabic-English ALLaM-7B-Instruct model to function calling with low-rank adaptation (LoRA). It combines an Arabic schema-aware prompt, dialect and no-call resampling, a token-weighted training objective, and deterministic validation of generated calls against the candidate tool schema. The same output supports Track A function calling and Track B reasoning-augmented calling. Training uses only the official task data. On the development set, the system achieves 0.9982 function accuracy and 0.8920 argument exact match, corresponding to overall A and B scores of 0.9345 and 0.9454. On the blind test set, the system reaches overall A and B scores of 0.9253 and 0.9380, ranking 3rd on both submission tracks and 1st of 21 teams on the Track C dialect-robustness diagnostic with a macro-averaged score of 0.9075 and the smallest dialect gap in the shared task (0.077). The results show that function selection is nearly saturated, while argument extraction remains the main source of error.

0.38

BanglaShield: A Script-Robust Safety Classifier and Benchmark for Bangla Native, Romanized, and Code-Mixed Prompts

cs.CL cs.LG ·2026-10-16 ·Mohsinul Kabir, Md. Tariquzzaman, Mohammad Rashidujjaman Rifat, Tharindu Ranasinghe, Nishat Raihan
benchmarkcode/datahot direction similarity 0.66
Abstract · 17 pages. Code and data: https://github.com/mraihan-gmu/BanglaShield-Model-Benchmark ; dataset: https://huggingface.co/datasets/md-nishat-00…

Safety classifiers for large language models (LLMs) are typically built and evaluated on standard-script text, yet Bangla, like many lower-resource languages, is routinely written in native script, in Latin-script romanization, and code-mixed with English, Hindi, or Urdu. We introduce BanglaShield, a resource and model for graded prompt-safety classification across these written forms. We consolidate 10,245 Bangla prompts from six sources under a four-level taxonomy (Safe, Offensive, Harmful, Severe) and render each prompt in three aligned forms that share labels and a group-aware split, enabling strictly paired comparisons. Benchmarking six contemporary LLMs, including a Bangla-specialized one, on 36,882 classification queries and 24,318 judged responses, we find that LLM classifiers label 47.0% of unsafe native-script prompts as Safe, rising to 63.4% under romanization, while response-level safety accuracy drops by 10.3 points. Three multilingual guard models also pass 30.3-57.3% of unsafe native-script prompts. Using a model's own classification as an input filter reduces harmful compliance by only about five points. In contrast, BanglaShield, a compact fine-tuned encoder, reaches 87.8 macro-F1 on native-script and 84.4 on romanized prompts, outperforming the best LLM on each form by 42.8-46.1 points with ~130x fewer parameters than Gemma 4 31B. We release the corpus, splits, judgments, and model.

0.34

Redteaming Leading Arabic LLMs with ASAS

cs.AI ·2026-08-22 ·Fidaa Abed, Haidar Khan, M Saiful Bari, Babar Khan, Abdalghani Abujabal
benchmark similarity 0.66
Abstract · 13 pages, 5 figues, 3 tables

As the adoption of large language models (LLMs) grows in Arabic-speaking regions, ensuring their safety and cultural alignment is increasingly critical. However, Arabic LLM safety remains underexplored, especially in adversarial evaluation settings. We introduce the Arabic Safety Index (ASAS), the first fully human-curated Arabic benchmark for redteaming LLMs. ASAS contains 801 prompts spanning 8 safety categories and 8 attack strategies, with ideal responses in Modern Standard Arabic (MSA). We conduct a redteaming evaluation across seven leading models with Arabic capabilities, including GPT-4o, Claude 3.7 Sonnet, and regional models such as ALLaM and FANAR. Human annotators rate responses using a structured 4-point safety scale, revealing that most models fail to defend against 50% of unsafe prompts. Our findings highlight major safety gaps in high-harm categories such as weapons and illicit substances, with direct and obfuscation-based attacks proving most effective. The results also show that language alignment does not readily transfer across languages, and that automated safety judges (e.g., GPT-4o) perform poorly compared to human annotators. ASAS provides a culturally grounded benchmark and redteaming protocol to drive progress in Arabic LLM safety.

0.19

Task Format as a Factor in Adaptation: A Controlled Comparison of LoRA and Full Fine-Tuning for Prompt-Safety Classification

cs.CL ·2026-10-24 ·John Alderete, Srirakshith Betageri, Jennifer Flynn, Alex Greenhalgh, Josh Rubin +1 more
similarity 0.65
Abstract · 33 pages (14 pages main text, plus references and appendices), 1 figure, 18 tables

Fine-tuning a language model for classification involves many design decisions. Comparisons of low-rank adaptation (LoRA) with full fine-tuning are common but often confounded, and task format is rarely studied. We compare both regimes on prompt-safety classification with a matched 400M encoder-decoder pair (Ettin) across three task formats, with up to 15 seeds per cell and pre-specified analyses. At the pre-specified learning rates, full fine-tuning leads LoRA by 2-3 pp in every architecture and format trained on the label alone. On the encoder, the gap closes once both rates are chosen by the same search. Training LoRA to write a reason after its verdict adds 4.0 pp and closes the gap, but a content-free sentence of equal length does not. Trained models score above a prompted 8B in domain but fall behind it on two of three out-of-distribution corpora, trailing by 8-23 pp on adversarial jailbreaks and flagging 27-66% of benign but harmful-looking prompts as unsafe, against the 8B's 2-3%. Together, the results suggest that the adaptation regime matters less than how each model is trained, its learning rate, and training target, and that in-domain accuracy alone is an incomplete guide to robustness.

0.28

TTLab at AlexandriaX-2026: A Fine-Tuned Surface Tagger for Arabic Machine-Translation Error-Span Detection and Classification

cs.CL cs.AIcs.LG ·2026-09-24 ·Ali Abusaleh, Bhuvanesh Verma, Alexander Mehler
publishedcode/data similarity 0.65
Abstract · Accepted at ArabicNLP 2026, shared task AlexandriaX-2026

We present TTLab's submission to the AlexandriaX-2026 Subtask~3 on Arabic MT error span detection and classification. Our system frames the task as token-level classification over surface forms, preserving character offsets to ensure exact alignment with the evaluation metric. To handle severe label imbalance, we employ a focal loss with class weighting and dialect-specific decoding thresholds. Among six Arabic pre-trained encoders, MARBERTv2 achieves the best overall performance of 40.8 and 40.91 on the development and test set, respectively, ranking $\nth{3}$ out of all participating teams. While our system localizes error spans effectively, classification of rare error types remains challenging, highlighting the need for data augmentation for tail categories. The code is available at ${\href{https://github.com/ENTAILab/arabic-dialectal-mt-error-span-detection}{\faGithub~ TTLab at AlexandriaX-2026}$

0.77

SZED at ArGuard Shared Tasks: Multi-Task Label-Attention and Cross-Modal Fusion for Arabic Hateful Meme Detection

cs.CL cs.AIcs.CVcs.LG ·2026-10-18 ·Selma Khelili, Zohra Abdeli
published similarity 0.64
Abstract

We present SZED's submission to ArGuard 2026 Track A: Arabic hateful meme detection. The task has two parts: binary classification (A1) and fine-grained multi-label tagging (A2). For A1, we fuse MARBERTv2 and CLIP ViT-L/14 embeddings, comparing concatenation, self-attention, and ensemble heads. For A2, we use a multi-task label-attention architecture with per-class query vectors and an auxiliary head for cascade calibration. We reach macro-F1 = 0.7597 on A1 and 0.3142 on A2 (averaged over 10 active classes). We report reproducibility across runs, per-class error patterns, and the effect of cascade calibration on the long-tailed A2 label space.

0.43

BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models

cs.CL cs.CR ·2026-08-22 ·Md. Rakibul Hassan, Muhammad Iqbal Hossain
publishedbenchmark similarity 0.64
Abstract · Accepted at the 4th International Conference on Computing Advancements (ICCA 2026). 8 pages

Bangla large language model (LLM) safety is difficult to evaluate with English-centric or standard-script benchmarks because Bangla users routinely write across scripts, spellings, code-mixed forms, and regional registers. This paper presents BanglaVeilGuard, a compact Bangla-first safety benchmark and lightweight prompt guard for six language forms: standard Bangla, Romanized Bangla, Banglish, code-mixed Bangla--English, noisy Bangla, and dialectal Bangla. The benchmark contains 2,366 quality-filtered prompts and a held-out 354-prompt evaluation split spanning unsafe, safe, and safe-sensitive requests. BanglaVeilGuard uses non-destructive multi-view normalization with a prompt-risk classifier and thresholded pre-generation gate, allowing it to screen prompts for heterogeneous target models without changing their weights. Across target-model families, guarded runs reduce attack success under deterministic response scoring from 93.8--100.0\% to 6.3\% for Claude Opus 4.8, BanglaLLama, and TituLLM; TigerLLM-1B with BanglaVeilGuard achieves 78.2\% accuracy with 8.8\% ASR. The prompt guard also attains 88.5\% unsafe recall, substantially above the evaluated prompt-only guard baselines. The main remaining cost is over-refusal on dialectal and noisy benign prompts, revealing a concrete safety-helpfulness frontier for Bangla LLM deployment.

0.55