CS Paper Radar

The Punjabi Geet Toxicity Dataset (PGT-FG): Fine-Grained Multimodal Annotations for Toxic Content in Punjabi Songs

Lakhjeet Kaur, Jaspreet Kaur, Komal Rani, Yougal Attri, Ramakant KumarJaydeep Kishore, Pravin Kumar

cs.CL ·submitted 2026-10-23 11:38 UTC · v1
datasetcode/dataunusual

Abstract

Resources for toxicity detection in Punjabi remain limited, particularly for musical content that combines linguistic and multimodal information. We present the Punjabi Geet Toxicity Dataset (PGT-FG), a segment-level resource developed to support research on toxicity detection in Punjabi musical content. The dataset was constructed from publicly available Punjabi songs collected from YouTube. An initial collection of 620 songs resulted in 465 successfully downloaded songs and 3,402 preliminary segments from 450 songs. After quality control and annotation, the final dataset contains 2,396 unique segments from 392 songs. Four native Punjabi annotators produced 2,870 annotation records, including a shared subset of 158 segments annotated by all four. Each annotation includes binary toxicity, eight fine-grained toxicity categories, severity, sentiment, mode, and an uncertainty flag. Inter-annotator agreement was measured with Fleiss' kappa on the shared subset, and majority-based resolution produced the final segment-level labels. Of the 2,396 segments, 705 (29.42%) were labelled toxic and 1,691 (70.58%) non-toxic. Violence was the most frequent category, occurring in 539 toxic segments (76.45%), followed by gang references (227, 32.20%) and drug references (133, 18.87%). The overall toxicity label achieved substantial agreement (kappa = 0.6521). PGT-FG provides a fine-grained resource for Punjabi toxicity detection and multimodal content analysis. The resolved dataset, with one final record per segment, is publicly available on Zenodo (https://doi.org/10.5281/zenodo.23040156). Original lyrics, lyric excerpts, and copyrighted audio or video content are not redistributed.

Closest papers

Nearest neighbours by meaning across the whole index.

Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms

cs.CL cs.AI ·2026-08-23 ·Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta, Sabik Bin Sultan, Abdullah Khan Zehady
top venuebenchmark similarity 0.57
Abstract

Bengali is the seventh-most-spoken language globally, yet LLM safety evaluation remains overwhelmingly English-centric. We introduce BanglaSafe, a benchmark of 879 Bengali prompts combining 309 natively authored prompts with 570 expert-reviewed prompts, spanning 17 culturally grounded harm categories and five prompting conditions that vary language, writing style, and authority framing. Evaluating 18 frontier LLMs, we find that over half of all responses are unsafe or partially unsafe (53.6%) while 14.7% contains strictly harmful content, and that the strongest observed effect is not the switch from English to Bengali but the choice of writing style within Bengali: the same harmful request phrased as a formal newspaper investigation succeeds 17 percentage points more often than the same request phrased as a casual message, with no adversarial engineering involved. We further show that existing safety classifiers struggle to reliably evaluate Bengali content, with even frontier models failing on nearly half of all cases.

0.49

BanglishDepYT: Depression Detection in Native Bengali Social Media with Emotion-Preserving Preprocessing and Domain-Adaptive Pretraining

cs.CL cs.CYcs.LG ·2026-11-01 ·Ayon Roy, Sadman Sharif, Syed Shams Islam
datasetcode/data similarity 0.55
Abstract · 39 pages

Depression affects millions of Bengali speakers across South Asia, yet automated detection systems have relied almost entirely on datasets translated from English. Such datasets cannot capture the emoticons, romanized Bengali words, and elongated spellings that characterize real expressions of distress. We introduce BanglishDepYT, a Bengali depression dataset collected natively from YouTube, comprising 191,000 unlabeled comments and 1,000 samples verified through dual bilingual annotation. We propose EPH (Emotion-Preserving Hybrid preprocessing), which converts emoticons to semantic tags, preserves Banglish tokens, protects negation words, and normalizes emotional elongation. A key finding is that preprocessing choice substantially affects performance on informal text: standard cleaning reduces the Banglish Survival Rate to zero (SVM F1: 0.7062), while EPH preserves signals (CESS: 0.9996 vs. 0.6694) and raises SVM F1 to 0.8570, an improvement of 0.1508. We further apply domain-adaptive pretraining (DAPT), continuing SahajBERT pretraining on unlabeled in-domain comments. DAPT-SahajBERT achieves the highest mean F1 (0.8161) and MCC (0.6272), significantly outperforming the baseline (F1: 0.7805, p = 0.0013, d = 1.450) and matching MuRIL (p = 0.589). These results suggest that targeted domain pretraining is a competitive alternative to large-scale multilingual pretraining. LLM-assisted annotation using Qwen2.5-7B achieves substantial human agreement (kappa = 0.611), enabling pseudo-labeling of 96,071 comments. Semi-supervised training improves AUC from 0.8499 to 0.8731 (p = 0.044), offering a scalable path to expanding annotation. All code, models, and datasets are publicly released to support reproducible research.

0.41

BanglaDial-Abuse: A Corpus-Grounded Dataset for Regional Dialect Identification in Abusive Bangla Text

cs.CL ·2026-10-01 ·Hasin Almas Sifat
code/data similarity 0.54
Abstract · 5 pages, 3 figures, 1 table. Dataset Version 1.0 available on Zenodo: 10.5281/zenodo.23074319

Regional linguistic variation remains an important challenge for Bangla natural language processing, particularly in informal and non-standard text. This paper introduces BanglaDial-Abuse, a balanced Bengali-script dataset developed for regional dialect identification in abusive and hostile Bangla text. The dataset contains 1,000 sentences distributed equally across four linguistic varieties: Standard Bangla, Chattagram, Sylhet, and Barishal, with 250 samples per class. The resource was constructed using a corpus-grounded synthetic procedure incorporating regional variation in pronouns, possessive forms, verb morphology, negation, interrogative structures, postpositions, vocabulary, and Bengali-script spelling conventions while preserving the underlying hostile or abusive meaning. Descriptive analysis shows broadly comparable sentence-length distributions but partially distinct lexical spaces across the four classes. Pairwise Jaccard vocabulary similarity ranges from 0.37 to 0.56. The primary task is four-class regional dialect identification rather than binary abusive-text detection. The dataset is publicly available through Zenodo under a Creative Commons Attribution 4.0 license. The current version is intended as a research and prototyping corpus rather than a native-speaker-validated gold-standard linguistic resource. Keywords: Bangla, Bengali, dialect identification, regional dialect, abusive language, low-resource NLP, Chattagram, Sylhet, Barishal, dataset

0.34

Gugak-VocalSet: A Multi-Genre Studio-Quality Dataset of Korean Traditional Singing with Sigimsae Annotations

cs.SD cs.AIcs.DBcs.DL ·2026-10-11 ·Danbinaerin Han, Wonil Kim, Seah Hong, Dongjoo Moon, Jaeok Lee +3 more
hot direction similarity 0.54
Abstract

We present Gugak-VocalSet, the first publicly available audio dataset for Korean traditional singing with dense annotations of sigimsae, a structured pitch contour system of vocal expressions. Gugak-VocalSet contains 24.5 hours of unaccompanied solo vocal recordings across 305 tracks of representative repertoire curated by the National Gugak Center, performed by 20 professional singers in a controlled studio environment with fixed key and tempo. In addition to genre, lyrics, and mood/timbre tags, the dataset provides a unified sigimsae ontology and approximately 68,000 segments (71,700 instances including multi-label) of audio-grounded sigimsae annotations; the ontology adopts a functional-position-based hierarchy designed for consistent application across genres, and 14 expert annotators specializing in Korean traditional music performed segment-level labeling accordingly. Baseline experiments on sigimsae classification and event detection show that domain-informed F0 representations remain competitive with, and on classification surpass, frozen probing of a general-purpose music foundation model, while pitch and spectral cues play complementary roles in event-level detection. Companion page: https://www.danbinaerin.com/Gugak-VocalSet/

0.33

GENDIS: Gender-Related Discourse and Stance in Multilingual YouTube Comments from India, Kenya, and Nigeria

cs.CL ·2026-10-13 ·Luise Koch, Manuel Tonneau, Niyati Malhotra, Sunny Rai, Ana Maria Munoz Boudet +3 more
dataset similarity 0.54
Abstract · 23 pages. Accepted at the Workshop on Online Abuse and Harms (WOAH 2026), co-located with EMNLP 2026

Research on gender-related online content is often operationalized through harm-centered tasks that collapse non-harmful gender-related and unrelated content, while existing datasets remain concentrated in high-resource languages and Global North settings. To bridge these gaps, we introduce GENDIS, a dataset of 9,600 multilingual, human-annotated YouTube comments from India, Kenya, and Nigeria (3,200 per country), annotated using a two-level framework that first identifies whether gender is referenced and then classifies gender-related content as negative, positive, or neutral. This separation provides a denominator for gender-related discourse independent of harm, enabling both prevalence estimation and characterization of its evaluative range. In the uniform random stratum, 23.8-35.0% of comments are gender-related, and among those with a resolved stance, 82% are neutral or positive. Across models, gender-reference detection is substantially easier than stance classification, especially for positive and neutral speech. Lightly supervised classifiers match frontier zero-shot large language models at substantially lower cost and outperform generic toxicity detection, while cross-country transfer remains limited. GENDIS provides a locally grounded resource for measuring when gender is invoked online and how that reference is framed.

0.39

KunSet: A Corpus and Benchmark for Lyrics Alignment and Rhyme Retrieval in Kunqu Opera

cs.SD eess.AS ·2026-10-21 ·Lejun Min, Siqi Chen, Delong Wang, Tong Zheng, Yunqi Lu +8 more
benchmark similarity 0.53
Abstract

Kunqu is one of the oldest extant forms of Chinese opera, yet its digitization remains scarce and difficult to annotate, with two challenges standing out: first, close analysis of Kunqu performance demands syllable-level lyrics alignment; second, analyzing Kunqu pronunciation requires its phonological attributes (e.g., rhyme) to be linked with each syllable. To address these challenges, we present KunSet, the first Kunqu performance corpus comprising 482,164 fine-grained annotations of syllable-level lyrics and phonological attributes in total media duration of 155.2 h with 106.3 h annotated. To ensure annotation quality, KunSet was preannotated by an iterative ML-assisted pipeline and corrected by over 200 trained annotators via a purpose-built online annotation system under a two-round review process. To assess KunSet as a music information retrieval (MIR) benchmark, we evaluate two tasks: lyrics alignment, by finetuning a forced-alignment model with an acoustic anchor decoder; and same-rhyme retrieval, using frozen Chinese HuBERT features. Models built with KunSet outperform their baselines. We release KunSet and model checkpoints to support MIR research on traditional-music corpora.

0.14

An Explainable System for Offensive Content Detection in Nastaliq Urdu: Corpus Curation, Transformer Benchmarking, and Word-Level Attribution

cs.CL cs.AI ·2026-10-11 ·Muhammad Shahid Iqbal Malik
similarity 0.53
Abstract · 15 figures, 15 tables

Automated moderation systems are deployed at scale, yet they remain opaque to the users and moderators affected by their decisions, and their coverage is almost entirely confined to English. For Urdu written in the Nastaliq script, no such system exists, and the annotated resources it would be built on are drawn largely from Twitter/X, where character limits constrain how users write. Interpretability has received little attention, leaving it unclear what such models actually learn. This study addresses these gaps in three steps. First, we curate and publicly release a corpus of 7604 Nastaliq Urdu Facebook comments, extending an earlier corpus by correcting text-level inconsistencies and re-examining the original labels. Second, we fine-tune four Urdu-capable transformers under a common grid-search protocol and compare them against twelve feature-classifier baselines on identical data partitions. The best configuration, Multilingual Representations for Indian Languages (MuRIL), reaches 87.05% accuracy and 87.06% macro F1, improving on the strongest baseline by 5.52 percentage points, a margin that exceeds the 95% confidence interval on test accuracy; the two leading transformers differ by less than the test set can resolve, and performance is stable across five optimizers. Third, we compute word-level SHAP attributions adapted to right-to-left rendering. Attribution mass concentrates on high-frequency slurs but extends to syntactically neutral function words and to political vocabulary characteristic of the source pages, indicating that aggregate performance may rest partly on lexical and topical regularities rather than a generalizable representation of offensiveness.

0.12

Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study

cs.CL ·2026-08-18 ·Hamidreza Saffari, Francesco Pierri
similarity 0.53
Abstract · 20 pages, 16 figures, 14 tables

Online video platforms can expose young users to harmful content, but independent audits remain difficult because video annotation is costly and moderation judgments vary across languages. We audit TikTok in France, Italy, and Sweden with sockpuppet accounts representing four age personas (13, 16, 19, 40), collecting 36,971 videos from passive For-You-page scrolling and active sessions that scroll, search for harm keywords, and scroll again. To scale annotation, we validate four multimodal LLMs against native-speaker labels on a 300-video reference set. Gemini 2.5 Flash with eight sampled frames plus text performs best (aggregate kappa = 0.42), at half the per-call cost of native-video upload, and we apply it to a 10% sample for approximately \$50 in total API spend across both modalities. Keyword search returns 35-56% harmful content, a 1.5-7.5x increase over the scrolling baseline in ten of twelve country-age combinations; the spike is temporary and flattens the age differences observed in France and Sweden. Under passive scrolling, Italy has the highest harm rate at every age, with Italian age-19 reaching 48.6%. Overall, MLLM-based auditing offers a scalable approach for cross-national youth-safety audits, while provider safety filters (1.1% refusal rate) under-count the most explicit harms.

0.41