CS Paper Radar

CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

Yifan Wang, Junyu Lu, Qifan Wang, Shun Zhang, Chaozhuo Li, Jiahao Liu, Zhijun Cao, Lingbin Bu, Fanliang Bu

cs.CL cs.AI ·submitted 2026-09-18 12:57 UTC · v1
benchmarkcode/data

Abstract

Chinese social media has generated a vast and continually evolving lexicon of internet buzzwords whose meanings are often non-literal and deeply rooted in local cultural and pragmatic contexts. Existing research has primarily focused on interpreting these buzzwords within Chinese, leaving largely unexplored whether LLMs can transfer such culturally grounded knowledge across languages and accurately convey the intended meanings in English. This cross-lingual capability is also critical for safety, as harmful expressions may obscure their offensive content through culture-specific homophony, euphemism, irony, or coded language. In this paper, we investigate the ability of advanced LLMs to understand Chinese internet buzzwords across languages. To this end, we introduce CIBuzzBench, the first benchmark for cross-lingual Chinese-to-English understanding of Chinese internet buzzwords. CIBuzzBench comprises 3,001 Chinese internet buzzwords annotated with English meaning explanations, English equivalents, category labels, and harmfulness labels. Based on these annotations, we design three evaluation tasks: Meaning Explanation, Cross-lingual Equivalent Matching, and Culturally Grounded Harmfulness Detection. We evaluate representative state-of-the-art proprietary and Chinese LLMs under both English- and Chinese-prompting settings. Our results show that LLMs continue to struggle with the cross-lingual understanding of Chinese internet buzzwords, particularly in fine-grained non-literal interpretation, robust equivalent matching under option perturbations, and calibrated harmfulness detection. These findings highlight the persistent challenges posed by culturally grounded language phenomena for multilingual LLMs and safety-oriented evaluation. The dataset and code are available at https://github.com/SuperYFan/CIBuzzBench.

Closest papers

Nearest neighbours by meaning across the whole index.

VARM-Bench: Benchmarking Verifiable Structured Reasoning in Chinese Abusive Speech Moderation

cs.AI ·2026-08-16 ·Mingyu Yuan, Shengtao Wen, Lingbing Guo, Zhen Bi, Xiang Chen
benchmark similarity 0.66
Abstract

The widespread circulation of abusive online content has increased the need for reliable moderation of Chinese social-media text. Existing Chinese benchmarks support label classification, fine-grained toxicity categorization, and target-aware extraction, but do not provide a unified representation for deterministically verifying the stated basis of a moderation decision. We introduce VARM-Bench, a benchmark for field-anchored chain-of-thought rationales in Chinese abusive-speech moderation. Each instance contains a concise natural-language rationale with explicit anchors for six decisions: target, target type, target explicitness, author stance, harmfulness label, and fine-grained category. Our deterministic protocol evaluates field correctness, target alignment, output validity, complete-record agreement, and hidden record errors conditioned on correct final decisions, without relying on an LLM judge. Under a common structured-output protocol, we evaluate language models across multiple model families using zero-shot prompting, taxonomy guidance, and structured CoT supervision, and analyze lexical-cue sensitivity and field-level errors. Results show that strong label-level performance can conceal substantial errors in complete moderation records. VARM-Bench provides an auditable and reproducible benchmark for evaluating verifiable moderation rationales in Chinese abusive-speech moderation.

0.04

Social Bias in Chinese Large Language Models under Social Media Expressions

cs.CL ·2026-10-10 ·Mo Zhou
publishedhot direction similarity 0.65
Abstract · 11 pages, 5 figures

Existing Chinese social-bias benchmarks for large language models largely rely on normalized or template-based expressions, leaving the influence of internet slang, emoji, and sarcasm insufficiently examined. Safety alignment may further compress conventional plausibility ratings toward the lower end of the scale. This study constructs 73 paired probes across gender, region, and occupation. Each probe contains a neutral expression, a social-media-style expression, and a reversed-identity control. The task is reframed from direct plausibility judgment to social consensus estimation. Social-media expressions are divided into stylistic packaging, which preserves semantic polarity, and sarcastic packaging, which reverses surface polarity. Six Chinese-capable large language models are evaluated through paired differences measuring identity bias, stylistic packaging, and sarcasm shift. All models show significant identity-bias effects; five show significantly negative stylistic-packaging effects; and sarcasm shifts display a clear cross-model gradient. These findings demonstrate that expression form is not a transparent carrier in bias evaluation and that lower scores under sarcasm should not be interpreted directly as bias mitigation. The proposed paired design offers a measurement approach for Chinese social-media contexts while also revealing the construct boundary and prompt sensitivity of social consensus estimation.

0.45

CULTURE-CMT: Benchmarking Contextualized Social Media Translation

cs.CL ·2026-11-01 ·Linjuan Wu, Dingming Li, Yiwen Qiu, Xinze Lyu, Ye Guo +3 more
benchmark similarity 0.63
Abstract · 24 pages. Preprint

Social-media comments are interactions, yet they are often translated and evaluated as isolated text. This mismatch is especially problematic when meaning depends on surrounding discussion, community-specific slang, or culturally situated expressions. We introduce CULTURE-CMT, a Chinese-English benchmark for contextualized social-media comment translation that restores the post and conversational context naturally available to users. Its 572 manually verified examples are organized into Slang, Context, and Both, enabling diagnostic analysis of distinct and interacting sources of translation difficulty. To support scalable evaluation, we further develop a retrieval-augmented cultural-effectiveness Judger that combines a calibrated scoring model with dynamically retrieved cultural knowledge. CULTURE-CMT reveals several non-trivial patterns. Providing interaction context improves most model configurations, but does not eliminate culturally ineffective translations. Examples requiring both contextual grounding and culturally situated language remain substantially more difficult than either source of difficulty alone. Moreover, larger models are not consistently better, and thinking produces highly model-dependent gains, including occasional degradation. In contrast, a 27B model adapted on benchmark-disjoint social-media data achieves the lowest observed invalidity rate among the evaluated configurations under thinking inference with context. Together, these results suggest that realistic social-media translation is not simply a matter of scaling models or reasoning longer, but of learning to translate meaning as it is constructed within evolving online interactions.

0.22

TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translation

cs.CL ·2026-09-16 ·Jianan Wu
benchmark similarity 0.62
Abstract · 11 pages

Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating large language models remain limited. We present TeochewBench, a human-reviewed benchmark comprising 300 Teochew Hanzi expressions for evaluating translation from Teochew Hanzi into Mandarin Chinese and English. The dataset covers five categories: basic vocabulary; everyday sentences; Teochew-specific expressions; tone, politeness, and context; and idiomatic, ambiguous, and culturally specific expressions. A primary Teochew-speaking reviewer examined all entries individually and revised them as needed, while two additional Teochew speakers verified selected items. Our main evaluation covers 11 official general-purpose post-trained models on the reviewed dataset in both translation directions, yielding 6,600 predictions. Two official base checkpoints provide 1,200 predictions for supplementary diagnostics, bringing the total to 13 models and 7,800 predictions. We additionally include a Hanzi-copy control, which returns the source input unchanged, to assess how shared Hanzi affect automatic scores for translation into Mandarin Chinese. Qwen3.5-27B achieved the highest overall chrF-style score among the evaluated checkpoints, at 60.63, followed by Qwen2.5-72B-Instruct at 56.61, Gemma-3-27B-IT at 56.36, and GLM-4-32B-0414 at 55.82. Across the 11 main-evaluation models, the mean chrF-style score decreased from 69.25 for low-specificity items to 27.52 for high-specificity items. High-specificity expressions received lower scores and exhibited smaller cross-model differences, suggesting that they constitute a shared low-scoring region across the model families evaluated here. The Hanzi-copy control further indicates that surface overlap in low-specificity items can substantially affect automatic scores for translation into Mandarin Chinese.

0.23

A Dataset for Misogyny Detection in Polish Social Media: Transferability of Hateful Speech Detection from General to Specific Types of Hate

cs.CL ·2026-11-15 ·Ewelina Gajewska
dataset similarity 0.62
Abstract

Harmful language detection is commonly evaluated on broadly defined hate speech and toxicity benchmarks, yet it remains unclear whether models that successfully detect harmful language in general can transfer this capability to more specific forms of hate. We investigate this problem for misogyny detection in Polish, an area for which dedicated resources remain scarce. We introduce a new dataset of Polish social media comments annotated for misogyny, gender-directed hateful speech. Using this resource, we first assess the reliability of LLM-assisted annotation and then evaluate general-to-specific transfer by comparing models trained on general harmful language data, safety-specialized models, and models fine-tuned directly for misogyny detection. Our results show that strong performance on general harmful language detection does not transfer uniformly to misogyny; safety-specialized models demonstrate substantially stronger generalization ability. The study contributes a new Polish resource for gender-directed harmful language and demonstrates that the scope and granularity of prior safety training are important factors in transferring harmful language detection from general to specific forms of hate.

0.05

When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation

cs.CL ·2026-09-08 ·Yiwen Qiu, Linjuan Wu, Dingming Li, Yizhou Liu, Zixuan Wang +5 more
benchmark similarity 0.62
Abstract

Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation.

0.44

From Detection to Characterization: A Large-Scale Study of Ragebait on Japanese X

cs.SI cs.CL ·2026-09-02 ·Zhiyang Qi, Kazuhiro Ito, Jinghui Chen, Hibiki Nakamura, Zhangxuan Chen +3 more
published similarity 0.61
Abstract · Accepted at WI-IAT 2026

Ragebait refers to online content intentionally designed to provoke anger or outrage and thereby increase attention and engagement. However, reliable large-scale detection and systematic analysis of ragebait remain limited, hindering efforts to understand its prevalence, impact, and mitigation. This study aims to develop an effective ragebait detection framework and to clarify the characteristics of ragebait at scale, providing a basis for understanding and mitigating emotionally provocative content online. We constructed a labeled dataset with the assistance of a large language model (LLM) and trained several Japanese language models for ragebait detection. The resulting ensemble classifier was then applied to a large-scale dataset of Japanese-language posts on X. Our analysis shows that ragebait is more prevalent in politically and socially contentious topics, including politics, discrimination, public health, and interpersonal conflict. Ragebait posts also spread faster and receive more negative reactions than non-ragebait posts, particularly anger, fear, disgust, sadness, and surprise. These findings demonstrate the utility of the proposed detector and provide a large-scale characterization of ragebait in Japanese online discourse.

0.19

YRC-Bench: Benchmarking Culturally Aware Machine Translation for Historical Heritage via Tri-Level Semantic Bridging

cs.CL ·2026-11-07 ·Yan Huang, Aobo Li, Jiayi Wang, Huidong Zhu
benchmark similarity 0.61
Abstract

Translating historical cultural heritage poses profound challenges for machine translation systems due to the acute semantic asymmetry across eras and cultures, where terse classical syntax frequently obscures domain-specific cultural knowledge. While Large Language Models have demonstrated remarkable prowess in generic translation benchmarks, they struggle to negotiate Culturally-Specific Items and diachronic shifts in domain-rich historical texts, such as Yellow River cultural records. To bridge this critical gap, we introduce YRC-Bench, the first human-curated, multi-parallel benchmark dedicated to historical heritage translation across Classical Chinese, Modern Chinese, and nine target foreign languages. YRC-Bench establishes a tri-level parallel corpus incorporating modern explanatory glosses as explicit semantic bridges and operationalizes translation strategies through paired conserved versus substituted CSI variants categorized by cultural translatability. Systematic evaluations of nine leading LLMs across bidirectional translation directions reveal that leveraging modern Chinese glosses - especially human-curated ones - dramatically elevates both lexical fidelity and semantic adequacy. Furthermore, multidimensional human preference audits by native experts unveil distinct cross-cultural preferences between Sinographic-tradition and non-Sinographic language groups.

0.17