CS Paper Radar

Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement

Rafael Macalaba, Aivin V. Solatorio, Patrick Michael Brock, Olivier Dupriez

cs.CL cs.AIcs.IRecon.EM ·submitted 2026-09-10 18:34 UTC · v1
benchmark

Abstract

Development and humanitarian organizations produce and support surveys, administrative registries, and other data resources to inform research, policy, and operations, yet systematically identifying where these datasets are referenced remains difficult. Such references are dispersed across research papers, project documents, humanitarian reports, and other unstructured text, limiting both the ability to trace data use and to identify potential gaps in data availability or dissemination. We present a weakly supervised framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents without first constructing a large manually labeled training corpus. A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents, which a frontier large language model (LLM) reviews in context, validating or rejecting candidates and correcting their extraction boundaries. The resulting annotations are supplemented with targeted synthetic and contrastive examples and used to fine-tune the lightweight model for large-scale extraction. We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. Across the full benchmark, the model achieves 74.1\% precision and 70.5\% recall at the mention level; among passages containing dataset references, precision reaches 89.5\%. At the passage level, the model achieves 88.2\% accuracy and 88.6\% specificity in distinguishing passages with dataset references from those without them. These results demonstrate a practical approach for constructing domain-specific supervision when labeled data are limited, and provide a technical foundation for larger-scale analysis of data use and potential gaps in the displacement data landscape.

Comment
22 pages, 1 figure

Closest papers

Nearest neighbours by meaning across the whole index.

Lightweight Machine Learning for Classifying Disaster-Related Social Media Posts

cs.NI cs.LG ·2026-10-17 ·Chengjun Jiang, Milena Radenkovic
similarity 0.67
Abstract

Classifying disaster-related social-media posts into humanitarian information labels is challenging because individual messages are typically short and context-limited, humanitarian categories can overlap semantically, and the CrisisBench label distribution is strongly imbalanced. This study investigates whether lightweight lexical machine-learning methods can provide effective and computationally efficient classification under these conditions. Using the CrisisBench v1.0 English humanitarian task, comprising 87,455 records across predefined Train, Development and Test splits, we conduct a controlled comparison of TF-IDF representations and lightweight classifiers, with Macro-F1 as the primary model-selection metric. Beyond aggregate predictive performance, the study jointly examines per-label behaviour, directional error patterns, relative decision margins, computational requirements, and the effect of domain-specific training through a tropical-cyclone specialist model. The selected balanced LinearSVC with unigram-bigram TF-IDF and C=0.5 achieved an Accuracy of 0.8282, a Macro-F1 of 0.6578 and a Weighted-F1 of 0.8315 on the held-out Test split. Lower decision margins were substantially associated with prediction errors, while the tropical-cyclone specialist achieved modestly higher in-domain aggregate performance with substantially lower feature and fitting requirements. The study contributes a controlled, label-sensitive and resource-aware evaluation of lightweight humanitarian text classification that characterises not only aggregate performance, but also where these models fail and how their behaviour changes under domain-specific training.

0.23

ALTO: African and Levantine Tech-Facilitated Gender-Based Violence Corpus

cs.CL cs.AIcs.CYcs.LG ·2026-11-04 ·Tianwei Zhang, Ivy Njoki Nyagwencha, Zahraa Dirani, Shalini Joshi, Scott A. Hale
similarity 0.67
Abstract · 8 pages main body, 10 pages appendix, 1 figure

Tech-facilitated gender based violence (TFGBV) is a pervasive online harm affecting women and gender minority groups through its many direct and covert forms. Current models and annotated datasets for gendered harm detection are dominated by data from, and models trained in, Global North languages. In collaboration with civil society organizations (CSOs) from Jordan, Francophone West Africa, and Kenya, we developed a community-led approach to crowdsource culturally and linguistically relevant instances of TFGBV in Levantine Arabic, African French, and Swahili. We annotated the crowdsourced data using active learning to develop a small but performant dataset (N=3022) called ALTO to improve TFGBV detection in the three languages. Through benchmarking and fine-tuning experiments, we show that existing harmful content detection models show poor TFGBV recall within ALTO and improve with fine-tuning. We found that models often miss covert harms across all languages, while error rates are markedly higher in Swahili. We report the experimental results and release the annotation taxonomy, dataset, and trained models.

0.14

CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation

cs.CL cs.AI ·2026-10-16 ·Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante, Chandan Manjunath, Laura Onasch-Vera +3 more
top venuecode/data similarity 0.65
Abstract · In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

We present CoDA, an open-source human-LM collaborative codebook development assistant that helps domain experts develop annotation guidelines for annotating new variables in unstructured text with language models. Existing language model annotation systems primarily focus on downstream classification performance, while assuming the availability of high-quality annotation guidelines, or codebooks. In practice, developing these codebooks is time-consuming and challenging, particularly for concepts that require expert knowledge. CoDA addresses this gap by collaborating with domain experts to iteratively develop codebooks through targeted review of incorrect language model predictions and incorporating expert feedback into the evolving codebook. We demonstrate CoDA on two real-world case studies: extracting health indicators from homelessness case management notes and classifying sentiment of wildlife trafficking posts on social media. Across both domains, CoDA produces codebooks that achieve downstream classification performance comparable to expert codebooks while reducing codebook development time from several weeks to two hours and requiring 5X fewer manually annotated examples. Applying CoDA to generate annotations at scale, we identified 2,410 case notes that contained health discussions or referrals and 10,196 social media posts expressing negative sentiment that may warrant investigation for potential wildlife trafficking. By accelerating codebook development, CoDA helps practitioners use language models to extract domain-specific insights from unstructured text at scale.

0.45

Towards Full-Text Scholarly Argument Mining using Entity-Anchored Claim Extraction

cs.CL cs.IR ·2026-10-29 ·Bhuvanesh Verma, Andy Lüking, Mounika Marreddy, Leon Lukas Hammerla, Alexander Mehler
similarity 0.65
Abstract

Argument mining in full-text scholarly articles remains an important yet challenging task. This difficulty stems not only from the need to model long-distance dependencies but also from the substantial effort required to annotate sentence-level argumentative components throughout lengthy documents. To address these challenges, we introduce a novel approach that leverages scientific entity mentions (e.g., tasks, datasets, and models) to identify and structure claims in scholarly literature. We validate our approach through a case study involving 50 full-text research papers on sentiment analysis. We first establish a reliable human gold standard (Krippendorff's $α= 0.80$), identifying 113 claim-bearing sentences and extracting 199 structured claim records from 300 candidate sentences. To address the annotation bottleneck, we employ a majority-voting ensemble of large language models (LLMs) to generate a silver-standard dataset comprising 1,487 claims. Encoders fine-tuned exclusively on the silver-standard dataset outperform those trained on a small human-annotated gold-standard dataset and demonstrate strong zero-shot transfer to NLP papers outside the training corpus.

0.22

HUMAID-NER: A Disaster Tweet Dataset for Joint Named Entity Recognition and Event Classification via Uncertainty-Weighted Multitask Learning

cs.CL cs.LG ·2026-09-15 ·Aijaz Ali, Nazish Basir, Sarfaraz Nawaz, Danish Nazir Arain, Haris Ali
publishedbenchmarkcode/data similarity 0.64
Abstract · 8 pages, 8 figures, 4 tables. Published in The Asian Bulletin of Big Data Management, Vol. 6, No. 1, pp. 138-152, 2026

Rapid extraction of structured information from social media is important for humanitarian response, yet existing disaster tweet resources mainly provide document-level category labels without span-level entity annotations. We introduce HUMAID-NER, the first named entity recognition dataset built on the HumAID benchmark, containing 60,000 English disaster tweets annotated in BIO format across ten operationally motivated entity types and yielding approximately 175,000 labelled entity spans. Annotations are generated through a reproducible three-stage hybrid pipeline combining a spaCy transformer model, disaster-domain EntityRuler patterns, and structured regular expressions with priority-based overlap resolution. We also propose a joint multitask learning framework that performs disaster-specific named entity recognition and humanitarian event classification using a shared RoBERTa-large encoder. To reduce task conflict during joint training, the model uses homoscedastic uncertainty weighting with learnable task parameters and a two-stage training schedule that freezes the lower 18 of 24 encoder layers in the second stage. On the HUMAID-NER validation set, the proposed system achieves NER span micro-F1 of 0.841 and classification macro-F1 of 0.761 simultaneously. A real-time web dashboard demonstrates end-to-end deployment. The dataset, models, and pipeline code are released to support reproducibility and future crisis informatics research.

0.48

Beyond the pale: Assessing prevalence and contents of extremist speech in LLM training data

cs.CL ·2026-08-14 ·Dmitry Nikolaev, Ashley A. Mattheis
theory similarity 0.64
Abstract · Accepted to the CPSS workshop @ KONVENS 2026

Despite a strong interest on the part of the research community in the topic of trustworthy and safe AI, the composition of the text corpora that large language models (LLMs) encounter in pre- and post-training has not yet drawn much attention. In this work, we address the question of whether LLMs are exposed to unfiltered, uncontextualised extremist speech. Using several definitions of extremist speech, stemming from official documents and research literature, and an extraction pipeline combining automated text processing with expert verification, we provide a lower bound on the prevalence of extremist documents in Dolma, an open training corpus underpinning the OLMo series of models. We show that Dolma is likely to include hundreds of thousands of documents containing extremist content and hate speech of several types, including direct calls for violence, and discuss the implications of this for data curation and model pre-training.

0.13

IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law

cs.AI cs.IR ·2026-09-18 ·Genis Skura, Roland Bouffanais, Didier Wernli
benchmarkdataset similarity 0.64
Abstract

International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its texts remain without token-level named entity recognition (NER) resources. We introduce IntLawNER, a NER dataset and benchmark for codified sources of international law, covering 2,987 gold-annotated sentences and 8,094 entity spans from International Court of Justice (ICJ) decisions, UN Security Council resolutions, and European Court of Human Rights (ECtHR) judgments, annotated with seven institution-specific entity types. We construct IntLawNER with a cost-effective hybrid algorithmic-agentic pipeline that reduces 468k source sentences to a compact annotation set through candidate retrieval, LLM-based vetting, and human review, with 89.6% of gold spans accepted unchanged from the silver layer. However, the silver-to-gold analysis reveals that human-machine aggregate agreement metrics can be misleading in domain-specific NER: Cohen's kappa=0.964 on boundary-matched spans masks a macro-F1 of 0.753 when missing entities, boundary errors, and label corrections are included. The benchmark shows that zero-shot span-based GLiNER collapses on entity types dependent on institutional function rather than surface form (0.243 micro-F1), while fine-tuned transformers struggle on rare labels. Carefully selected few-shot examples that demonstrate label contrasts improve every LLM over zero-shot prompting, with Claude Opus 4.6 reaching the best score of 0.873 micro-F1. We release IntLawNER as a benchmark and reusable resource for extracting references in international legal texts.

0.08

EpiLens: Lightweight RAG for Epidemiological Knowledge Acquisition

cs.CL cs.DLphysics.soc-ph ·2026-10-19 ·Vincenzo Perri, Ciro Cattuto, Daniela Paolotti
code/data similarity 0.63
Abstract

To prepare for future outbreaks, researchers need to know what studies have been done and what data they use. Mapping a large literature by hand is slow, while large language models alone can misstate or conflate sources. We present EpiLens, a lightweight, open-source retrieval-augmented generation package that applies a fixed retrieval template and output schema to every paper in a corpus and links outputs to source passages. We build a corpus of 2,297 papers from the sources of a multi-pathogen parameter study and evaluate EpiLens's paper-classification workflow against 214 manually labelled papers. Run locally with an open-weight model, so that licensed full texts never leave our hardware, the classifier is comparable to supervised baselines trained on the manual labels, and ahead of them on geography, despite the lack of task-specific training data. The study's early extraction sheets cite published sources for 327 estimates. We link each estimate to a cited document or to corpus studies included by a review. Then, we use the classifier's labels to compare the papers associated with different parameters and pathogens. EpiLens labels more papers linked to reproduction numbers than fatality risks as model-based (53% versus 15%, compared with 62% versus 24% in the manual labels). The gap is consistent with the frequent use of transmission models to estimate reproduction numbers and largely persists within pathogens.

0.29