CS Paper Radar

Curating Merchant-Matching Training Data with Two Confidence-Gated Local LLM Judges

Donghao Huang, Jinling Pei, Zhaoxia Wang

cs.AI cs.LG ·submitted 2026-09-27 19:55 UTC · v2

Abstract

Merchant matching resolves a noisy payment descriptor to a retrieved merchant entity or returns no match. A key challenge in curating training labels is distinguishing teacher abstention from evidence that no acceptable entity exists: false no-match labels contaminate pseudo-labeled data, while conservative labeling reduces coverage. We investigate whether agreement between two local large language model judges improves pseudo-label reliability. A label is retained only when the judges agree, with separate ordered thresholds for selections and abstentions that guarantee disjoint positive and negative label sets. Retrospective replay on 2,000 expert-annotated queries shows that higher selection thresholds can improve positive-label purity, whereas higher abstention thresholds increase false no-match labels. At thresholds (0.86, 0.80), Muse Glimmer 30B and Gemma 4 31B jointly label 1,633 queries (81.7% coverage) at 96.88% purity; positive and negative purities are 99.47% and 93.38%. This exceeds either constituent model at the same thresholds by more than two percentage points, with lower coverage. A split-half check finds only 0.14 percentage points of threshold-selection optimism. A symmetric threshold of 0.86 adds 40 erroneous no-match labels, while 46 false abstentions persist even with no confidence threshold. Across five matched within-model comparisons, higher reasoning effort yields no clear F0.5 gain and increases median latency by 1.8-5.0 times. These results motivate separate thresholding and auditing for positive and negative pseudo-labels. The study establishes label purity, not student utility; fresh-data curation and student fine-tuning remain necessary to demonstrate downstream value.

Comment
7 pages, 2 figures, 5 tables, Accepted for publication in 2026 IEEE International Conference on Data Mining Workshops (ICDMW), SENTIRE 2026 Workshop

Closest papers

Nearest neighbours by meaning across the whole index.

Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery?

cs.CL cs.AI ·2026-10-03 ·Nishanth Nayakanti, Prasang Gupta, Ashutosh Bilthare, Kevin Paul
similarity 0.61
Abstract · 16 pages, 5 figures

In many review workflows the verdict is the only thing retained. The passages behind it are not marked, because that annotation costs far more than recording the decision. We measure how much of that evidence a small language model can recover when it is post-trained on the verdicts alone, with no human evidence labels at any stage. On ContractNLI the human evidence spans are held out until evaluation. Matching the recorded verdict and agreeing with those spans are not the same thing: across six systems the two scores are only weakly related and rank the systems differently, so accuracy is a poor guide when the citations have to be reviewable. Label-only training on the bare verdict reaches accuracy 0.896 and span F1 0.564. Rejection sampling, which keeps a generated trace only when its verdict matches the record and then picks one by an automatic source-grounding score, reaches 0.797 and 0.556, against 0.747 and 0.493 before training. Verbatim citation rises from 0.597 to 0.729 under label-only training and to 0.701 under rejection sampling. One seed on one corpus cannot say which method is better, but both improve the evidence without anyone annotating it.

0.26

CAVA: Complementary-Label-Aware Data Valuation

cs.LG cs.CV ·2026-10-19 ·Tan-Ha Mai, Hsuan-Tien Lin, Vu Nguyen
hot direction similarity 0.60
Abstract · 39 pages, 18 figures

Complementary-label learning (CLL) trains a classifier from labels that name a class an instance does not belong to. Such labels are cheap to collect from crowdworkers or vision-language models (VLMs), making them attractive for large-scale annotation. However, both sources can mistakenly assign the true class as the complementary label. Such annotation errors are particularly harmful in CLL because the model is explicitly trained to suppress the correct answer. Optimal transport (OT) data valuation identifies harmful samples by comparing training data with a clean validation set, but transferring it directly to CLL creates a semantic mismatch: label $c$ means not class $c$ in the training but class $c$ in the validation. We show that this makes the OT label groups noisy mixtures whose behavior depends on the noise rate. The transferred score is informative only at sufficiently high noise, becomes uninformative at the crossover rate $1/M$ for $M$ classes, and reverses below it, where practical CLL annotation typically operates. We propose CAVA, which introduces a complementary label cost and a label margin that isolates the contribution of each complementary label to the transport potential. The margin keeps a consistent direction across noise rates, detects diagonal noise with a parameter-free threshold of closed-form precision, and allows detected samples to be recycled as ordinary-label supervision. On synthetic and human-annotated benchmarks, CAVA separates diagonal noise almost perfectly where directly transferred OT valuation performs worse than random. With only a fraction of the clean validation data, it also improves every CLL objective we evaluate, showing that OT-based valuation becomes reliable once the semantics of complementary supervision are modeled explicitly.

0.34

When Disagreement Labels Do Not Identify F1 Rankings

cs.LG ·2026-11-01 ·Roberto Fernández-Barrios, Iker Pastor-López, Asier González-Santocildes, Pablo García Bringas
similarity 0.60
Abstract · 6 pages, 1 figure, 1 table. Submitted to ESANN 2027, submission ES2027-34. Code and reproducibility artifact: https://doi.org/10.5281/zenodo…

Agreement labels cancel from an accuracy comparison but can change an F1 ranking. We specialize partial identification to an explicit audit after labeling disagreements: two agreement counts give exact bounds, certifying a ranking or a choice within a specified F1 tolerance under every completion. Across 3,200 frozen model pairs, shared completions tighten bounds and a 0.01 tolerance reduces labeling costs. A zero-loss (weak-winner) control attributes most savings to positive tolerance, not ties. Guarantees concern the evaluated batch, not future data.

0.27

How Many Labels Does Model Choice Need? Certificates and Budgets for Selective Prediction

cs.LG stat.ML ·2026-09-16 ·Tetsuji Kuboyama
theory similarity 0.58
Abstract · 22 pages, 13 figures, 3 tables. Includes proofs and experimental details in the main text

Classifiers can make identical predictions yet require labels to compare their selective performance: confidence ranks weight the same errors differently. We quantify this requirement for the area under the generalized risk-coverage curve (AUGRC). A prelabel lower bound rules out insufficient budgets. With all labels known, a covering linear program bounds the minimum number of labels sufficient to fix the winner (the certificate size) within $K-1$ labels for $K$ candidates. For fixed $K$, independent uniform orders and identical predictions, the prelabel bound approaches one quarter of the pool. With iid Bernoulli errors independent of the orders, every exact acquisition policy reads almost all labels asymptotically, although a two-candidate certificate needs only half. Across 108 feature-panel comparisons on nine datasets, disagreement labels settle every accuracy choice but no AUGRC choice. A 20% budget is ruled out in 96 conditions; certificates need 56-57% on average. On ten conditions with pretrained image classifiers, confidence-score choice reads 68-91% of 10,000 labels for exact selection and 50-67% with AUGRC tolerance $5\times10^{-4}$. An exact stopping test works with any acquisition order. Together, these results link confidence ranks to label budgets and certified model comparison.

0.45

When Keywords Drop but Classifiers Hold: Soft Refusals under KV Cache Compression

cs.LG cs.CL ·2026-09-16 ·Kang Chen, Xiuze Zhou, Hong Chen, Yuanguo Lin
similarity 0.58
Abstract · 8 pages, 5 figures

KV cache compression is widely used for long context LLM inference under memory constraints, while deployed systems typically score refusals after generation with keyword filters or learned classifiers. Such monitors are intended to indicate whether a model declined a harmful request under the serving regime actually used. However, it remains unclear whether matched compression that preserves task accuracy also preserves agreement between lightweight lexical monitors and stronger refusal classifiers. We study this with a paired protocol on n=200 harmful prompts with a long filler context: each prompt is answered once under full retention and once under matched eviction after a shared prefill, and the same replies are scored by keyword heuristics, the HarmBench Llama-2-13B classifier, an auxiliary LLM judge, and humans on disagreements. On Qwen2.5-3B, keyword refusal falls from 98.0% to 80.5% (McNemar p~1e-8) while classifier refusal stays near ceiling (99.0%-99.5%) and MMLU accuracy is unchanged (50.0%); human labels predominantly follow the classifier, consistent with soft refusals. The gap is not universal and weakens under short fillers and paired SnapKV, so safety auditing under compression should rely on several judges matched to the serving context rather than on keyword rates alone.

0.41

PersianAnonymizer: Evaluating LLM-Labeled Training for Efficient NER-based Anonymization in Persian

cs.CL ·2026-09-01 ·Mohammad Hossein Shalchian, Mostafa Amiri, Amir Mahdi Sadeghzadeh
published similarity 0.58
Abstract · 10 pages, 3 figures, 6 tables. Published at LREC 2026

We target practical anonymization of Persian customer chats by training a compact NER model from LLM-labeled supervision and selecting the best labeler for deployment. We compare three instruction-tuned LLMs: DeepSeek-V3-0324, GPT-OSS-120B, and Qwen3-235B-A22B-Instruct-2507, to produce span annotations under a shared JSON protocol, yielding four corpora (OSS_ZeroShot, Qwen_ZeroShot, Qwen_FewShot, DeepSeek_FewShot). A MatinaRoberta-based token-classifier is trained per corpus and evaluated with token-level Precision/Recall/F1 (overall and per-class). We also report Label Coverage Recall (LCR), the proportion of gold non-O tokens predicted as non-O, and quantify cross-labeler behavior via a token-level Venn on test annotations. Finally, we contrast test-set annotation latency of the LLMs on H200 nodes with the trained NER's test-time labeling on a single RTX 3090. Results show that supervision from OSS_ZeroShot yields the strongest macro-F1 and LCR, while the resulting NER labels an entire 40K-message test set in approximately 2 minutes on one consumer GPU. This establishes a practical path to high-quality, low-cost anonymization for Persian industrial data.

0.48

Screening LLM Judges Without Ground Truth: A Preliminary Evaluation of an AI Dental Hygiene Coach

cs.AI ·2026-10-17 ·William DeShong, Barath Kannan, Auberon Palma, Brenda Bradshaw, James F. Leathrum, +1 more
similarity 0.57
Abstract · 29 pages, 2 figures, 19 tables

Evaluating a fine-tuned coaching model with expert graders is slow and expensive, and an LLM judge cannot be trusted without ground truth. We present a staged pipeline that screens candidate LLM judges and student models before expert review, using explicit decision rules at each stage: judge output audit, similarity with hosted judges, bias checks, and decision robustness. We demonstrate it on an AI dental hygiene coach. The case study uses synthetic, clinically unvalidated training and reference data to demonstrate the operation of the pipeline, rather than to establish the clinical performance of the resulting models. Two fine-tuned small language models (Granite-4.1-8B and Qwen3-8B), each trained at three data sizes, were scored on seven metrics by five judges (three local, two hosted). Two local judges, Qwen3.8-27B and Gemma-4-31B-it, were not distinguishable from the agreement between the hosted judges on six of seven metrics by rank correlation, and Qwen3.8-27B was the most similar to the hosted judges overall. No judge met our criterion for favoring its own model family or training-data source. The student comparison did not separate the two models on any metric under the pipeline's rule, so the choice of Granite-4.1-8B, which led in 29 of 35 judge-by-metric cells, is a lean that depends partly on truncated Qwen3-8B outputs. The pipeline also surfaced problems that raw scores hid, including the truncated outputs and hallucination flags from one hosted judge that no other judge corroborated. No dental hygiene expert has graded these responses, so the findings are preliminary and the pipeline's final stage, anchoring to expert judgement, is proposed but not performed.

0.26

What Transfers from Training-Only Priors? Separating Knowledge, Ordinal Targets, and Optimization

cs.LG ·2026-10-08 ·Qiye Lu, Jiang Ji, Liang Zhang
similarity 0.57
Abstract · Substantially expanded revision with matched distillation and optimizer controls, public-credit and bankruptcy experiments, TabICLv2 query/c…

Training-only teacher access lets a compact student deploy with local features and exported weights. This established distillation interface does not identify what transfers or when it helps. We separate label, query, capacity, and source-access budgets on three public credit archives and a previously unexamined bankruptcy archive. Shared tree parameters, shuffles, and optimizer interventions expose limitations of historical Ranking Prior Alignment; no tested single repair consistently rescues it. Heldout source queries and matched warmup and mappings retain a rank-listwise advantage over mapped staged KD in 24 credit configuration means and all six teacher--capacity means of a frozen new-archive test, with explicit query failures and stronger source-weight delivery under a different interface. A pretrained TabICLv2 follow-up at 100 labels initially improves matched supervision in only one of eight means; six of nine source-predicted policy gains reverse on target. A registered seen-archive intervention adds 500 or 2,000 unlabeled queries: all eight matched AUC means become positive under selected and fixed alpha. However, outer effects are mixed and no conditional paired-draw interval for these supervised contrasts lies wholly above zero. Increasing teacher context at 500 student labels does not beat matched supervision in any of eight ordinary-context means, despite stronger direct teacher performance. Rank KD retains substantial probability errors. Independent reloads verify zero guarded external inference accesses. Results support query/context boundary diagnostics, without universal superiority, population significance, a prospective transfer rule, or an LLM-specific advantage.

0.21