CS Paper Radar

Better Retrieval, Limited Clustering Gains: A Controlled Study of Multilingual Company Entity Resolution

Yijiashun Qi, Yuxuan Li, Hanzhe Guo

cs.AI ·submitted 2026-10-04 22:19 UTC · v2
published

Abstract

Company name retrieval and final entity clustering can respond differently to encoder adaptation. We study this dependency through a controlled pipeline: train multilingual E5 encoders with matched positive examples and random or hard negatives, select checkpoints on development data, retrieve within equal country-specific edge budgets, and apply unchanged pair classifiers and guarded Union-Find clus- tering. Evaluation uses a subsequently collected GLEIF snapshot; a stratified, assisted human audit and a post-hoc temperature sweep examine label and parameter sensitivity. On 3,633 names with source-silver labels, adapted retrieval raises direct pair recall from 53.74% to 76.98%, whereas cluster recall moves from 32.54% to 33.33%. Most newly retrieved positives fail the existing classifier threshold. The study contributes a controlled comparison and pair-level diagnosis of this loss between retrieval and confirma-tion. It shows why evaluating a replacement retriever requires measuring the resulting entity partition, while leaving joint matcher retraining and independent gold-label evaluation for future work.

Comment
accepted by ICEKIM 2026

Closest papers

Nearest neighbours by meaning across the whole index.

From Location Phrases to Geographic Entities: Task-Adapted Retrieval for People Search

cs.AI cs.LG ·2026-08-29 ·Yanbo Li, Chujie Zheng, Jiahao Xu, Chetan Bhole, Lingyu Zhang +6 more
benchmark similarity 0.62
Abstract · 11 pages, 1 figure, 9 tables

People search must map free-form location phrases to geographic entities used as structured retrieval filters. Lexical standardizers handle canonical names well but are brittle to aliases, misspellings, metropolitan expressions, and same-name ambiguity. We formulate this task as graded, set-valued entity retrieval over a fixed ontology. We identify three coupled design requirements: distinguishing identity-preserving variation from knowledge-dependent aliases, controlling false negatives among valid same-name entities, and separating stable transformations from mutable entity knowledge. We realize them in a prompt-asymmetric bi-encoder with calibrated alias support, bounded ambiguity-aware negatives, and editable entity documents that support localized updates without retraining. Across a fixed production-derived development benchmark and a public GeoNames transfer task, task adaptation improves substantially over frozen encoders and standard token baselines. Controlled development ablations show that specialized supervision contributes beyond standard task fine-tuning and encoder scaling. On GeoNames, the adapted model improves known-target Recall@1 throughout zero-to-moderate character overlap, while character n-grams retain a small aggregate Target Recall@5 advantage. In a blinded human comparison on a stratified production challenge set, our model raises relevant P@1 from 28.0% to 46.0% (p=0.012). Fixed-query endpoint estimates improve on non-canonical queries and remain close to control on frequent queries; a randomized live experiment detects no engagement regression. These results support task-adapted geographic entity retrieval as a practical replacement for the incumbent taxonomy-based standardizer, with the largest relevance gains on non-canonical queries.

0.29

Billiger.de Products: A Bilingual Entity Matching Benchmark

cs.CL ·2026-09-29 ·Aaron Steiner, Ksenia Elagin, Ralph Peeters, Johannes Knopp, Christian Bizer
benchmarkcode/dataunusual similarity 0.62
Abstract · 23 pages. Describes benchmark version 1.1 (repository tag v1.1.0). Data, code, and reference results: https://github.com/wbsg-uni-mannheim/b…

Existing product matching benchmarks primarily contain English-language product data and are often dominated by a single product category, such as electronics. This paper introduces Billiger.de Products, a bilingual German and English entity matching benchmark covering thirteen consumer product categories, including difficult-to-handle categories such as clothing and furniture. The benchmark data originates from the German price comparison platform billiger.de. Following the design of WDC Products, the benchmark offers multiple variants that differ in the fraction of corner cases, the size of the development set, and the fraction of entities unseen during training. An aligned English translation of every offer keeps all pairs, splits, and labels fixed, while cross-language test sets combine German and English records within individual pairs. We validate the benchmark using six supervised matchers and zero-shot GPT-5.2 on both language versions and the cross-language test sets. The validation shows the difficulty of the benchmark. The comparison of the results on the English version of the benchmark to the results on the German version shows that most matchers score on average higher on the English version. The difference is largest for RoBERTa and HierGAT, while the zero-shot LLM runs are largely insensitive to the language. Comparing the F1 scores achieved by PLM-based matchers on the English version of Billiger.de Products with their performance on existing English-language benchmarks, such as WDC Products and Abt-Buy, shows that Billiger.de Products is more difficult than these benchmarks.

0.50

ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents

cs.CL ·2026-09-19 ·Mahdi Astaraki, Mohammad Khodadad, Reza Namazi, Mohammad Arshi Saloot, Amir Reza Behzad Moghadam +2 more
top venuebenchmarkdatasetcode/data similarity 0.62
Abstract · Accepted to the EMNLP 2026 Industry Track. 22 pages including references and appendices; 7 pages of main text

Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a different language than the query. However, existing benchmarks do not adequately capture domain-specific cross-lingual retrieval or the retrieval-depth and recoverability failures that aggregate recall hides. In this work, we benchmark CLIR in the chemical domain, with a focus on patent data. We construct a multilingual dataset from Google Patents and the European Patent Office (EPO) data, spanning five languages (covering major Eastern and Western languages) and reflecting the diversity and complexity of real-world industrial documentation. Using this dataset, we systematically evaluate eight state-of-the-art embedding models for cross-lingual retrieval. Our results show a substantial performance gap between monolingual and cross-lingual settings: for the best-performing model, Recall@10 drops from 0.72 to 0.53 in cross-lingual setting. Retrieval depth also degrades significantly, with relevant documents ranked lower across languages in cross-lingual scenarios. Furthermore, some multilingual embedding models that perform strongly in monolingual settings exhibit sharp declines when queries and documents are in different languages, providing practical insights for model selection in cross-lingual use cases. These findings highlight critical limitations of current approaches and emphasize the need for more robust cross-lingual retrieval methods in domain-specific settings. Our benchmark provides actionable insights for model selection and establishes a controlled diagnostic evaluation framework for CLIR over industrial technical text. Data and code are publicly available at https://github.com/MohammadKhodadad/Multi-Lingual-QAC.

0.49

Reliability of LLM Judges for Evaluating Entity Alignment

cs.AI ·2026-10-07 ·Vaibhava Lakshmi Ravideshik, Mayank Kejriwal
top venuebenchmarkcode/data similarity 0.61
Abstract

Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset--judge--prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system's decision label is visible. Judge discrimination collapses under label-exposing prompts (J-ROC-AUC 0.12--0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.98--1.00 for NeuSymEA) and significant recovery on biomedical pairs (0.93--0.95). Counterfactual label flips show that the visible label causally shifts scores by more than 2 points in 53--99% of pairs (FSR), with sensitivity growing with model capability, a pattern we call the frontier model paradox: stronger judges are more label-sensitive, not less, yet also better label-free predictors. The direction of the shifts is mixed, so we report this as an observed pattern. A blinded two-annotator human evaluation (102 pairs, Cohen's $κ$ = 0.902) provides a gold-label reference. We release the first biomedical EA benchmark (MeSH--SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment.

0.45

Do Multilingual Encoders Produce Language-Consistent Semantic IDs?

cs.IR cs.CL ·2026-10-01 ·Abhinav Bohra, Anuj Bohra
similarity 0.60
Abstract · 7 pages, 8 tables. Accepted as a short paper at WiNLP 2026, co-located with EMNLP 2026

Semantic IDs (SIDs) compress item embeddings into discrete code sequences used in generative retrieval. We ask whether a multilingual encoder is sufficient for different-language renderings of the same product to receive language-consistent SIDs. Using Amazon ESCI listings rendered in English, Spanish, and Japanese, we test whether translations remain close to their English source, whether residual quantization is unusually sensitive to translation-induced movement, and whether multilingual or language-balanced quantizer fitting improves SID agreement. Multilingual E5 places translations measurably apart: under an English-heavy fit, a Japanese translation preserves the first SID code of its English counterpart in only 7.7% of cases, compared with 89.0% for an English rewording. Distance-matched product-directed controls produce nearly the same full-SID mismatch as translation, providing no evidence that the quantizer selectively amplifies language directions. Balancing the fitting mixture makes codebook use more uniform but further reduces cross-lingual prefix agreement: Spanish first-code consistency falls from 28.3% to 6.6%, while an English-only fit preserves it for 67.6% of Spanish translations. These results show that multilingual exposure and balanced codebook use alone do not guarantee language-consistent SIDs.

0.34

From Confusion to Clarity: Confusion-Aware Retrieval and Knowledge Injection for Text Classification

cs.CL cs.AI ·2026-09-01 ·Manish Gupta, Chaitanya Giri, Jayasimha Talur
top venue similarity 0.60
Abstract · EMNLP 2026 (Industry Track)

Large language models (LLMs) struggle to classify text into taxonomies with many semantically similar labels, as the distinctions are domain-specific and not captured by pre-training. To handle large label spaces, a common approach retrieves top-$K$ candidate labels by embedding similarity and prompt the LLM to choose among them. However, top-$K$ retrieval reduces the number of candidates but does not help the model tell similar ones apart. When two similar labels both appear as candidates, the model lacks the signal to choose correctly between them. We propose a framework that (1) identifies which label pairs the model struggles to distinguish, (2) expands the candidate set to include confusable labels, and (3) generates targeted rules to differentiate between similar candidates. The framework requires no fine-tuning, and the generated rules transfer to smaller, cheaper models. On three benchmarks (WOS, Flipkart, LEDGAR), our approach improves Macro F1 by up to 10.0pp over retrieval baselines, with smaller models (2B--20B) gaining up to 11.5pp via cross-model transfer.

0.71

Domain-Specific Text Embedding Models for Entity Resolution

cs.IR cs.AIcs.LG ·2026-08-17 ·Khajesh Sapram, Srivardhani Raju, Kishore Konda
similarity 0.60
Abstract

General-purpose text embedding models are designed to capture semantic similarity but are not optimised for distinguishing entity records that represent the same real-world business or person. This limitation affects applications such as entity resolution and duplicate record retrieval, where small textual differences may either preserve or change identity. This paper investigates whether domain-specific triplet fine-tuning can adapt pretrained embedding models for identity-sensitive retrieval. A synthetic dataset of business and person records was created with identity-preserving variations and challenging non-matching examples. Two widely used embedding models were evaluated before and after fine-tuning using a margin-based similarity evaluation. The results show substantial improvements in separating true matches from highly similar non-matches, demonstrating that domain-specific triplet training can effectively reshape general-purpose embedding spaces for entity retrieval. These findings suggest that targeted fine-tuning provides a practical approach for improving embedding models in data quality management and information retrieval applications.

0.37

Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes

cs.CL cs.AIcs.LG ·2026-11-05 ·Vladimir Beskorovainyi
code/data similarity 0.60
Abstract · 15 pages, 7 tables. v3: corrected abbreviation, slang and seed-spread statements after a per-option bias correction; third unseen dataset wi…

Typed-decision models score options that the caller supplies at request time, so their accuracy depends on label wording that the model's developer does not control. We compare two releases of a 307M-parameter multilingual cross-encoder, Ru-Jev, trained on thirteen Russian-language tasks; the later release adds an augmentation that presents labels in several surface forms. Holding questions fixed and varying only how eighteen routing labels are worded, we evaluate ten label vocabularies at five and eighteen candidates, checked for overlap with the training data. On held-out vocabularies made of ordinary or borrowed words (jargon, office slang, department names), the augmented release has higher point accuracy in all fourteen cells, by up to 0.24. Several of these gains rest partly on one label that the earlier release almost never chose; fitting one bias per option removes part of them, while the jargon gain largely survives. On abbreviations and opaque codes the augmented release gives no usable gain: its loss on 17 abbreviations unrelated to training (-0.060) is one label and disappears after the same correction. The comparison is between two trained releases, not a controlled ablation, and the earlier recipe varies across seeds by up to 0.098 on the strictest slice. On two datasets neither release saw, accuracy moves by -8.4 and +5.6 points, narrowed to -3.4 and +1.4 by the per-option biases; on a third, scientific rubrics, the augmented release is ahead of both runs of the earlier recipe after the correction. Rankings of ten training checkpoints disagree across the three unseen datasets, so none of them should choose the checkpoint. We also report four evaluation hazards from comparing the two releases.

0.35