CS Paper Radar

X-Embedding: Two-Stage Training for Multilingual Patent Prior-Art Retrieval

Wentao Wu, Xu Wang, Linqing Chen, Fenhong Zhu, Hanmeng Zhong, Yuanxiang Huangfu

cs.IR ·submitted 2026-10-21 10:07 UTC · v1

Abstract

Multilingual patent prior-art retrieval requires identifying X-category references that are particularly relevant to the novelty or inventive step of a claimed invention, beyond topical similarity across languages and document views. We present X-Embedding, a two-stage framework that first learns patent-domain representations and then optimizes X-category retrieval relevance. The first stage aligns complementary sections within patents and corresponding sections across different-language patent-family members, learning from both patent structure and multilingual expressions. The second stage learns X-category relevance using examination-report X citations as positives and negatives drawn from retrieval candidates and A-category citations. Matryoshka representation learning (MRL) enables 64- and 1024-dimensional embeddings within one model. Evaluation on a common Chinese, English, and Japanese collection with four query views yields 89.09% overall macro Hit@100 at 64 dimensions, exceeding Qwen3-Embedding-0.6B and RoBERTa by 21.25 and 3.68 percentage points, respectively. Under the same second-stage training and evaluation conditions, two-stage training outperforms direct supervised fine-tuning by 3.19 points, with gains of 5.34 and 4.72 points for English and Japanese and higher scores in nine of twelve language-view combinations. Overall macro Hit@10 and Hit@20 also improve by 3.71 and 3.79 points, respectively. These results support sequential patent-domain and task adaptation for improving aggregate X-category retrieval on this multilingual collection.

Comment
12 pages, 3 figures, 4 tables

Closest papers

Nearest neighbours by meaning across the whole index.

ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents

cs.CL ·2026-09-19 ·Mahdi Astaraki, Mohammad Khodadad, Reza Namazi, Mohammad Arshi Saloot, Amir Reza Behzad Moghadam +2 more
top venuebenchmarkdatasetcode/data similarity 0.74
Abstract · Accepted to the EMNLP 2026 Industry Track. 22 pages including references and appendices; 7 pages of main text

Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a different language than the query. However, existing benchmarks do not adequately capture domain-specific cross-lingual retrieval or the retrieval-depth and recoverability failures that aggregate recall hides. In this work, we benchmark CLIR in the chemical domain, with a focus on patent data. We construct a multilingual dataset from Google Patents and the European Patent Office (EPO) data, spanning five languages (covering major Eastern and Western languages) and reflecting the diversity and complexity of real-world industrial documentation. Using this dataset, we systematically evaluate eight state-of-the-art embedding models for cross-lingual retrieval. Our results show a substantial performance gap between monolingual and cross-lingual settings: for the best-performing model, Recall@10 drops from 0.72 to 0.53 in cross-lingual setting. Retrieval depth also degrades significantly, with relevant documents ranked lower across languages in cross-lingual scenarios. Furthermore, some multilingual embedding models that perform strongly in monolingual settings exhibit sharp declines when queries and documents are in different languages, providing practical insights for model selection in cross-lingual use cases. These findings highlight critical limitations of current approaches and emphasize the need for more robust cross-lingual retrieval methods in domain-specific settings. Our benchmark provides actionable insights for model selection and establishes a controlled diagnostic evaluation framework for CLIR over industrial technical text. Data and code are publicly available at https://github.com/MohammadKhodadad/Multi-Lingual-QAC.

0.49

Large Language Model Consensus and Corpus-Adaptive Topic Modeling for Classifying and Mapping Patent Innovation

cs.AI cs.IR ·2026-11-06 ·Sercan Ozcan, Enes Ozgozler, C. Okan Sakar
similarity 0.68
Abstract

Patent analytics increasingly relies on transformer representations and large language models (LLMs), yet linking conceptual classification, topic formation and interpretation requires explicit decisions about model roles and the evidence retained at each stage. In this design-oriented study, we propose and demonstrate a patent analytics pipeline that links these tasks and keeps each decision open to audit. Three LLMs separately classify each patent by novelty (radical or incremental) and innovation type (product, process, service). Radical and innovation-type labels each require two supporting votes, with Incremental used as the fallback novelty label. Topics are discovered by clustering PatentSBERTa embeddings, with parameter ranges scaled to corpus size. A patent-specific stopword list is applied only when topics are represented with class-based term frequency--inverse document frequency (c-TF-IDF), and a second embedding model supports inter-topic similarity, hierarchical refinement and visualization. An LLM then labels each topic from its ranked keywords and representative patents. We demonstrate the pipeline on 30,964 public-sector-related patents retrieved from Lens.org. Across five category corpora ranging from 591 to 29,215 patents, the pipeline produced 74 category-specific topic instances with a macro-average document coverage of 98.1% and topic diversity of 0.900--1.000, and fewer than 1% of classification outputs returned UNKNOWN. Because every model vote and the evidence behind each topic label are retained, analysts can audit and revise the outputs. The demonstration establishes the feasibility and traceability of the pipeline; testing its labels against expert coding and benchmarking it against alternative pipelines are identified as the next steps. The pipeline offers a reusable basis for LLM-assisted patent analysis in technology intelligence.

0.19

Benchmarking Embedding Geometries and Clustering Algorithms for Unsupervised Patent Claim Classification

cs.AI cs.CL ·2026-10-09 ·Sujith Subramanian, P N Bhargav Teja, Susen Sarkar, Rajasekhara Reddy Duvvuru Muni
similarity 0.67
Abstract

Prior-art retrieval systems are structurally organized around classification codes, making cross-code discovery difficult. This paper asks if an unsupervised representation trained on patent text can recover relevant connections that code-restricted search cannot, and if hyperbolic space helps. We trained a hyperbolic and a matched Euclidean Variational Autoencoder (VAE) on 45,000 US patents (96 runs, 6 seeds). The hyperbolic model retrieves genuinely cross-category prior art 52% more often than the Euclidean model (18.76% vs. 12.34% of top-10 results, 3,000 queries). This represents a practical gain for the motivating search problem. The hyperbolic model agrees less with official classification (NMI 0.2392 vs. 0.3206), which we explain geometrically: the two measures capture different goals, and only one aligns with prior-art search needs. In a study of 10 technology domains (50,000 more patents), we find that a domain's classification breadth--the number of distinct CPC subgroups spanned--predicts approach performance (Pearson r=-0.642, p=0.045, N=10), and that patents per subgroup is an even stronger predictor (r=0.715, p=0.020). This provides a checkable signal for practitioners to determine where the method is most valuable. We report all findings with statistical transparency and release a complete, 91-test automated implementation.

0.23

From Rules to Neural Graphs: Scalable Structured Prediction for Patent Prior Art Search

cs.IR cs.CL ·2026-10-01 ·Nikolai Zenovkin, Sebastian Björkqvist
published similarity 0.64
Abstract · Accepted for publication at the ECML PKDD 2026 conference (Applied Data Science track)

Patent search requires processing documents routinely exceeding tens of thousands of tokens. Most neural retrieval approaches operate on truncated inputs, limiting their effectiveness. Graph-based retrieval addresses this by representing each patent as a structured invention graph, but constructing these graphs relies on brittle rule-based parsers. We present the neural parser, which adapts biaffine attention from dependency parsing to predict invention graphs directly from patent text. Our local biaffine attention restricts pairwise scoring to a sliding window, reducing complexity from $O(n^2)$ to $O(n \cdot w)$. Since local and global scoring share the same weights, the model trains on short sequences and deploys on documents exceeding 40,000 tokens without retraining. Distilled from 1 million rule-parsed documents, it surpasses its teacher at 3$\times$ lower inference cost: neural graphs improve citation recall by 0.5% on short queries and 1.1% on full documents in a downstream Graph Transformer retrieval system.

0.30

Teaching Language Models to Narrow Boolean Patent Searches from Search Feedback

cs.IR cs.AIcs.CL ·2026-11-17 ·Seiya Kawano, Koji Marusaki, Hirofumi Nonaka, Koichiro Yoshino
published similarity 0.64
Abstract · 13 pages, 3 figures, 3 tables. Accepted at NLPIR 2026

Patent prior-art searchers issue Boolean queries line by line, broadening to find relevant documents and narrowing until candidates can be screened. We distill 38,397 Japan Patent Office search reports into executable expert strategies, re-execute them, and evaluate generated strategies by citation recall and the smallest set containing each citation. Prompted models write valid, high-recall strategies but leave citations in sets too large to screen. We adapt a generator offline from scored rollouts in two forms: strategy-level learning from a screening-aware strategy score, and line-level learning from citation-preserving narrowing credit, which upweights the query lines keeping a found citation in a smaller set. On shared rollouts, line-level credit outperforms strategy-level scores on screening measures; iterated, it cuts the median citation-containing set from 768 to under 50 and places more citations in sets under 100 documents than experts, at lower recall. Strategy-level preference learning reaches near-complete recall; the two forms are complementary.

0.44

Nothing Left Behind: Embedding Retrieval and Language Model Screening for High Recall Evidence Search

cs.CL ·2026-10-27 ·Giuliano Martinelli, Abelardo Carlos Martinez Lorenzo, Satvik Garg, Linxi Wang, Riccardo Orlando +2 more
similarity 0.61
Abstract · 8 pages, 2 figures

AI systems that ground their answers from retrieved documents usually need only a few relevant passages, but applications such as systematic reviews and legal discovery need nearly all of them, because their conclusions rest on the full body of evidence. These applications have traditionally searched with curated taxonomies, which can only find what their vocabulary describes. We compare taxonomy-based retrieval with an alternative neural retrieval based on sentence embeddings and followed by language model screening in ImpactAI, an evidence synthesis application built on 7,354 randomized trials. Measured against the studies included in 29 systematic reviews, the share of studies found rises from 45% to 85%, and the screening model agrees with human annotators as well as annotators agree with each other. This gain comes at a cost in speed: at the median, retrieving and screening evidence for a query takes 77 seconds, against 6 seconds for taxonomy retrieval. The results suggest finding evidence with embeddings, organizing it with taxonomy labels, and setting how many candidates to screen according to the time available for each query.

0.32

Self-Knowledge Retrieval Augmented Generation Framework for Patent Matching

cs.IR cs.CL ·2026-08-11 ·Jian Zhang, Songlin Lei, Zhuohao Yang, Bangli Liu, Ziwei Wang +4 more
published similarity 0.61
Abstract · Accepted by IEEE CSCWD 2026

Patent retrieval and matching based on large language models (LLMs) play a vital role in intellectual property protection. However, due to the complex structure of patent documents, dense technical terminology, and multi-modal information, traditional methods struggle to accurately identify subtle differences between patents. Existing LLM-based patent matching approaches typically rely on domain-specific pretrained or instruction tuning, which often entail high manual labeling costs and catastrophic forgetting. While retrieval-augmented generation (RAG) methods introduce external knowledge they fail to fully leverage LLM's capability to automatically parse patents and mine deep semantic relationships. To address these limitations, this paper proposes a self-knowledge RAG framework that guides LLMs to autonomously extract key technical entities and construct hierarchical ontological structures from patent matching queries, thereby enabling query expansion and precise retrieval. The method integrates the FAISS retrieval with a generative matching mechanism, leveraging self-knowledge to enhance the model's understanding of patent innovations and significantly improve retrieval and matching accuracy. Experimental results demonstrate the outstanding performance of the proposed method on real-world patent datasets, validating its effectiveness and application potential.

0.17

Retrieval Needs Multivectors: An Exponential Separation

cs.IR cs.DBcs.LG ·2026-08-21 ·Mihir Agarwal, Viraj Agrawal, Sabyasachi Basu, Ankit Garg, Kirankumar Shiragur
benchmark similarity 0.61
Abstract

Recent works have highlighted the expressive limitations of embedding based retrieval models through both theoretical analyses and challenging benchmarks such as LIMIT. While multi-vector embeddings consistently outperform single-vector embeddings, the precise representational gap between them remains poorly understood. In this work, following Jayaram's work, we provide the first explicit family of query and document sets, together with their relevance matrices, for which single-vector embeddings that rank all relevant documents above irrelevant ones require exponential size, whereas polynomial-size multi-vector embeddings suffice. Our result establishes an exponential separation between the expressive power of single-vector and multi-vector embeddings for the task of ranking of documents as opposed to approximating numerical scores as in the work of Jayaram. Motivated by our theoretical construction, we introduce ANDOR, a new retrieval benchmark that naturally instantiates these hard examples. We show that state-of-the-art single-vector embedding models perform poorly on ANDOR in the zero-shot setting and exhibit only marginal improvements after fine-tuning, highlighting the inherent difficulty of the benchmark compared to prior work. In contrast, multi-vector models consistently outperform their single-vector counterparts and improve substantially with fine-tuning, closely aligning with our theoretical predictions.

0.41