CS Paper Radar

Matryoshka Hash Representations for Model-Aware Compact Semantic Retrieval

Peichun Hua, Yunming Xiao

cs.IR cs.AIcs.LG ·submitted 2026-09-07 09:38 UTC · v2

Abstract

Retrieval-augmented generation (RAG) depends on dense retrieval: each document is stored as a learned vector, and queries are answered by nearest-neighbor search. Keeping one full-precision vector per document dominates index cost at corpus scale, so systems replace each vector with a short code of a few bytes (quantization). Standard quantizers such as product quantization (PQ) pick the code that best reconstructs the original vector, but reconstruction is only a proxy for retrieval: at a few bytes per document, a model-aware code trained on the retriever's ranking signal uses the budget better. A single code is even more useful if its short prefixes are each directly searchable, letting a deployment set its efficiency-quality operating point without re-encoding the corpus. But training all prefixes under one objective makes the early bits a compromise across budgets: short codes improve while the full-width code degrades. Binary codes sharpen the conflict, since a gradient step either flips a stored bit or leaves it unchanged. We introduce Matryoshka Hash Representations (MHR), a two-stage procedure that separates full-width training from prefix organization. MHR first learns a longer binary code, then freezes the model and trains zero-initialized residual code adaptors for directly searchable prefixes. Documents are stored at one bit per coordinate, queries keep continuous logits as in PQ, and search runs on FAISS FastScan. Trained on MS MARCO and zero-shot transferred to seven BEIR datasets, one MHR code reaches 0.502 NDCG@10 at 16 bytes and 0.556 at 32 bytes, against 0.404 and 0.524 for the best compact baselines; the gains hold across training seeds, on all 13 public BEIR datasets, and against off-the-shelf binary Matryoshka embeddings with stronger backbones. The same code also helps shortlist candidates for reranking and prune low-storage graph indexes such as LEANN.

Comment
12 pages, 2 figures, 12 tables

Closest papers

Nearest neighbours by meaning across the whole index.

HHR: Hierarchical Hash Retrieval for Efficient LLM Generation

cs.AI ·2026-10-01 ·Lianjun Liu, Tiantian Zheng, You Huang, Weiqi Yan, Mingte Qiu +3 more
code/data similarity 0.71
Abstract

Efficient long-context inference is essential for large language models (LLMs), yet it poses a severe computational bottleneck. Hash-based retrieval offers an efficient alternative by encoding queries and keys into binary codes and using Hamming distance for key selection. However, this leads to a critical mismatch between Hamming distance and attention relevance. Query-Key logits depend jointly on directional similarity and feature magnitudes, whereas hash binarization discards magnitude information, causing both false-positive retrieval of low-logit keys and false-negative omission of high-logit keys. To address these failures, we propose Hierarchical Hash Retrieval (HHR), a coarse-to-fine framework that progressively improves retrieval accuracy through Geometry-Aware Key Routing (GKR) and Learned Hash Projection (LHP). GKR learns a head-wise orthogonal transformation to redistribute feature magnitudes and derive more discriminative page-level logit bounds, enabling effective pruning of low-logit keys while preserving important candidates. LHP then learns a head-wise projection space that aligns Hamming distance with the true Query-Key relevance ranking for fine-grained retrieval. By combining GKR and LHP, HHR suppresses false positives and recovers false negatives, substantially improving the fidelity of hash-based sparse attention. Extensive experiments across diverse LLMs and benchmarks demonstrate that HHR achieves superior performance over existing methods. For example, on LongBench, HHR improves the average score by 1.10 points and, at a context length of 128K, achieves up to a 3.30x decoding speedup and a 2.83x end-to-end speedup for Llama-3.1-8B-Instruct. The code is publicly available at https://github.com/lianjunl13-sudo/HHR.

0.33

MRVQ: One Resident Index for Dimension- and Rate-Elastic Vector Search

cs.AI cs.IR ·2026-10-02 ·Sean Culatana, Shang-En Huang, Kang Li
similarity 0.68
Abstract

Dense-retrieval services must switch among embedding-prefix dimensions and index bit rates as latency, quality, and memory budgets change. Tuning a quantizer separately for each rate gives the best quality, but the retrieval tier then holds several code streams and quantizer states at once. We introduce Matryoshka Residual Vector Quantization (MRVQ), a post-hoc residual quantizer for frozen embeddings. Its maximum-rate code can be truncated two ways: dropping residual stages lowers the rate, and dropping embedding coordinates lowers the dimension. One resident artifact therefore serves every (dimension, rate) pair we evaluate. Across FiQA and NFCorpus, four embedding families, and {4, 8, 16}-byte codes, MRVQ is the lowest-RAM design we evaluate. It uses 17.8-22.0x less memory than three separately trained QINCo2 indices, and 1.89-2.02x less than a lean shared-model steelman. The saving is not free: per-rate QINCo2 is 0.026-0.107 nDCG@10 better on FiQA. But MRVQ beats PQ, OPQ, and AdANNS-OPQ at matched code size. We also evaluate a low-build-cost PCA-scalar design that attains quality comparable to RaBitQ and its extension while fitting 420x faster at the median. Finally, we report two negative results: QINCo2 collapses when trained at high rates, and a ranking-bound hypothesis misses its pre-specified acceptance criteria. MRVQ is therefore a low-memory operating point for elastic retrieval, not a universal quality winner.

0.11

CARE: Contrastive-Anchored Residual Encoding with Listwise Reconstruction for Retrieval-Oriented Semantic IDs

cs.IR ·2026-11-16 ·Yuan Xu, Dawei Chen, Xiaoguang Lin, Xin Chen, Yulong Ou +1 more
similarity 0.68
Abstract · 18 pages, 8 figures, 5 tables

Generative retrieval replaces nearest-neighbor search with the generation of a document's semantic ID (SID), so the SID tokenizer bounds what the retriever can recall. Common SID constructions (residual k-means, RQ-VAE, finite scalar quantization, capacity balancing) optimize criteria defined on document embeddings alone and do not control whether a query can still find its relevant documents after quantization. We propose two retrieval-oriented SID tokenizers. CARE-SID (Contrastive-Anchored Residual Encoding for Semantic IDs) quantizes in a retrieval-aligned space: a zero-initialized residual alignment network is trained with a query-document contrastive loss before quantization, an anchor loss and a reconstruction target fixed to the original embedding prevent drift and collapse, and reference-vector rating residual quantization assigns the codes. CARE-LR (CARE with Listwise Reconstruction) then distills, for each training query, the similarity distribution that the original embeddings induce over a candidate list into the SID reconstructions, through a function-preserving layer-resolved head. Both keep a document-only encoding interface. We judge SIDs without training a generator, by row-budget recall in a bucketing view (the partition) and a reconstruction view (the SID with its reconstruction). In offline evaluation on industrial video search data, both tokenizers achieve higher reconstruction-view recall than residual k-means, and CARE-LR improves further on CARE-SID, mainly through the listwise distillation. A comparison of two document embeddings shows that contrastive alignment helps most when the embedding has not been trained on search data, and that the embedding affects SID recall far more than the quantizer. CARE-SID is deployed in the generative retrieval stage of an industrial video search system.

0.17

WeSearch: A Unified Model-Centric Retrieval Framework for WeChat Search

cs.IR cs.CL ·2026-10-31 ·Kepu Zhang, Yuzhe Cai, Yue Shu, Dawei Chen, Jinhai Zhu +20 more
similarity 0.67
Abstract

We present WeSearch, a unified, model-centric retrieval framework for large-scale, continuously evolving content search. Prevailing methods assign documents fixed hierarchical Semantic IDs, implicitly assuming each document follows a single coherent semantic path-an assumption that often breaks down for multi-topic documents. WeSearch instead retrieves query-relevant clusters and then ranks atomic documents within the resulting query-specific candidate space, decoupling candidate pruning from fine-grained relevance modeling. Combined with relevance-aware training, compact document representations, and incremental index updates, this design enables weekly-or daily when needed-model refreshes. Deployed as the primary semantic retrieval path in WeChat Search alongside a lexical syntax tree, WeSearch shows consistent one-week A/B gains on corpora from hundreds of millions to billions of documents, and a one-month full-traffic A/B test confirms stable improvements of +0.68% CTR and +2.27% per-user article clicks, together with a 41% reduction in retrieval resource cost and a 19 ms reduction in graphic-text retrieval latency. WeSearch is now fully deployed in WeChat Search, serving 100% of production traffic.

0.19

Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers

cs.IR cs.CL ·2026-09-11 ·Anubhav Shrestha, Safal Shrestha, Minwu Kim, Torsten Suel, Keith Ross
similarity 0.66
Abstract

Recent work has shown that fine-tuning decoder-only large language models (LLMs) for retrieval yields strong first-stage retrievers, with effectiveness improving as backbones grow in size. However, every query and document must pass through the full model, so encoding cost increases with model size. Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative models, yet remain underexplored as retrievers. We systematically study MoE backbones for retrieval by training MoE and dense LLMs from several families using the same procedure, evaluating them across diverse datasets, and measuring query encoding time under the same serving configuration. We show that MoE retrievers outperform dense retrievers with comparable active parameter counts by up to 3.0 nDCG@10 points on BEIR. One of our strongest MoE retrievers matches an 8B dense retriever with 59% fewer active parameters and 18% lower query encoding time. We further show that the number of experts used for query encoding can be reduced without retraining or re-indexing, retaining more than 99% of retrieval effectiveness while reducing query encoding time by up to 26%. Recent rerankers provide only modest additional gains over strong MoE first stages, which often match or exceed the reranked configurations we evaluate. Together, these results show that MoE LLMs can be strong and efficient first-stage retrievers.

0.39

Leveraging Binary Hashing for Scalable Index Assignment in Semantic Communications

eess.SP cs.IR ·2026-11-16 ·Leonardo Roque, Kuangyun Zhu, Louis-Adrien Dufrène, Quentin Lampin, Grégoire Lefebvre +1 more
benchmarkhot direction similarity 0.66
Abstract · 6 pages

Transmitting semantically structured embedding vectors over digital channels is costly in bit rate, while conven- tional vector quantization approaches face limitations in scalabil- ity, encoding complexity, and index assignment. This paper inves- tigates binary hashing techniques from the information retrieval literature as semantic indexing strategies for general-purpose networks. We study random Locality-Sensitive Hashing (LSH) along with Principal Component Analysis (PCA)-, Canonical Correlation Analysis (CCA)-, and ITQ (Iterative Quantization)- based variants in a semantic communication pipeline over a Binary Symmetric Channel. Embeddings are compressed into binary codes, reconstructed through Hamming retrieval from a shared table, and evaluated on downstream classification. Ex- periments on the Massive Text Embedding Benchmark (MTEB) and the Banking77 dataset show that data-dependent hashing improves upon random LSH, while ITQ refinement increases robustness to channel noise. Among the evaluated methods, CCA- ITQ achieves the best classification accuracy across the tested channel conditions. These results indicate that binary hashing is an efficient and scalable alternative to quantization-based embedding compression for semantic communications.

0.40

DEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval

cs.IR cs.AI ·2026-08-18 ·Jingyuan Wang, Richong Zhang, Zhijie Nie, Mingxin Li, Yanzhao Zhang
benchmarkcode/data similarity 0.66
Abstract

Large language models (LLMs) can both expand underspecified queries and encode text as dense representations, suggesting a unified model for query expansion and retrieval. Existing systems usually rely on prompted expansions, independently trained modules, or staged optimization, leaving generated expansions only indirectly aligned with the retrieval loss that judges them. We train a single decoder-only LLM end to end, where the same model generates the expansion and encodes both the expanded query and candidate documents. This unified setting creates a moving-target problem: retrieval supervision should improve query-side expansion, but the same update also shifts the document embeddings that serve as retrieval targets. We introduce Document Embedding Preservation Tuning (DEPT), which keeps tuned document embeddings close to cached initial embeddings while allowing retrieval gradients to pass through straight-through decoding into the generator. DEPT converts joint query--document movement into query-side adaptation against approximately stable, whitened document embeddings that support index reuse and online hard-negative mining. Experiments with Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct on five datasets in BEIR benchmark show that DEPT improves average retrieval quality over training-free, independently trained, and staged unified baselines, while ablations isolate the effects of preservation, whitening, end-to-end expansion training, and online negatives. Code is available at https://github.com/ILSparkle/DEPT.

0.34

MiSTiQ: MicroScaling Trellis Quantization for Extreme KV Cache Compression

cs.LG ·2026-11-16 ·Ofir Gordon, Lior Dikstein, Paul Balanca, Idan Achituve, Douglas Orr +1 more
similarity 0.65
Abstract · 26 Pages, 7 Figures, 7 Tables

Modern large language models (LLMs) increasingly support longer contexts and serve more concurrent requests, increasing both sequence lengths and batch sizes. Both trends increase KV-cache size and memory traffic during inference. KV-cache quantization reduces these costs, with recent approaches using codebook-based vector quantization to achieve extremely low bit widths. However, hardware constraints limit codebook size, and most methods reconstruct the cache in higher-precision floating-point formats for attention, limiting the computational benefits of quantization. To address these limitations, we introduce MiSTiQ, a drop-in replacement for codebook-based vector quantizers that combines trellis-coded storage with microscaling (MX) formats for low-precision computation. We combine trellis quantization with beam search to encode the KV cache over large reconstruction spaces without large codebooks. To support MX computation, we derive MX-Hash functions that generate reconstruction values directly in the target MX element format. These functions use the scale-conditioned marginal distribution of block-normalized Gaussian inputs. On Llama3 and Qwen3, MiSTiQ achieves lower WikiText perplexity and higher GSM8K accuracy than the evaluated baselines across several block-scale formats at approximately 1.25 effective bits per element. With NVFP4, it also achieves the highest LongBench average on both models.

0.20