CS Paper Radar

Optimal Re-Ranking Depth

Siqing Huo, Andrew Parry, Debasis Ganguly, Charles L. A. Clarke

cs.IR ·submitted 2026-11-04 04:00 UTC · v1
top venue

Abstract

Second-stage neural rankers are commonly applied with a fixed re-ranking depth, assuming that retrieval effectiveness saturates as depth increases. Prior research has questioned the assumption that increasing the re-ranking depth yields linear performance gains, further suggesting that optimal re-ranking depth varies considerably from query to query. In the past, studying such phenomena was methodologically difficult given the scale of manual annotation required. With the advent of LLM-based relevance judgments we can now more easily undertake such studies, in this case to pinpoint the optimal re-ranking depth on a per-query basis. Using dense LLM-based relevance judgments over a typical re-ranking pipeline, we show that many queries exhibit a well-defined optimal re-ranking depth, beyond which effectiveness stagnates or degrades. We formulate re-ranking depth as a query-specific property and study whether it can be predicted a priori from first-stage retrieval characteristics. Through a large-scale analysis, we find that most standard query performance prediction (QPP) methods are ineffective for this task. In contrast, a predictor derived from LLM-assessed first-stage ranking quality, which we term IR-DCG@10, can reduce the average re-ranking depth by up to a factor of 3 while preserving overall effectiveness, depending on the specific first- and second-stage rankers used. Under oracle selection of optimal depths, we further show that retrieval effectiveness can improve by more than 7% while reducing the average re-ranking depth by a factor of 5 on the MSMARCO DEV collection. Given our promising preliminary findings, we would encourage the use of automatic judgments to facilitate research otherwise infeasible under manual annotation. Towards this point, we release relevance judgments, our codebase, and experimental artefacts to support reproducibility and further research.

Journal ref
In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '26), Melbourne, VIC, Australia, 2026, pp. 3799-3804
DOI
10.1145/3805712.3809953

Closest papers

Nearest neighbours by meaning across the whole index.

Seek: Self-Evaluative Exploration for Knowledge Retrieval

cs.IR ·2026-09-24 ·Amin Bigdeli, Radin Hamidi Rad, Negar Arabzadeh, Sajad Ebrahimi, Hai Son Le +2 more
top venuebenchmark similarity 0.73
Abstract · Accepted at CIKM 2026

LLM-based retrievers and rerankers have advanced passage ranking, yet both paradigms interact with the corpus in a single pass and commit to the resulting candidate set, leaving relevant documents permanently unrecoverable once missed. We introduce Seek, Self-Evaluative Exploration for Knowledge Retrieval, a training-free framework that addresses this limitation through iterative corpus interaction at test time. At each round, an LLM generates pseudo-passages conditioned on accumulated relevance feedback, a retriever surfaces fresh candidates, and a dedicated assessor assigns graded relevance judgments that guide subsequent rounds. On TREC Deep Learning, Seek matches trained rerankers in ranking quality while consistently improving Recall@100 over single-pass BM25. On the reasoning-intensive BRIGHT benchmark, Seek with Qwen2.5-7B achieves an 82% relative gain over BM25, surpassing all trained baselines, and Seek with GPT-4.1 reaches 37.4 average nDCG@10, exceeding the strongest baseline by 37%.

0.28

The Retriever Should Remember: Experience-Amortized Reranking for Long-Term Agent Memory

cs.AI ·2026-08-24 ·Qi Feng, Chris Ding, Jicong Fan
similarity 0.72
Abstract

Long-term language-model agents accumulate memories across interactions, but their retrievers typically do not accumulate retrieval experience. Semantic retrieval is efficient, but embedding similarity does not always reflect whether a memory contains evidence relevant to the current query. Large language model (LLM) rerankers provide stronger query-conditioned relevance scores, yet stateless reranking repeatedly scores a large candidate pool and discards these scores after each query. We introduce EARM, an experience-amortized reranking framework that treats previously acquired LLM relevance scores as reusable retrieval experience. EARM stores sparse query--memory relevance scores in an online matrix, learns their shared structure through causal matrix completion, and combines a small set of newly observed scores with estimated scores to rerank the remaining candidates. The scoring budget decreases as experience accumulates, changing LLM reranking from a repeated per-query expense into a retrieval capability learned over an agent's lifetime. Experiments on long-term conversational memory show that mixed observed-and-estimated reranking improves answer accuracy over semantic retrieval by up to 6.62% and remains effective when only 17.5% of candidates receive direct LLM relevance scores, thereby substantially reducing the inference overhead of LLM reranking. These results motivate a broader view of agent memory: a long-lived agent should remember not only past content, but also how that content has proved useful for retrieval.

0.40

Do Retrieval Ranking Models Capture Event Order? A Study of Timeline-Constrained Retrieval

cs.IR ·2026-11-03 ·Alon Eirew, Lidan Avisar, Kfir Bar, Ido Dagan
similarity 0.71
Abstract

Semantic retrieval models aim to match documents to a query's "meaning", yet it remains unclear whether they successfully encode temporal relations between events in their ranking mechanisms. We investigate this question by studying timeline-constrained document retrieval, where a query is an ordered sequence of natural-language event statements while a relevant document should match those events in a compatible temporal progression. We then test whether widely used dense retrievers and neural rerankers capture such constraints, and whether explicitly preserving temporal order improves ranking, using a controlled setting over a novel focused evaluation collection. Experiments show that state-of-the-art semantic retrievers and rerankers provide only limited improvements over lexical retrieval, suggesting that their semantic matching capabilities do not reliably capture temporal constraints. We then compare a timeline-aware method with both semantic matching and standard rerankers and find that it yields substantial improvements. This demonstrates that considering temporal order can provide a strong ranking signal in this setting, suggesting future research on improving temporal modeling in semantic retrievers.

0.09

Re$^3$Rank: Listwise Reranking with Reasoning-Conditioned Retrieval Heads

cs.CL ·2026-11-02 ·Ruiyi Yan, Zhuoyuan Mao, Yugo Murawaki, Yiwen Guo
similarity 0.70
Abstract

Attention-based rerankers score each candidate by the attention mass that retrieval heads route to it from the question, but the question, serving as a fixed query, carries limited information for locating relevant evidence. We propose Re$^3$Rank, a listwise reranker built on reasoning-conditioned retrieval heads (RRHeads). A generator first writes a reasoning trace over the question and the candidates, and a scorer then uses the trace tokens as the attention query of a few selected RRHeads, reading relevance as continuous scores. We present a straight-through estimator to propagate the ranking loss through the discrete trace, so that the generator and the scorer can be trained jointly for ranking. Averaged over 9 benchmarks, Re$^3$Rank outperforms 11 baseline rerankers: our 4B variant exceeds the strongest baseline by 3.11 NDCG@10 points on average, and even our 0.6B variant surpasses rerankers with up to 32B parameters.

0.15

Query Expansion Should Be Coordinated: Dense Expands, Sparse Anchors

cs.IR cs.CL ·2026-08-16 ·Chunran Zhang
code/data similarity 0.70
Abstract · 12 pages, 4 figures. Code and artifacts: https://github.com/ln-one/dense-expands-sparse-anchors

Retrieval-augmented generation (RAG) systems rely on retrieval modules to ground large language model (LLM) outputs. LLM-based query expansion enriches retrieval with document-like passages, but evaluations of hybrid retrieval often fuse fixed top-L prefixes of dense and sparse rankings. Because L controls cross-channel contributions and ranking access, it can alter measured expansion gains. We therefore evaluate complete-list effectiveness and record per-channel replay stopping depths required to certify the ordered top-K. This changes the design: because both rankings determine the fused result, their query constructions should be coordinated rather than designed independently. We present DESA (Dense Expansion and Sparse Anchoring), which shares generated references across channels but specializes their integration. Orthogonal residual expansion adds new semantic directions to the dense query, whereas score-product anchoring reorders the original sparse support without admitting expansion-only matches. The same references thus play complementary roles: Dense expands; Sparse anchors. Across seven BEIR datasets, DESA improves nDCG@10 and Recall@20 over the unexpanded query by 3.82% and 2.38%, while reducing dense and sparse replay stopping depths by 36.90% and 36.56%.

0.60

Support or Magnitude? Characterizing Neuron Relevance for Instruction Data Selection

cs.CL ·2026-10-12 ·Yifan Le
similarity 0.69
Abstract · 23 pages, 16 figures, 7 tables

Model-internal signals, such as neuron activations and gradients, have been used for instruction data selection, yet how neuron relevance should be characterized remains underexplored. In particular, should selection prioritize the magnitude of relevance or how broadly it is distributed across neurons? To investigate this question, we construct sample-level relevance profiles using AttnLRP-guided attribution and introduce two complementary criteria: Effective Relevance Support (ERS), which measures the effective support of positive relevance, and Positive Relevance Magnitude (PRM), which captures its average strength. Our analyses reveal that these characterizations produce distinct sample rankings and selection preferences despite sharing the same underlying relevance profiles. On Qwen2 models, PRM exhibits stronger associations with activation-based selection, whereas ERS produces substantially different rankings. These differences extend to task composition, within-task sample preferences, and semantic selection patterns, with varying degrees of agreement across backbones. Experiments across three language models and five downstream benchmarks demonstrate the practical value of relevance-guided selection, with ERS achieving the best overall average performance rank among the evaluated methods. Our findings highlight that how neuron relevance is distributed and characterized, rather than its magnitude alone, matters for instruction data selection.

0.21

The Emergence of Relevance Through Axiomatic Attention Patterns During LoRA Fine-Tuning

cs.CL cs.AIcs.IR ·2026-08-24 ·Matthew Perlman, Atharva Nijasure, James Allan
top venue similarity 0.69
Abstract · Accepted to EMNLP 2026 Findings. 17 Pages. 25 Figures. 5 Tables

LoRA fine-tuning is standard for adapting LLMs to reranking, but it remains unclear where in the network task-specific relevance behavior is learned and what attention-level changes accompany that learning. Through ablation and attention experiments, we identify where LoRA attention updates to RankLLaMA improve performance and whether those gains coincide with interpretable relevance-oriented attention patterns such as lexical matching, rarity sensitivity, and query-document interaction. We find that given LoRA fine-tuned MLPs throughout the network, restricting LoRA attention updates to a compact mid-network region is sufficient for recovering over half of the performance gained by applying LoRA to all attention layers, and that omitting attention fine-tuning in this region hurts performance more than elsewhere in the network. Additionally, we show that regions where applying LoRA affects performance the most overlap with regions where fine-tuning increased attention to axiomatic IR features. Rarity sensitivity, document-query interaction, and several compositional features are highly correlated with gains in ranking performance. Our results support an interpretable, correlational account of how relevance-oriented behavior emerges during LoRA fine-tuning and point toward improved strategies for adapting rerankers.

0.52

Beyond Recall Saturation: Scaling Hard-Negative Reranker Training for Industrial Legal Retrieval

cs.CL ·2026-10-17 ·Yunhan Li, Mingjie Xie, Zeyang Shi, Gengshen Wu, Min Yang
similarity 0.68
Abstract · 11 pages,6 tables, 4 figures

Industrial retrieval systems often improve quality by enlarging embedding candidate pools or adopting complex training pipelines. We show that, under a sufficiently large but practical candidate pool, embedding recall exhibits diminishing returns, motivating a closer examination of reranker quality. On Chinese legal retrieval benchmarks, embedding top-300 reaches a high-recall operating region, while relevant evidence may still be ranked below the output cutoff. We compare hard-negative strategies for reranker training, including embedding-mined, self-mined, and dynamic-negative variants, under aligned retrieval protocols that evaluate reranking after candidate retrieval. Among the evaluated staged configurations, an embedding-to-self-mined pipeline provides a favorable trade-off; additional dynamic stages do not improve all held-out metrics. The selected pipeline scales from a 0.6B to a 4B reranker. Relative to the native 4B reranker, the trained 4B Stage2 model improves reported metrics on the STARD and CSAID benchmarks and on an internal development set, and is preferred in 72.4\% of non-tied comparisons on 410 de-identified, reconstructed internal operational-query inputs in a latency-sensitive offline preference evaluation. These results support evaluating the retrieval and reranking stages separately when selecting an optimization target for industrial retrieval systems.

0.26