CS Paper Radar

The Optimal Ranking Tradeoffs Between Two Ranking Scores in Two-Class Classification

Sébastien Piérard, Adrien Deliège, Marc Van Droogenbroeck

cs.PF cs.CVcs.LG ·submitted 2026-10-16 11:00 UTC · v1

Abstract

The accuracy, true negative rate (negative class recall), true positive rate (positive class recall), negative predictive value (negative class precision), positive predictive value (positive class precision), class-specific intersections over unions, class-specific F-scores, Cohen's kappa, and balanced accuracy are scores that one intuitively wants to maximize when developing machine learning methods. However, they lead to different rankings of such methods. So, how can we establish a ranking that is an optimal tradeoff between any pair of these scores? It turns out that all these scores are special cases of ranking scores: a continuum of scores parameterized by the relative importance values assigned to the true negative, false positive, false negative, and true positive outcomes. Since all these scores are known to induce meaningful rankings, we seek the optimal tradeoffs among them. We consider the tradeoff between two scores to be optimal when it minimizes the Fréchet variance regarding the Kendall distance (also known as the bubble-sort distance) between rankings. For any given set of classifiers to rank, the optimal tradeoff lies at the center of the shortest (geodesic) path on the manifold of all rankings induced by the ranking scores. However, in the parameter space of the ranking scores, it is rarely achieved by averaging the importance values corresponding to the initial scores. In this paper, we present an exact analytical solution for determining a ranking score that is an optimal tradeoff for any given pair of initial scores, in the common case where the class priors are the same for all compared performances. We also experimentally investigate what happens when performance follows some distributions with fixed priors.

Closest papers

Nearest neighbours by meaning across the whole index.

Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

cs.CL cs.IRcs.LG ·2026-08-27 ·Markus Frohmann, Mahdiyar Ali Akbar Alavi, Elizabeth Lingg, Navid Rekabsaz
code/datahot direction similarity 0.60
Abstract · 9 pages main text, 45 pages total

In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one prompt, each still receiving its own score. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or which chosen/rejected pair enters preference training. Because the candidates share that prompt, reordering them changes their scores. The same query over the same candidates should still yield the same decision. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66-0.84 when reordered. No prompt-time change we test resolves that dependence: the only one that improves ranking quality does not measurably improve decision stability. We introduce order-consistency SFT (OC-SFT), which attenuates it in the weights by penalizing disagreement between a candidate's scores across orderings. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks. It is also more stable on 12 base models than order-averaged distillation, which trains on labels averaged across permutations. One OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison of such scorers should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://github.com/thomsonreuters/presentation-dependence.

0.68

When Greedy Is Sorting: Exact and Certified Ideal Rankings for Novelty Metrics

cs.IR ·2026-11-07 ·Kehan Liu, Zhaoyun Chen, Xianquan Yan
similarity 0.57
Abstract

The ideal ranking that normalizes novelty metrics is NP-hard to compute in general, so widely used $α$-nDCG evaluators approximate its denominator with a greedy rule that rescans every remaining document at each rank. We identify a regime in which this ideal ranking is exact and cheap for both $α$-DCG and intent-aware expected reciprocal rank (ERR-IA). When each document is positively judged for at most one intent, each metric reduces to the classical allocation of a fixed budget across chains of decreasing marginal gains. A metric-specific virtual-score transformation followed by a multiway merge, which we call ChainMerge, returns the globally optimal top-$K$ ranking in $O(G+K\log G)$ time from $G$ preordered intent groups, or in $O(n+K\log G)$ time from $n$ globally preordered candidates, instead of the $Θ(nK)$ of the greedy rule. When intent judgments overlap, exact $α$-DCG optimization is NP-hard under the standard logarithmic discount and $α=0.5$, even when every document has three intents and every intent occurs in two documents. For arbitrary overlap, a polynomial-size LP over nested ranking prefixes upper-bounds the ideal score and certifies the optimality gap of any feasible ranking. In the official TREC Web 2009-2012 judgments, 64.8% of positive documents serve a single intent, yet only 20 of 198 complete queries do so throughout. Beyond this exact regime, the LP certifies 189 of the 198 greedy denominators as globally optimal, and the resulting score intervals preserve the winner in all eight matched comparison groups. On 65,238 MIND-small candidates, ChainMerge returns identical optimal objectives 17-1,011 times faster than the greedy rule. For $α$-DCG, these results trace the hardness of the ideal ranking to intent overlap.

0.35

Rank-Deviation Quality: A Distance-Aware Metric for Multi-Answer Retrieval and Ranking Evaluation

cs.IR ·2026-08-26 ·Xiaokun Zhou, Alessandro Moschitti, Danielle Class
similarity 0.57
Abstract

We introduce Rank-Deviation Quality (RDQ), an evaluation metric for retrieval and ranking systems that adapts to queries with varying numbers of reference items, from a single correct answer to many valid results. RDQ scores a candidate ranking against an ordered reference list (ORL): each retrieved reference item contributes its output-position weight multiplied by a rank-deviation penalty, and items outside the ORL receive zero credit. Application-specific parameters control tolerance to misordering. Larger values emphasize retrieving valid reference items, whereas smaller values place more weight on matching their reference order. The output-position weights can reflect visibility in the application's interface, such as a vertical list or a carousel. Unlike metrics that require absolute relevance grades, RDQ operates on ordinal rankings, which annotators can produce through pairwise or listwise judgments. Unlike rank-correlation measures such as Kendall's tau, RDQ accounts for both which items are returned and how they are ordered. On a 5,000-query point-of-interest (POI) dataset with 12 systems, RDQ has the highest median empirical power@100 among the 13 evaluated metric configurations. It reaches mean tau >= 0.8 agreement with its own full-query ordering at 200 queries; RBP(0.9), the strongest tested non-RDQ configuration, reaches the same threshold at 250. On TREC Deep Learning benchmarks, where NDCG uses native graded labels and RDQ uses ordinal tiers derived from them, RDQ reaches comparable median power at n=25, while NDCG is higher at n=100.

0.54

Fair Bubble Sort: Provably Optimal Fair Ranking with Continuous Sensitive Attributes

cs.LG cs.CYcs.DS ·2026-11-17 ·Antonio Ferrara, Fabio Vitale, Atsushi Miyauchi, Francesco Bonchi
top venuetheorycode/data similarity 0.57
Abstract · 35 pages, 9 figures, 4 tables. Accepted at NeurIPS 2026. Code: https://github.com/Ambress92/FairBubbleSort

Most fair-ranking methods assume categorical protected attributes, requiring continuous attributes such as age or income to be discretized into groups. This introduces arbitrary thresholds, loses fine-grained information, and ignores the natural ordering of the attribute. In this paper, we study the problem of fair ranking with continuous protected attributes without relying on thresholding. We introduce Continuous Pairwise Parity (CPP), a fairness notion based on Kendall correlation between ranking positions and the protected attribute, and seek rankings that satisfy a prescribed CPP constraint while minimizing their Kemeny distance from an initial score-based ranking. To solve this problem, we propose Fair Bubble Sort (FBS), an efficient post-processing algorithm based on targeted adjacent swaps. We prove that FBS is exactly optimal for the unweighted Kemeny distance and provide approximation guarantees for the score-weighted setting. FBS is output-sensitive, with running time determined by the number of swaps required to meet the fairness constraint. Experiments on synthetic and real-world datasets show that FBS is scalable and consistently outperforms discretization-based baselines, achieving a superior fairness-accuracy Pareto front.

0.54

Characterizing Predictive Multiplicity in Ranking and Its Fairness Implications

cs.LG ·2026-11-11 ·Ainhize Barrainkua, Mikel Malagón
similarity 0.57
Abstract · 32 pages, 6 figures

Prediction problems often admit competing models that achieve comparable performance yet produce conflicting outputs, a phenomenon known as predictive multiplicity. While this challenge has been extensively studied in classification tasks, its implications remain largely unexplored in ranking frameworks, which are critical for high-stakes societal decisions such as the allocation of welfare resources and social assistance. This paper investigates the phenomenon of predictive multiplicity in score-based ranking problems. We introduce formal definitions to capture multiplicity in ranking settings, supported by foundational theoretical guarantees. Furthermore, we examine the dual relationship between predictive multiplicity and algorithmic fairness in ranking problems: we analyze how model multiplicity can act as a potential source of unfairness, while demonstrating how exploring this decision space provides a principled avenue for discovering less discriminatory rankings. Finally, we complement our theoretical contributions with insightful empirical evaluations on both synthetic and real-world datasets, validating our analytical findings and illustrating their practical implications. Taken together, this work highlights the real-world relevance of predictive multiplicity in ranking and sheds light on the potential of leveraging this effect to achieve fairer outcomes.

0.20

Beyond Ranking: Are Decision Models Calibrated Relevance Scorers?

cs.IR ·2026-11-08 ·Shengyao Zhuang, Xueguang Ma, Matthias Petri, Jimmy Lin
similarity 0.55
Abstract

Decision models such as Jev are a new kind of relevance scorer. Their typed score primitive takes a question, an ordered set of textual relevance levels and a state, and returns a probability for each level directly, rather than token likelihoods or generated labels. Jev's vendor promises that these probabilities are calibrated. This motivates revisiting 'Beyond Yes and No', which improved zero-shot pointwise LLM reranking using likelihood-derived scores over fine-grained relevance labels but left calibration against human assessors as future work. In this paper, we first reproduce its open-model FLAN-UL2 setting on eight BEIR collections, confirming its ranking trends, and then run Jev and two open-weight decision models with the same protocol. Jev and the 27B pplx-decider rerank ahead of FLAN-UL2 and three cross-encoders on average. We then compare Jev's native probabilities with the human assessors' labels and find them overconfident: of the DL21 pairs it scores above 0.9, only 66% are relevant. Since this is mostly an error of scale, a two-parameter calibration fitted on 2,257 DL19 pairs removes much of it and transfers to DL21. After calibration, Jev has a lower DL21 Brier score than FLAN-UL2 and the three cross-encoders, mainly because it separates relevant from non-relevant pairs better. Finally, with calibrated probabilities, choosing each query's cut-off by expected F1 needs no parameter of its own, beats a fixed top-10 cut and matches a cut-off threshold tuned on DL19.

0.26

Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory

cs.IR ·2026-09-28 ·Fabian David Schmidt, Donato Crisostomi, Carlos Lassance, Nils Reimers
benchmarkcode/data similarity 0.55
Abstract · 51 pages. Code and data: https://github.com/cohere-ai/rcp-ndcg

Rerankers decide which documents users and LLMs see, yet their standard metric, nDCG, relies on human relevance labels that are costly, sparse, noisy, and discretely graded. As rerankers approach each other in quality, nDCG on these labels therefore increasingly fails to separate them. LLM judges could supply dense labels. Relative judgments within one query tell even close candidates apart, yet their scores share no scale across queries. Absolute grades share one scale but are too coarse to distinguish documents of similar relevance. We propose Rubric-Calibrated Preferences (RCP), which combine both kinds of judgment. A listwise Bradley-Terry tournament orders each query's documents, and a rubric of yes/no criteria of increasing stringency provides an absolute standard. Item Response Theory (IRT), which scores test-takers based on their answers to common questions, then uses the shared criteria to put all queries' tournament scores on one scale. RCP's retrieval metric, RCP-nDCG, replaces nDCG's discrete labels with the resulting calibrated relevance probabilities. Against blind grades from 46 external annotators, calibration raises the correlation between a query's mean score and its mean human grade from 0.538 to 0.795. The probabilities rank a useful document above a non-useful one with probability 0.910 (AUC, chance 0.5), versus 0.651 for the benchmark labels. When the annotators' grades prefer one of two rerankers and exactly one metric agrees, that metric is RCP-nDCG in 72.4% of 185 comparisons (chance about 53%). On TREC-DL, RCP-nDCG sides with NIST assessors' grades on every reranker pair that these grades separate significantly. RCP-nDCG also resolves many of nDCG's ties and separates 1.9 times as many reranker pairs on NanoBEIR. Rubric calibration thus turns relative LLM judgments into dense relevance labels that are comparable across queries and agree with human judgment.

0.28

Discrimination, Calibration, and Ranking: A Unified Matrix Framework

cs.LG cs.AI ·2026-11-01 ·Ronaldo Cristiano Prati, Farangis Johari
code/data similarity 0.55
Abstract · 88 pages, 8 figures. Lean 4 formalisation and experiment code: https://github.com/rcprati/discrimination-calibration-ranking. Implemented in…

Discrimination, calibration, and ranking are the three pillars of classifier evaluation, yet no single algebraic language spans all three. A recent indicator-matrix account unifies discrimination measures (precision, recall, F1, Matthews correlation) across binary, multiclass, multilabel and related settings, using Boolean cell operations on indicator label matrices, generalised to soft labels by t-norms. Calibration measures (Brier score, log-loss) and ranking measures (nDCG, AUC) fall outside it. We close the gap with two further classes of cell operators: deviation operators phi(y - yhat), which give a Calibration Framework, and rank- and slice-dependent operators (positional gains for DCG, column concordance cells for AUC), which give a Ranking Framework. Both inherit the micro/macro/exemplar aggregation architecture. For calibration, the three Brier aggregations collapse to one scalar; the Murphy decomposition holds in matrix form (unbinned, binned and isotonic); and, within the Brier-type class-indexed family, a measure is skew-invariant if and only if it depends only on the class-conditional errors alpha_j and beta_j, the calibration analogues of recall and specificity. For ranking, raw DCG collapses across aggregations while nDCG does not; per-class nDCG and per-class AUC are related by a row-column duality; micro-AUC differs from pooled ROC-AUC; raw positional scores are never strictly proper, and nDCG and AUC need not be proper. The skew-invariant per-class building blocks share one class-conditional expectation template. Experiments on two multilabel benchmarks illustrate the distinctions. Many of the results are machine-checked in Lean 4; the proofs, the experiments and the aggmetrics.scores package are public: https://github.com/rcprati/discrimination-calibration-ranking

0.40