CS Paper Radar

Yielding Fairness: Coupled Learning and Fairness for Value-Weighted Ranking

Ami Tavory, Yaniv Sheena, Shaked Bar, Yair Zagdansky

cs.LG ·submitted 2026-11-01 06:49 UTC · v1

Abstract

Risk screening and recommendation systems pass or block accounts that generate both value and loss. Under a blocked-value budget, a cap on expected value forgone through blocking, the optimal ranking orders accounts by expected value per unit expected loss, a ratio of conditional expectations. This rule can conflict with accepted harm or net-harm criteria, blocking accounts that another criterion would pass at the same budget. We call these cases account-level objections and define group complaint disparity as the difference between groups' normalized realized objections. Weighted binary cross-entropy (W-BCE) implements the value-per-loss order with one score. Objection penalties modify the ranking, while group multipliers scale those penalties differently by group, tracing tradeoffs between loss capture and complaint disparity. This preserves the value-budgeted ranking structure and makes the cost of reducing contestable blocks explicit. Across public recommendation, insurance, credit, and mortgage datasets, we compare undamped W-BCE with ordinary BCE using yield Gini, an area summary of loss capture versus blocked value. Logarithmic weight damping is evaluated separately. Held-out evaluations measure capture-disparity tradeoffs. Validation-selected group-aware complaint penalties improve complaint closure in some cohorts. A corrected comparison normalizes and enlarges candidate policy banks using validation-only price sweeps and retains all ten applicable sources. The source-level results do not establish a statistically significant average advantage over group-blind pricing at the fixed minimum retained loss capture. We also compare with blocked-count, blocked-value, and protected-loss parity postprocessors, which do not directly constrain realized objections, and measure objections in candidate-count and loss units.

Comment
17 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

A Rank Graduation metric for Algorithmic fairness

cs.LG stat.APstat.ML ·2026-09-30 ·Dalia Atif, Paolo Giudici
similarity 0.62
Abstract · 44 pages, 5 figures

Fairness assessment in algorithmic decisions that affect individuals, such as credit scoring, often relies on parity measures calculated at the aggregate group level. Such measures may not reveal which individuals experience unfairness or which explanatory factors contribute to it. In this paper, we propose a rank-based framework that evaluates fairness through the distribution of model prediction errors, thereby linking fairness assessment with predictive accuracy and explainability. The framework combines Rank Graduation Fairness (RGF), its integrated measure AURGF, a centered Cramer--von Mises permutation test, and a feature removal procedure for fairness explainability. We evaluate the methodology using logistic regression, random forest, gradient boosting, and a multilayer perceptron. The simulation study shows that protected-group imbalance can reverse descriptive fairness comparisons, whereas the proposed inferential procedure correctly distinguishes fair from unfair mechanisms. Its application to HMDA mortgage data produces model rankings that differ from those obtained with classical fairness criteria. Tree-based models, rather than logistic regression, provide the strongest combination of predictive accuracy and rank-based fairness, while the fairness null hypothesis is rejected for all four models. The persistence of disparity across statistical, bagging, boosting, and neural network specifications, together with the feature removal results, indicates that the observed unfairness is not specific to a single algorithm or predictor, but is associated with group differences embedded in the characteristics of the lending data. These findings support a broader approach to trustworthy artificial intelligence that combines predictive accuracy, fairness measurement, statistical inference, and explainability.

0.17

Fairness in Link Prediction Beyond Demographic Parity: A Reproducibility Study

cs.LG cs.SI ·2026-08-10 ·Valentijn Oldenburg, Floris de Kam, Stef de Wildt, Jarno Nilson Balk
publishedcode/data similarity 0.60
Abstract · Published in Transactions on Machine Learning Research (05/2026)

In fair ranked link prediction, demographic parity ($Δ_\mathrm{DP}$) is a common fairness metric. Yet, Mattos et al. (2025) argue that it fails to detect exposure bias because it ignores where links appear in the ranking. In this study, we reproduce this claim by showing that $Δ_\mathrm{DP}$ can indicate aggregate parity even when some subgroup-pair links are systematically ranked lower than others. The proposed rank-aware Normalized Discounted KL-divergence (NDKL), however, does detect such disparities. We also reproduce the effectiveness of MORAL, a post-processing method that improves exposure-based fairness while maintaining competitive utility. Beyond reproduction, we assess robustness using synthetic homophily settings, categorical sensitive attributes, and additional fairness and utility metrics, including subgroup-pair-adapted Attention-Weighted Rank Fairness (AWRF). Overall, our results show that exposure-based metrics uncover biases hidden by $Δ_\mathrm{DP}$ and that MORAL reduces these biases with minimal utility loss across diverse settings and datasets. We release a corrected, reproducible implementation at https://github.com/Floris93100/reproducing-MORAL.

0.48

Efficient Conformal Calibration for Paired Recommendation Decisions

cs.LG ·2026-10-31 ·Xinyu Qiao, Yichen Lin, Zhewen Chen, Xue Wang, Tao Yao
similarity 0.58
Abstract

Risk control can suppress useful changes to a frozen recommendation rule. We study how to recover valid activation from the same observed fractional losses. Two specified internal calibrators can lose efficiency by compressing losses into their sum or clipping auxiliary inverse counts before averaging. We propose Averaged Conformal Risk Calibration, or ACRC, which averages before clipping. A product identity proves expected-risk validity, while pointwise bounds characterize the additional activation. For fixed populations sampled without replacement, we connect activation to signed utility under heterogeneous users and switching constraints. Near the budgets, positive population mean gain and the stated nondegeneracy conditions give positive leading-order utility increments despite sampling dependence. Across six datasets, changing only calibration improves discounted net utility by $0.010226$ over raw and $0.008337$ over RB. A separately frozen Jester confirmation gives $0.011025 \pm 0.001082$ over raw with the rule and filter fixed, using a five-seed approximate 95 percent paired interval. Against information-matched SCoRE selecting risk or reward scores, the mean utility difference is $+0.015934 \pm 0.001440$. These gains use more mean realized risk within the declared budgets. Upstream failures delimit the benefit: efficient calibration releases useful modifications when the proposals themselves have value.

0.33

Fair Bubble Sort: Provably Optimal Fair Ranking with Continuous Sensitive Attributes

cs.LG cs.CYcs.DS ·2026-11-17 ·Antonio Ferrara, Fabio Vitale, Atsushi Miyauchi, Francesco Bonchi
top venuetheorycode/data similarity 0.57
Abstract · 35 pages, 9 figures, 4 tables. Accepted at NeurIPS 2026. Code: https://github.com/Ambress92/FairBubbleSort

Most fair-ranking methods assume categorical protected attributes, requiring continuous attributes such as age or income to be discretized into groups. This introduces arbitrary thresholds, loses fine-grained information, and ignores the natural ordering of the attribute. In this paper, we study the problem of fair ranking with continuous protected attributes without relying on thresholding. We introduce Continuous Pairwise Parity (CPP), a fairness notion based on Kendall correlation between ranking positions and the protected attribute, and seek rankings that satisfy a prescribed CPP constraint while minimizing their Kemeny distance from an initial score-based ranking. To solve this problem, we propose Fair Bubble Sort (FBS), an efficient post-processing algorithm based on targeted adjacent swaps. We prove that FBS is exactly optimal for the unweighted Kemeny distance and provide approximation guarantees for the score-weighted setting. FBS is output-sensitive, with running time determined by the number of swaps required to meet the fairness constraint. Experiments on synthetic and real-world datasets show that FBS is scalable and consistently outperforms discretization-based baselines, achieving a superior fairness-accuracy Pareto front.

0.54

Surrogate Losses for Fairness-Aware Classification: Fisher Consistency and Calibration

cs.LG stat.ML ·2026-11-01 ·Changcheng Song, Yi Yang, Xiangyu Chang, Xiaoling Lu, Yufeng Liu
hot direction similarity 0.56
Abstract

Fairness-aware classification seeks to balance predictive accuracy and fairness across demographic groups. Although surrogate losses facilitate the optimization of fairness-aware objectives, it remains unclear when minimizing these losses yields optimal decisions under the original fairness-aware risk. This paper investigates the statistical validity of surrogate-based fairness-aware classification under demographic parity and equal opportunity, considering both attribute-blind and attribute-aware settings. We introduce fairness-aware Fisher consistency and characterize it through conditions on the derivatives of the surrogate losses. The characterization shows that consistency depends not only on the individual loss functions but also on their interaction within the fairness-regularized objective. We further establish a calibration inequality that relates the excess fairness-aware risk to the excess surrogate risk, providing a quantitative connection between surrogate optimization and the original classification objective. Numerical experiments illustrate how deviations from the consistency conditions affect fairness-aware excess risk and how the regularization parameter influences the well-posedness of surrogate optimization. An application to consumer credit default prediction demonstrates that appropriately calibrated fairness regularization can substantially reduce demographic disparities while maintaining predictive performance.

0.25

Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning

cs.LG stat.MLstat.ME ·2026-10-30 ·Bin Luo, Param Venkat Vivek Kesireddy, Anil Vallepu
theory similarity 0.56
Abstract

Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight $w_i\in[0,1]$ that enters the likelihood linearly -- DPO is $w\equiv1$, a reversed comparison $w_i=0$ -- and estimates the weights jointly with the policy under a minimax-concave penalty on $-\log w_i$. The penalty keeps every comparison the model agrees with at exactly $w_i=1$ and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order $\log n$. Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.

0.26

Rethinking Fairness Metrics for Multi-Class and Multi-Group Assessment: Not Everything is Binary

cs.LG cs.AI ·2026-11-02 ·Matteo Grazioso, Fabio Naccarella, Chiara Gallese, Marco S. Nobile
similarity 0.56
Abstract · Submitted to ACM Transactions on Intelligent Systems and Technology (TIST)

Artificial Intelligence, and Machine Learning in general, is becoming pervasive across all domains and disciplines. By means of this data-driven approach, algorithms can learns complex task just by learning from samples. However, if the data-set is biased, they might lead to the development of unfair computer systems, that is, characterised by a prejudice towards specific individuals or groups, which might get an unmotivated output from the system. This circumstance lead to the development of multiple metrics designed to quantify the level of unfairness of an algorithm. However, a generalised assumption in the literature is that the sensitive features and the output labels are binary. When the data is not binary, it is forcefully binarized using a one-versus-the-rest simplification. In this work, we discuss the limitations of existing metrics and propose novel, mathematically sound, and generalized multi-group and multi-class formulations of the metrics. Ultimately, our results highlight the empirical assessment distortion caused by binarization strategies, which reveals the limitations of binary fairness metrics in the multi class setting.

0.22

JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

cs.AI ·2026-09-29 ·Qi Cao, Kangning Liu, Xuan Kan, Shunwen Tan, Yang Pei +8 more
similarity 0.56
Abstract

LLM judges are inherently subjective, often favoring different responses in pairwise comparison when neither option is objectively wrong. To study this subjectivity, we introduce JudgeProfile, a framework that dissects LLM evaluation into perception (how a judge compares two responses across specific attributes like clarity, correctness, and detail) and prioritization (how much each attribute influences the final choice). We curate SubjectiveSet, a dataset of 50,013 response pairs from 17 public data sources, evaluated by 21 LLM judges across 87 attributes. We find a hidden consensus in perception: judges frequently agree on attribute judgments even when their overall choices diverge. Building on this separation, we first characterize each judge's prioritization using attribute weights estimated from its own overall choices. These weights differ across judges even when estimated from the same attribute judgments. We then learn new weights from reference labels to adapt their decisions to a target evaluation standard. Reweighting perceived attributes improves average held-out agreement with reference labels from 66.48% to 71.97%, outperforming fine-tuning and rubric prompting. Our findings show that understanding and steering the subjectivity of LLM judges requires attention not only to what they perceive, but also to how they prioritize it.

0.11