CS Paper Radar

Topic-Level Fairness in Retrieval-Augmented Generation

Jaeyoung Park, Hantian Zhang, Kexin Rong, Steven Euijong Whang

cs.AI ·submitted 2026-10-26 01:19 UTC · v1

Abstract

Group fairness in retrieval-augmented generation (RAG) systems is typically evaluated using metrics aggregated across queries. However, such metrics can conceal topic-level disparities: a system may favor women on questions about human rights and men on questions about actors and filmmakers, appearing nearly neutral overall even though neither topic is fair. We therefore study topic-level fairness, which requires group fairness to hold within each topic rather than only in aggregate. To reduce the topic-level disparity, we adapt the reranker separately for each topic with a Bradley-Terry objective under direct supervision from the generated output: preference pairs built from the generator's own output carry the direction that topic requires. With preference pairs built from RoMQA alone, the adaptation reduces the mean absolute per-topic disparity by 51% on RoMQA and by 40% on TREC Fair Ranking while retaining accuracy. No baseline reaches the reduction, even when modified to treat each topic separately rather than apply one correction to every topic. Neither does our method with one reranker for all topics, which follows the direction that most topics require.

Comment
21 pages, 8 figures, 16 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Robustness Analysis of Pareto-based Joint Evaluation of Fairness and Relevance in Recommender Systems

cs.IR ·2026-10-20 ·Theresia Veronika Rampisela, Maria Maistro, Tuukka Ruotsalo, Christina Lioma
top venuecode/data similarity 0.63
Abstract · Accepted to ACM Transactions on Recommender Systems (TORS). Extension of arXiv:2502.11921(1)

Recommender Systems (RSs) can be evaluated for many aspects, two of them being recommendation relevance and fairness. These two aspects can be evaluated either in isolation, using a measure for each aspect; or jointly with a single`relevance-aware' fairness measure that quantifies fairness with respect to relevance. Yet, neither approach provides a simultaneous evaluation of both relevance and fairness: the first approach may result in two different rankings of models, each corresponding to an evaluation aspect; the second approach often provides an imbalanced evaluation of the two aspects. To address this gap, we propose Distance to Pareto Frontier (DPFR), a new approach to jointly evaluate RS fairness and relevance. DPFR is based on a Pareto frontier of relevance and fairness scores from existing measures. The frontier is generated from user-item interaction data and represents various Pareto-optimal trade-offs between the two aspects. Thus, the distance from a point in the frontier provides a joint score of fairness and relevance. As DPFR can be used with existing measures, it is modular and intuitive. Our experiments with 4 recommenders and 3 fairness-aware re-rankers on 6 real-world datasets reveal inconsistent associations between existing measures and DPFR, showing that they are not reliable proxies for evaluations based on Pareto optimality. We further examine the robustness of DPFR as a joint evaluation approach for fairness and relevance by conducting experiments with artificial recommendations and with various fairness-relevance trade-offs, as well as by analysing DPFR's agreement to simple measures of fairness and relevance. These analyses show that DPFR is more robust than existing joint evaluation approaches. Finally, we provide guidelines for evaluating fairness and relevance jointly with DPFR. Our code: https://github.com/theresiavr/robustness-DPFR-recsys-evaluation.

0.53

Aggregating User Preferences while Ensuring Equity, Diversity, and Inclusion using Graph Summarization

cs.SI cs.AI ·2026-10-05 ·Adji Marieme Sita Cissé, Malek Mouhoub
similarity 0.62
Abstract · 22 pages, 3 figures. Interactive dashboard: https://sitaacisse-boop.github.io/EDI/

Aggregating the preferences of diverse user groups into a collective outcome raises fundamental challenges of equity, diversity, and inclusion (EDI): classical aggregation rules such as Borda and Condorcet have no mechanism to prevent results from systematically favoring majority groups, collapsing onto homogeneous items, or under-representing minorities. We address this problem through EDI-constrained graph summarization. User preferences are modeled as a weighted attributed bipartite graph, and a greedy coarsening algorithm iteratively merges user nodes while enforcing three structural EDI criteria: an equity gap constraint ($ΔE$), an intra-list diversity constraint (ILD), and a group inclusion constraint. Rather than correcting fairness after aggregation, our method embeds EDI preservation directly into the graph structure. We evaluate across five datasets spanning four domains: MovieLens 100k and 1M, libimseti.cz, Rate My Professors, and OpenAlex (2018-2023). Our method, AURORA, achieves the largest and most consistent diversity gains over classical voting rules, and on MovieLens 100k at $k = 20$ it simultaneously improves all three EDI criteria over both Borda and Condorcet. On Rate My Professors, it combines high diversity (ILD = 0.808) with the highest female item representation (60%), at a moderate equity cost, and it achieves the lowest equity gap ($ΔE = 0.031$) on OpenAlex, where Borda-based methods recommend zero female authors. On libimseti.cz, the only dataset where the sensitive attribute is present on both sides of the bipartite graph, our method does not reduce the equity gap, a limitation we connect to prior findings that demographic parity is not always an appropriate target. These results demonstrate that embedding EDI constraints into aggregation structure yields more robust fairness-diversity trade-offs than post-hoc approaches.

0.29

Yuanbao-Reranker: Document-Grounded Synthesis and Preference Distillation for Text Reranking

cs.IR ·2026-11-07 ·Hongyuan Xu, Dezhi Ye, Jiahao Cao, Jie Liu, Haijin Liang +2 more
similarity 0.61
Abstract · 23 pages, 6 figures, 7 tables

High-precision text reranking is a critical component in modern search, recommendation, and agentic retrieval-augmented generation (RAG) systems. We present Yuanbao-Reranker, a family of general-purpose dense rerankers spanning 0.6B, 4B, and 8B parameter scales built upon the Qwen3 backbone. To construct diverse and reliable ranking supervision at scale, we pair answer-first and query-type-guided document-grounded synthesis with a three-stage curation funnel that verifies positive support, eliminates semantic redundancy, and enforces balanced source-by-length quotas. For each curated pair, we screen real retrieval pools with a teacher score margin to filter out potential false negatives and sample Hard, Medium, and Moderate negative tiers, yielding 2.75M $1{:}3$ ranking groups. We train the models by jointly optimizing pointwise relevance cross-entropy and a bounded minimum-gap hinge objective over both positive--negative and negative--negative pairs. This formulation transfers fine-grained teacher preferences across candidate tiers without over-constraining well-separated predictions, supporting both cross-scale and same-scale distillation followed by layerwise adaptive model merging. Across five comprehensive benchmarks spanning Chinese (CMTEB-R), multilingual (MMTEB-R), English (BEIR), long-document (MLDR), and code (MTEB-Code) retrieval, Yuanbao-Reranker consistently leads every parameter tier: the 0.6B variant reaches 70.96 Avg. to surpass 1B--4B active-parameter baselines, the 4B variant reaches 73.63 Avg. to outperform the 8B teacher (72.68), and the 8B variant achieves a state-of-the-art 74.21 Avg., with substantial gains generalizing across distinct first-stage retrievers.

0.24

FAIDER: Fairness-Aware Item-Level Decoding for Large Language Model Recommenders

cs.AI ·2026-11-06 ·Ziwei Wu, Jingrui He
similarity 0.59
Abstract

Large language model (LLM) recommenders can produce semantically different recommendations when a user's protected attribute changes, even if the interaction history remains fixed. Controlling this disparity is challenging when model weights and full-vocabulary logits are inaccessible. We propose FAIDER (Fairness-Aware Item-level DEcoding for Recommendation), which incorporates fairness into list construction by selecting complete item candidates rather than manipulating token-level logits or repeatedly repairing finished outputs. The central idea is to separate the operational criterion for a violation from how strongly generation is constrained. Conformal-inspired empirical calibration fixes a cross-group semantic-disparity threshold, while online decoding applies either hard filtering or a soft penalty adjusted through dual-inspired feedback to target a prescribed violation rate. Separately, we establish a pathwise guarantee: under a uniform enforceability assumption, the soft decoder's empirical violation rate exceeds its target by at most $O(1/T)$. Experiments across recommendation benchmarks and LLM backbones show that Soft FAIDER improves both measured semantic fairness and ranking quality over the evaluated prompt-repair baselines. It also generally achieves a more favorable fairness-utility balance than the hard variant, indicating that stricter per-recommendation constraints do not necessarily yield lower measured disparity.

0.11

Fair Like Us? Auditing LLM Alignment in Resource Allocation

cs.AI cs.CYcs.GT ·2026-08-31 ·Qishen Han, Hadi Hosseini, Joshua Kavner, Samarth Khanna, Sujoy Sikdar +1 more
top venue similarity 0.59
Abstract · Accepted at EMNLP 2026 (Main Conference)

Fair allocation of scarce, indivisible resources is an important challenge in many societal problems. While there are several formal theories of fairness, no single definition can always be satisfied. As large language models (LLMs) are increasingly used to support decisions and act as agents, they raise new concerns about distributional justice: their judgments are not directly tied to any specific fairness framework and may violate key normative principles. In this work, we introduce a general method for evaluating fairness reasoning in LLMs. We study first-person fairness judgments across a broad set of models and compare them directly with human responses on matched scenarios and elicitation conditions. We find that LLMs tend to prefer stricter fairness constraints than humans, show more self-interested behavior, are sensitive to how information is framed, and are difficult to align with human judgments using fine-tuning with current datasets.

0.42

Characterizing Predictive Multiplicity in Ranking and Its Fairness Implications

cs.LG ·2026-11-11 ·Ainhize Barrainkua, Mikel Malagón
similarity 0.59
Abstract · 32 pages, 6 figures

Prediction problems often admit competing models that achieve comparable performance yet produce conflicting outputs, a phenomenon known as predictive multiplicity. While this challenge has been extensively studied in classification tasks, its implications remain largely unexplored in ranking frameworks, which are critical for high-stakes societal decisions such as the allocation of welfare resources and social assistance. This paper investigates the phenomenon of predictive multiplicity in score-based ranking problems. We introduce formal definitions to capture multiplicity in ranking settings, supported by foundational theoretical guarantees. Furthermore, we examine the dual relationship between predictive multiplicity and algorithmic fairness in ranking problems: we analyze how model multiplicity can act as a potential source of unfairness, while demonstrating how exploring this decision space provides a principled avenue for discovering less discriminatory rankings. Finally, we complement our theoretical contributions with insightful empirical evaluations on both synthetic and real-world datasets, validating our analytical findings and illustrating their practical implications. Taken together, this work highlights the real-world relevance of predictive multiplicity in ranking and sheds light on the potential of leveraging this effect to achieve fairer outcomes.

0.20

WARP: Wasserstein-Aligned RAG for Population Opinions

cs.IR cs.CL ·2026-08-24 ·Aman Singh Thakur, Aditya Agrawal, Alwarappan Nakkiran, Alex Karlsson
top venue similarity 0.59
Abstract · Pre-print

RAG systems are increasingly used to summarize what large collections of documents say. A user asks "What do people think about X?" and receives an answer that reads as consensus. But standard top-k retrieval ranks documents by query similarity, not by how faithfully they represent the population, so minority views quietly disappear. Existing fixes fall short. Diversity re-rankers like MMR and DPP spread retrieved documents apart, but with no target distribution to aim for. Calibration methods based on KL or JS divergence do target one, yet treat opinion bins as unordered: confusing strong positive with strong negative costs no more than an adjacent-bin miss. We introduce WARP, a family of post-retrieval algorithms that calibrate retrieved evidence to the population's opinion distribution. WARP first recovers underrepresented opinions that cosine ranking may bury, then uses Wasserstein-1 distance to select documents whose sentiment-intensity distribution matches the population target, capturing the ordinal structure ignored by KL and JS divergence. We develop three variants for dense, sparse, and variable candidate pools, trading off calibration quality and speed. Across three review domains spanning 35K documents, 156 queries, and 26 entities, WARP's domain-matched variants reduce distributional error by at least 43% with sub-second latency. These gains carry through to generation: a five-judge LLM panel prefers WARP-generated answers in 86% of decided comparisons at k <= 5.

0.54

Fairness in Link Prediction Beyond Demographic Parity: A Reproducibility Study

cs.LG cs.SI ·2026-08-10 ·Valentijn Oldenburg, Floris de Kam, Stef de Wildt, Jarno Nilson Balk
publishedcode/data similarity 0.58
Abstract · Published in Transactions on Machine Learning Research (05/2026)

In fair ranked link prediction, demographic parity ($Δ_\mathrm{DP}$) is a common fairness metric. Yet, Mattos et al. (2025) argue that it fails to detect exposure bias because it ignores where links appear in the ranking. In this study, we reproduce this claim by showing that $Δ_\mathrm{DP}$ can indicate aggregate parity even when some subgroup-pair links are systematically ranked lower than others. The proposed rank-aware Normalized Discounted KL-divergence (NDKL), however, does detect such disparities. We also reproduce the effectiveness of MORAL, a post-processing method that improves exposure-based fairness while maintaining competitive utility. Beyond reproduction, we assess robustness using synthetic homophily settings, categorical sensitive attributes, and additional fairness and utility metrics, including subgroup-pair-adapted Attention-Weighted Rank Fairness (AWRF). Overall, our results show that exposure-based metrics uncover biases hidden by $Δ_\mathrm{DP}$ and that MORAL reduces these biases with minimal utility loss across diverse settings and datasets. We release a corrected, reproducible implementation at https://github.com/Floris93100/reproducing-MORAL.

0.48