CS Paper Radar

Revisiting N2DCG: An Empirically Grounded Reformulation of Carousel Recommendation Evaluation

Jingwei Kang, Santiago de Leon-Martinez, Maarten de Rijke, Harrie Oosterhuis

cs.IR cs.HC ·submitted 2026-08-22 09:40 UTC · v1
top venue

Abstract

Carousel interfaces have been widely used in video and music streaming services, yet it remains unclear how to properly evaluate recommender systems in these two-dimensional layouts. N2DCG has been proposed to address this gap by adapting NDCG to carousel-based recommendation, but it relies on unverified assumptions borrowed from the single-list web-search setting that do not transfer well to two-dimensional carousel layouts. We identify two substantial limitations of N2DCG: its ideal ranking, used for normalization, violates carousel constraints, and its discount function does not reflect user browsing behavior observed in empirical data. To address both limitations, we propose a reformulation of N2DCG that normalizes appropriately by respecting constraints and uses an empirically grounded discount function. We validate the proposed metric, showing that it better reflects users' empirical behavior on real-world eye-tracking data and better predicts the comparison results of carousel layouts simulated based on empirical examination patterns.

Journal ref
Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM 2026), Rome, Italy, November 7-11, 2026
DOI
10.1145/3799682.3839894

Closest papers

Nearest neighbours by meaning across the whole index.

From Click Modeling to Offline and Off-Policy Evaluation in Carousel Recommendation

cs.IR cs.HC ·2026-08-22 ·Jingwei Kang
top venue similarity 0.68
Abstract

Carousel interfaces are widely used in modern recommendation systems. Unlike traditional interfaces that present a single ranked list, carousels simultaneously present several ranked lists to the user, as horizontally swipeable rows stacked on top of each other. In this design, the rankings are closely tied to the two-dimensional layout. Consequently, user behavior is shaped not only by item preference, but also by row organization, viewport constraints, and item context. This tight coupling between ranking and presentation complicates the interpretation of user feedback, introducing new challenges for recommendation evaluation. My PhD research aims to address these challenges by rethinking how carousel clicks are modeled and how carousel recommendation policies can be evaluated from logged interaction data. So far, I have studied how users interact with carousel interfaces and developed a click model design framework that prioritizes mathematical relationships between observed variables over latent behavioral assumptions. Building on these results, my ongoing work includes a project using discrete choice models to represent clicks as choices, alongside a project that develops carousel-specific offline metrics. As a next step, I plan to develop off-policy evaluation methods that estimate the performance of recommendation policies from logged interactions. Taken together, the expected contribution of my thesis is a connected body of work that links carousel click modeling with offline and off-policy evaluation, so that carousel recommendation policies can be improved more reliably.

0.57

TREND-10K: A Comprehensive Dataset for Next-Generation Video Quality Assessment Based on Preference-Driven Media

cs.AI ·2026-08-11 ·Ziheng Jia, Zicheng Zhang, Junqi Zhang, Jiaying Qian, Jiarui Wang +2 more
similarity 0.63
Abstract

The increasing prominence of short-video platforms, coupled with the advanced commercialization of AI-generated content (AIGC) videos, has led to a shift in the types of video media trend consumed by users in their daily lives. Traditional user-generated content (UGC) is gradually being replaced by professional short dramas and AIGC entertainment. Consequently, VQA for contemporary media content has become increasingly important. This requires a unified evaluation framework that can handle diverse video content and evolving media trends. In this context, we introduce TREND-10K, a next-generation comprehensive VQA dataset consisting of the trend-driven part and the static part, containing $10,000$ videos across a wide spectrum of content types. The trend-driven part is based on the TREND-Search framework, which captures user preference profiles from trending lists on online platforms and formulates sampling strategies based on these profiles. The static part, on the other hand, is composed of supplementary samples selected from publicly available datasets. To support unified evaluation for various video types, we incorporate three evaluation dimensions: technical, aesthetic, and AIGC-trace. Experiments show that our dataset ensures high annotation quality and exhibits remarkable generalization across multiple content categories. In conclusion, our work presents a robust framework for advancing VQA, addressing challenges caused by the temporal evolution of user perceptual habits and preferences.

0.04

SPADE: Escaping the Popularity-Similarity Frontier to Measure Serendipitous Recommendations

cs.IR cs.AIcs.LG ·2026-09-25 ·Tobias Vente, Maarten Peirsman, Noah Daniëls, Hannu Toivonen, Bart Goethals
similarity 0.62
Abstract

Recommender systems engineer serendipity to foster active exploration and break predictable consumption cycles. The problem with existing offline beyond-accuracy metrics is that they often either isolate historical similarity or global popularity. We aim to design an evaluation metric that examines similarity, popularity, and actual user relevance. To achieve this, we introduce SPADE (Serendipitous Pareto Distance Evaluation). SPADE maps all items into a two-dimensional space to directly calculate a user-specific Pareto frontier of maximally popular and historically similar items. The final serendipity score is then computed by averaging the minimum Euclidean distance from this boundary strictly for the correctly recommended test-set items. Evaluating SPADE across five datasets and five baseline algorithms confirms its effectiveness; our results show that the metric successfully prevents algorithms from exploiting beyond-accuracy measures with irrelevant or non-personalized recommendations, reliably isolating serendipitous discoveries.

0.16

DRanker: A Transformer-Based Multi-Task Ranking Model for the Disney+ Homepage

cs.LG ·2026-10-11 ·Priya Nirmal Singh Khokher, Daniel Nemirovsky, Benjamin P. Whitesell, Ethan Sukrae Lee, Eric Wulf +2 more
top venue similarity 0.61
Abstract

We present DRanker (Disney+ Ranker), a transformer-based multitask ranking model serving personalized recommendations on the Disney+ homepage. We describe the encoder-only architecture over enriched engagement-history sequences paired with a lightweight causal debiasing mechanism, and the inference stack that enables transformer-based ranking at scale. Through iterative development, we systematically evaluated more complex alternatives and found that simpler designs consistently outperformed them on offline metrics, qualitative behavioral evaluation, and in production. Deployed globally, DRanker drives over 1% lift in our key engagement metric and 0.5% lift in daily active users across all experiences powered by the ranker. We share practical lessons on data engineering, debiasing, and architectural choices that shaped our ranking system at scale.

0.30

Ranking Bandits for Carousel Interfaces with Observable Browsing Depth

cs.LG ·2026-10-04 ·Takuma Yasuda, Atsuyoshi Nakamura, Dai Hai Nguyen
theory similarity 0.60
Abstract · Some mistakes are fixed. One author is added

Carousel interfaces allow a recommender system to directly observe how far a user has browsed. This signal distinguishes displayed but unclicked items from items that were never displayed, whereas conventional ranking-bandit models, including cascade and position-based models, generally treat examination as latent. We formulate a ranking-bandit problem in which a learner presents a list of $L$ items, observes the user's maximum browsing depth, and receives click feedback only for positions up to that depth. The objective is to maximize the expected number of clicks under an unknown item-attractiveness vector and a browsing-depth distribution. We propose three algorithms based on UCB, Thompson Sampling, and DMED, all of which update item statistics only from observed exposures. We derive an instance-dependent logarithmic upper bound for our UCB-based algorithm and an asymptotic upper bound for our DMED-based algorithm that coincides with the lower bound as its parameter $α\downarrow 0$, establishing asymptotic optimality in this limit. Simulations in synthetic shallow- and deep-browsing environments, together with experiments parameterized from RecGaze interaction logs, show that OD-TS attains final mean regret similar to PBM-TS, while the proposed methods achieve lower final mean regret than PBM-UCB.

0.14

Beyond Raw Engagement: A Counterfactual Observability Framework for Recommender Systems at Netflix

cs.IR cs.AI ·2026-09-19 ·Chaoran Guo, Ding Tong, Ting-Po Lee, Scarlet Chen
similarity 0.59
Abstract

Understanding the performance of large-scale recommender systems remains an underexplored challenge, especially for content creators and model developers. The raw engagement signals available to them, such as views and clicks, conflate content quality, model behavior, presentation bias, and audience reach, making it hard to attribute outcomes to the right cause. In this work, we present a general evaluation framework that enhances observability across multiple recommender systems at Netflix and demonstrate its effectiveness through several production deployments. The framework treats recommender-system observability as a counterfactual measurement problem: estimating what the recommender would have done, and what engagement would have followed, in the absence of a specific content item or model decision. We articulate three stakeholder-centered observability principles for content creators and model developers, and propose measurement methodologies covering bias reduction, relativity, and incrementality, applicable to both single-stage and cascading recommender systems and serving both audiences from a single measurement foundation.

0.21

SwapRec: Warming Up Cold Items Through Training-Time Swaps

cs.IR cs.MM ·2026-09-01 ·Marta Moscati, Jan Malte Lichtenberg, Davide Abbattista, Antonio De Candia, Laura Boggia +1 more
published similarity 0.59
Abstract · Accepted at DaQuaMRec @ RecSys 2026: Second International Workshop on Data Quality-Aware Multimodal Recommendation

Interactions with cold items negatively impact real-time personalization of ID-based recommender systems. This is because the use of such interactions degrades user preference estimates, whereas excluding cold items from the user profile prevents real-time recommendation updates. In industrial scenarios, one heuristic often applied to address this shortcoming at inference time is to replace, i.e., "swap", cold-start items by their most similar "warm" neighbor, where similarity is inferred from the items' side information. In this paper, we demonstrate that sequential models, most often used for real-time personalization, are not robust to such swaps, and propose SwapRec, an approach to address this issue. SwapRec relies on using the same swap heuristics already at training time. We apply SwapRec to state-of-the-art models for sequential recommendation and analyze its impact by means of quantitative experiments in three recommendation domains (online shopping, movie, music). The experimental results show that, irrespective of the underlying sequential architecture, our easy-to-implement SwapRec approach allows for substantially more accurate recommendations when in presence of interactions with cold items, simultaneously leading to a larger percentage of cold items in the recommendation lists.

0.23

Explainable Recommendations at Scale: LLM Rationales for YouTube Music Artist Discovery

cs.AI cs.IR ·2026-09-20 ·Xiao Liu, Yanwei Song, Srivaths Ranganathan, Yuan Chen, Zheyun Feng +5 more
similarity 0.59
Abstract

Modern music streaming platforms face a persistent tradeoff: exploiting familiar content versus driving the exploration of novel items. While users frequently desire discovery, they hesitate to select unknown artists over proven favorites. Providing transparent, natural language rationales that explain why an unexplored item is recommended lowers this barrier. However, while Large Language Models (LLMs) excel at this nuanced explainability, their real-time deployment is severely bottlenecked by prohibitive inference costs and computational overhead. In this paper, we present an industry case study of a decoupled recommendation architecture that successfully scales exploration without compromising latency. Our system isolates LLM inference asynchronously offline, pre-computing personalized candidate pools of undiscovered artists alongside tailored rationales. Large-scale online A/B experiments validate our design. We demonstrate that combining LLM-backed recommendations with these explanatory rationales significantly reduces the trust barrier for new content, yielding statistically significant improvements in both user exploration and overall engagement on the discovery surfaces.

0.19