CS Paper Radar

Ask More, Hear Less: How a Review Prompt Changes What Online Ratings Show

Zihan Chen, Lei Nico Zheng, Di Zhu

cs.HC cs.CYecon.GN ·submitted 2026-11-02 06:50 UTC · v1

Abstract

Platforms ask customers for online reviews to reduce self-selection, but theory conflicts on whether review solicitation makes online ratings more or less informative. Selection theory implies that asking helps, voice theory that it dilutes the news, and research on rating inflation that compression destroys information. We reconcile these views with signal detection theory, which separates what a rating knows (its sensitivity) from what it shows (its criterion). We test the framework on Steam, a leading PC video game store, whose client began in October 2019 to prompt players for a review after about three hours of play. Using a difference-in-discontinuities design at that inferred threshold in 21.6 million Steam reviews of 315 games, we find that review counts just above the threshold become 4.25 times those just below, and the jump in the positive share grows by 6.6 percentage points. Agreement with critics changes little near launch and erodes later. The rating's movement changes far more: pass-through from reviews written in the first two hours falls from 0.73 to 0.18 just above the threshold, against 0.81 to 0.64 just below, and a fixed rating tier responds to 12% of downturns in those reviews above the threshold, down from 42%, against 41% and 47% below it. What the stream knows changes less. At equal review counts its sensitivity falls more above the threshold, because added reviews respond about half as much and scores near the ceiling are noisier per review, but at natural counts added volume offsets this loss, leaving a change of -0.03 relative to the stream below (95% CI -0.13 to 0.06). Failures that developers acknowledge, which run through complaints, still move the score after launch. Review prompts therefore mainly change how a rating must be read, and platforms that solicit reviews should recalibrate their rating labels.

Comment
Working paper. 35 pages, 9 figures, 24 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Statistical Foundations for a Google Play User-Review Sentiment Index: Signal Fusion, Shrinkage, Distributional Validation, and Dynamic Smoothing

cs.CL stat.MEstat.AP ·2026-09-25 ·Marco Mandap
similarity 0.62
Abstract · 29 pages, 5 tables, 3 figures; includes a deterministic synthetic stress test and a descriptive application to 6000 Google Play reviews from…

The decision problem is whether valence among the users who write reviews of an app has changed enough to warrant investigation of a release, outage, or support issue. We propose a statistical specification for that decision, applied to written Google Play reviews collected under a declared locale, language, sort order, retrieval cap, and window rule. That endpoint returns a selected collection rather than a random sample, so the index describes review writers under the declared protocol and not all app users. What is new is not a sentiment classifier but an auditable interface: an ordinal threshold model is the preferred representation of star ratings, a calibrated two-signal generalized least-squares estimator is retained as an interpretable linear approximation when held-out labels support its conditional-unbiasedness assumption, the estimand is declared before the estimator, and every downstream module--uncertainty, diagnostics, and regularization--is tied to that declaration. The framework separates the latent average of a fixed collected set from the mean of a protocol-defined population of review writers, so measurement and between-review variation are not conflated, and treats helpfulness and recency weights as a different policy target. Shrinkage, cross-app empirical Bayes, histogram and dependence diagnostics, and dynamic smoothing are specified as separate modules with defaults and mandatory sensitivity analyses. A reference implementation, a declared 6000-review three-app collection, and a synthetic stress test exercise the displayed equations: fusion reduced error in clean and small-window scenarios, coordinated contamination caused severe bias and interval failure, and the collected sample reproduces the abstention rules. No real-review validation is claimed.

0.27

How Much Were You Told? Measuring External Information in Peer Reviews

cs.CL ·2026-09-23 ·Matthieu Dubois, Pablo Piantanida, François Yvon
benchmark similarity 0.58
Abstract

Conference policies distinguish using Large Language Models (LLMs) to polish one's own review from delegating the critique, but current Artificial Text Detection (ATD) methods largely measure surface form rather than the origin of its content. We instead measure the external information carried by a review: information not explained by the reviewed paper and a generic reviewing instruction. We propose Self-Conditioning, an unsupervised information-theoretic estimator that compares the likelihood of a review under its production context with its likelihood when that context is augmented with hints extracted from the review itself. On the IntelLabs peer-review benchmark, Self-Conditioning separates fully-delegated from machine-polished reviews with AUC up to $1.0$ while remaining largely insensitive to surface rewriting. Moreover, as generators receive increasing amounts of externally-provided information, their scores move monotonically towards the human regime, unlike standard ATD baselines. High-temperature sampling can evade the estimator, but at the cost of output quality.

0.09

The Role of AI in Online Reviews

cs.CL cs.AIcs.HCcs.SI ·2026-08-30 ·Valeria Lerman, Oren Rigbi, Yaniv Dover
published similarity 0.58
Abstract

The rapid adoption of large language models (LLMs) creates new opportunities for strategic content generation on online platforms, including potentially harmful forms of manipulation that may undermine platform effectiveness and reshape platform dynamics. However, measuring such activity is difficult because AI-generated content is rarely directly observable. We introduce an empirical approach that leverages discrete LLM supply shocks - abrupt changes in model prices and capabilities, and contrasts verified with non-verified reviews to identify changes in platform activity associated with generative AI supply improvements. We apply this approach to more than 13 million reviews from Trustpilot, one of the leading online platforms for business reviews. A robust finding is that following LLM supply shocks, unverified reviews shift toward greater negativity: more 1-stars, fewer 5-stars, and lower ratings, with effects driven primarily by new model releases and concentrated among firms with the lowest and highest review volumes, suggesting that strategic AI use may reshape platform competition dynamics. We further find that LLM supply shocks trigger short, concentrated bursts of review activity. Together, these findings suggest that generative AI is already reshaping how reputation and competition operate on online platforms.

0.40

Association between Player Perception and AI Disclosure on Steam: An Exploratory Study

cs.IR cs.HC ·2026-10-19 ·Fabian Wiesinger, Johanna Pirker, Denis Helic
published similarity 0.58
Abstract

The use of generative AI in game development is growing rapidly and is the subject of intense debate in which player reactions play a central role. This paper contributes to this discussion by analyzing Steam reviews to examine whether games with AI disclosure are associated with different review patterns. Using a dataset across multiple disclosure groups, we apply linguistic analysis including keyword-based, word frequency, and characteristic word analysis to investigate the AI-related discussion on Steam. Our results show that AI-disclosed games are associated with higher levels of AI-related criticism in negative reviews. Our findings provide a first exploratory step towards understanding how players perceive AI disclosure.

0.20

Do Reviewers Still Reward Lexical Complexity? A Frozen-Rater Study of Preference Drift in 124K ICLR Reviews

cs.CL cs.DLcs.LG ·2026-09-08 ·Jiabin Zheng
code/data similarity 0.55
Abstract · 23 pages, 8 figures, 11 tables. Code and the machine-readable records behind every number: https://github.com/Biajin-PKU/frozen-rater-drift

Large language models have collapsed the cost of producing lexically elaborate prose, and whether peer reviewers still reward it is a question about the evaluator, not about the text. When the association between a writing cue and review scores moves across years, the reviewers may have changed, the submissions may have changed, or both, and a regression of scores on text cannot say which. We separate the two with a frozen rater: 81,850 machine reviews of ICLR submissions from 2018 to 2025, all generated in one February-April 2025 window with one model family and one prompt, so that its year-to-year coefficients track submission composition alone and the human-minus-frozen trend difference identifies reviewer preference drift. On 32,638 submissions with 124,615 human reviews, the human coefficient on non-domain lexical complexity falls from +0.142 to -0.015 while the frozen rater moves from +0.080 to +0.082; the three-way difference-in-differences is -0.0100 (q=0.013), and forty random-wordlist placebos through the same specification centre on zero. Humans still reward sentence-length variability, which the frozen rater never registers, while the frozen rater still pays for lexical complexity at its earlier rate. Every claim is held to a double gate of false-discovery control and interval exclusion, and the findings that failed adversarial re-testing are reported. Reviewers discounted a cue whose production cost collapsed, as models of manipulable signals prescribe; an LLM judge calibrated to historical human preferences inherits the earlier schedule and drifts out of alignment while its agreement with humans on totals stays ordinary.

0.43

Needs Your Help: Understanding Platform-Directed Rating Participation in Community Notes on X

cs.HC ·2026-09-16 ·Shuning Zhang, Changxi Wen, Jiuchang Wang, Dai Shi, Gabriele Lenzini +2 more
similarity 0.54
Abstract

Community-based fact-checking is promising in countering misinformation, yet its scalability is constrained by slow rating accumulation. To address this challenge, platforms such as X implement platform-directed rating mechanisms, specifically through ``Needs Your Help'' algorithmic prompts, to target unresolved notes. Using a dataset of over 220 million rating contributions -- including 1.9 million platform-directed ratings -- on X, we examine contributors' response to note prompts, notes' resolution, and raters' spillovers. We found (i) at note level, population-sampled ratings concentrate on recent notes with certain helpfulness and high disagreement. Once sampled, population-sampled rating was associated with faster and more transitions to resolved statuses. (ii) At rater level, following raters' first observed population-sampled rating, raters exhibit significant yet modest increases in daily ratings, rating pace and tag usage, while other behaviors show no change. These highlight the promise of algorithmic nudges to guide volunteer attention toward contested content, accelerating consensus while sustaining rater engagement.

0.27

Echoes in the Algorithm: Analyzing the Fidelity of User Preferences Against Realized Platform Reach

cs.HC ·2026-09-08 ·Emelia Hughes, Tim Weninger
benchmark similarity 0.53
Abstract

What does popular content look like when platforms withhold the usual cues? On TikTok, users still form impressions about which videos are taking off even when likes and view counts are hidden, delayed, or pushed to the margins of the interface. We study this problem through TokOrNot, a web-based game in which participants compared pairs of TikTok videos and reported (i) which one they preferred and (ii) which one they believed had reached a larger audience. We benchmark these judgments against verified public view counts, which we use as a bounded proxy for realized platform reach. Across 3,513 judgments from 363 participants, participants identified the higher-reach video only modestly above chance (56.75%, 95% CI: 56.01-58.55). Preference aligned with the higher-view video at a similar rate, while preference and prediction matched in 83.48% of trials (95% CI: 83.12-85.95). Performance also varied across content categories. Taken together, these results do not suggest that users can reliably read platform success from content alone. Instead, they point to a looser and more uncertain interpretive process in which reach judgments often track personal taste or other weak heuristics when explicit popularity cues are absent. We discuss the implications for algorithmic literacy and for interface designs that reduce visible metrics without leaving users to infer reach from uneven or idiosyncratic cues alone.

0.23

Twice-Told Critiques: Claim-Level Homogenization of Peer Review in the LLM Era

cs.CL cs.AIcs.DL ·2026-11-19 ·Ilya Gusev
similarity 0.52
Abstract

Peer review relies on several readers finding different problems. We measure critique overlap in conference and journal reviews. At ICLR, measured overlap rose from 12.0% in 2026 to 21.3% in 2027, in reviews from the same stage of the review process, with year-on-year rises at NeurIPS, ICML and the F1000 and PLOS journals. A third ICLR review of a paper added 5.46 new problems to the other two in 2027, against 6.12 in 2026. We apply MELD, a detector of LLM-generated text, to all reviews. Within a paper, two reviews that MELD flags overlap more than two unflagged reviews: by 3.0 points at ICLR in 2024-2025, by 4.5 points in 2026-2027, and by 5.4 points across five journal sources. These results provide evidence that LLM-flagged reviews of one paper are less distinct from each other than unflagged reviews, and suggest that the diversity of critiques among reviews of a single paper is falling.

0.43