CS Paper Radar

Statistical Foundations for a Google Play User-Review Sentiment Index: Signal Fusion, Shrinkage, Distributional Validation, and Dynamic Smoothing

Marco Mandap

cs.CL stat.MEstat.AP ·submitted 2026-09-25 16:54 UTC · v2

Abstract

The decision problem is whether valence among the users who write reviews of an app has changed enough to warrant investigation of a release, outage, or support issue. We propose a statistical specification for that decision, applied to written Google Play reviews collected under a declared locale, language, sort order, retrieval cap, and window rule. That endpoint returns a selected collection rather than a random sample, so the index describes review writers under the declared protocol and not all app users. What is new is not a sentiment classifier but an auditable interface: an ordinal threshold model is the preferred representation of star ratings, a calibrated two-signal generalized least-squares estimator is retained as an interpretable linear approximation when held-out labels support its conditional-unbiasedness assumption, the estimand is declared before the estimator, and every downstream module--uncertainty, diagnostics, and regularization--is tied to that declaration. The framework separates the latent average of a fixed collected set from the mean of a protocol-defined population of review writers, so measurement and between-review variation are not conflated, and treats helpfulness and recency weights as a different policy target. Shrinkage, cross-app empirical Bayes, histogram and dependence diagnostics, and dynamic smoothing are specified as separate modules with defaults and mandatory sensitivity analyses. A reference implementation, a declared 6000-review three-app collection, and a synthetic stress test exercise the displayed equations: fusion reduced error in clean and small-window scenarios, coordinated contamination caused severe bias and interval failure, and the collected sample reproduces the abstention rules. No real-review validation is claimed.

Comment
29 pages, 5 tables, 3 figures; includes a deterministic synthetic stress test and a descriptive application to 6000 Google Play reviews from three apps

Closest papers

Nearest neighbours by meaning across the whole index.

Ask More, Hear Less: How a Review Prompt Changes What Online Ratings Show

cs.HC cs.CYecon.GN ·2026-11-02 ·Zihan Chen, Lei Nico Zheng, Di Zhu
similarity 0.62
Abstract · Working paper. 35 pages, 9 figures, 24 tables

Platforms ask customers for online reviews to reduce self-selection, but theory conflicts on whether review solicitation makes online ratings more or less informative. Selection theory implies that asking helps, voice theory that it dilutes the news, and research on rating inflation that compression destroys information. We reconcile these views with signal detection theory, which separates what a rating knows (its sensitivity) from what it shows (its criterion). We test the framework on Steam, a leading PC video game store, whose client began in October 2019 to prompt players for a review after about three hours of play. Using a difference-in-discontinuities design at that inferred threshold in 21.6 million Steam reviews of 315 games, we find that review counts just above the threshold become 4.25 times those just below, and the jump in the positive share grows by 6.6 percentage points. Agreement with critics changes little near launch and erodes later. The rating's movement changes far more: pass-through from reviews written in the first two hours falls from 0.73 to 0.18 just above the threshold, against 0.81 to 0.64 just below, and a fixed rating tier responds to 12% of downturns in those reviews above the threshold, down from 42%, against 41% and 47% below it. What the stream knows changes less. At equal review counts its sensitivity falls more above the threshold, because added reviews respond about half as much and scores near the ceiling are noisier per review, but at natural counts added volume offsets this loss, leaving a change of -0.03 relative to the stream below (95% CI -0.13 to 0.06). Failures that developers acknowledge, which run through complaints, still move the score after launch. Review prompts therefore mainly change how a rating must be read, and platforms that solicit reviews should recalibrate their rating labels.

0.24

Presentation Sensitivity in Typed Decision Models for Subjective Text Evaluation

cs.CL ·2026-11-04 ·Jatin Agrawal, Radhika Mamidi
similarity 0.61
Abstract · 27 pages, 1 figure, includes appendices

Typed decision models offer structured choices, scores, and probabilities for low-cost automated evaluation. We present a controlled audit of presentation sensitivity in Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces. Across six datasets with 9,000 multiply annotated humor, helpfulness, and summarization items, we collect 702,000 judgments under interventions that vary positions, identifiers, authorship claims, endorsements, decision framing, and rating rubrics while preserving the evaluated texts. Jev's second-position probability advantage reaches 6.11 percentage points on summaries; reversing reader endorsements changes Perplexity's selected humor text on 40.40% of pairs; and rejection framing reduces Clef's summary agreement with human preferences by 23.36 points. Rating formats also shift score levels while often preserving rankings. Across these findings, repeatability, presentation robustness, and human alignment emerge as distinct properties of an evaluator. Reliable use therefore requires validating the decisions and score interpretations on which an evaluation pipeline depends.

0.16

Measuring Smartphone User Experience through a Hierarchical Metric Framework via Social Media Reviews

cs.HC ·2026-09-19 ·Xiaoteng Pan, Mingang Lan, Chenrui Zhang, Yu Su, Weijie Liu +3 more
similarity 0.60
Abstract · 36 pages, 5 figures

Smartphone user experience (UX) is widely expressed in user-generated online discourse across platforms, creating opportunities for in-the-wild measurement at scale. However, existing UX instruments and review-mining approaches do not provide a smartphone-oriented, theory-grounded hierarchical measurement specification that supports consistent aggregation and comparison across heterogeneous platforms. In this research, we propose a hierarchical smartphone UX measurement framework and an interpretable computational pipeline that translates cross-platform reviews into structured UX metrics. The pipeline extracts localized experience evidence units, maps them to the hierarchy via coarse-to-fine classification, and quantifies evaluations with a unified five-level satisfaction sentiment model. We apply the approach to a stratified subset of approximately 20,000 Chinese social media reviews covering four major smartphone brands across three platforms. The resulting metrics separate what users discuss, captured by normalized mention frequency, from how they evaluate it, captured by mean five-level sentiment scores. This work provides a scalable and interpretable evidence base for cross-brand comparison and metric-level interpretation beyond raw review volume or single-platform observations.

0.30

Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review

cs.SE ·2026-09-05 ·Haoran Yu, Lifei Liu, Danping Zhang
similarity 0.60
Abstract

Code review is a key quality checkpoint between AI-generated code and production. As AI coding agents submit pull requests at scale, it is unclear whether reviewers reduce scrutiny with repeated exposure and whether review comments reveal this change. We study 11,429 reviews from 400 repeat reviewers over 207 days, paired with 10,104 human-authored inline comments from AIDev. Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods (Wilcoxon p = 8.6 x 10^-8; Cohen's d = 0.25). However, four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles (all Spearman absolute rho <= 0.53, p >= 0.11; Bonferroni-corrected Mann-Whitney p >= 0.36). A logistic-regression classifier based on these features reaches F1 = 0.485, below the majority-class baseline. Sentence-embedding structure does carry signal: reviewers' late-period comment centroids shift farther from their early-period centroids than under within-reviewer random permutations (Wilcoxon p < 0.001), and a small MLP using three embedding statistics reaches F1 = 0.74 under 5-fold reviewer-stratified cross-validation. Granger analysis shows that approval-rate changes predict later shifts in technical specificity at all tested lags (p < 0.001), while the reverse direction is significant at only one of four lags. Reviewer adaptation is therefore detectable in latent distributional structure rather than in classical lexical metrics, and language shifts follow rather than precede changes in approval behavior.

0.23

Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics

cs.CL ·2026-09-20 ·Shakiba Amirshahi, Sajad Ebrahimi, Hai Son Le, Negar Arabzadeh, Ebrahim Bagheri
top venue similarity 0.59
Abstract · Accepted at CIKM 2026

Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high score because it is fluent, organized, and polished, rather than because it provides a strong evaluation of the paper. This risk is especially important in AI-assisted reviewing, where reviewers may use LLMs to improve clarity or presentation while preserving the underlying judgments. We propose a statistical framework for testing whether peer-review evaluation metrics capture substantive review quality beyond surface-level linguistic form. The framework compares original human reviews with faithful LLM rewrites that preserve the same evaluative content while changing wording and presentation. Using a dataset comprising 4,044 meaning-preserving rewrites derived from 674 human reviews from ICLR and NeurIPS, we evaluate 29 content-oriented peer-review evaluation metrics drawn from four prior works through complementary tests of surface sensitivity and robustness. Although these metrics are intended to capture review properties beyond surface-level, writing-dependent characteristics, we find that sensitivity to rewriting is widespread. Under our primary analysis, 23 metrics assign significantly different scores to reviews whose evaluative content is preserved, while only six satisfy our robustness criterion. The patterns are largely consistent across two LLM judge models, suggesting that the issue is not specific to a single judge. These findings show that many peer-review evaluation metrics partially conflate review quality with linguistic presentation, and indicate that robustness to meaning-preserving rewriting should be validated before such metrics are used to compare human-written, AI-assisted, and AI-generated reviews.

0.48

How Much Were You Told? Measuring External Information in Peer Reviews

cs.CL ·2026-09-23 ·Matthieu Dubois, Pablo Piantanida, François Yvon
benchmark similarity 0.59
Abstract

Conference policies distinguish using Large Language Models (LLMs) to polish one's own review from delegating the critique, but current Artificial Text Detection (ATD) methods largely measure surface form rather than the origin of its content. We instead measure the external information carried by a review: information not explained by the reviewed paper and a generic reviewing instruction. We propose Self-Conditioning, an unsupervised information-theoretic estimator that compares the likelihood of a review under its production context with its likelihood when that context is augmented with hints extracted from the review itself. On the IntelLabs peer-review benchmark, Self-Conditioning separates fully-delegated from machine-polished reviews with AUC up to $1.0$ while remaining largely insensitive to surface rewriting. Moreover, as generators receive increasing amounts of externally-provided information, their scores move monotonically towards the human regime, unlike standard ATD baselines. High-temperature sampling can evade the estimator, but at the cost of output quality.

0.09

SAGE (Subjectivity-Aware Generation Estimator): A Linguistic Framework for AI-Generated Review Detection *

cs.CL ·2026-11-02 ·Gonzalo Freijedo Aduna, Charles Yaiche, Patrick Bas, Alda Mari
benchmarkcode/data similarity 0.58
Abstract

We propose a simple, linguistically motivated approach to detecting AI-generated peer reviews based on the inherently human trait of subjectivity. We capture subjectivity at two levels: the semantic level of evaluative language and modality and the pragmatic level of speech acts. Using features derived from speech act annotations and model-predicted subjectivity scores, we train a lightweight classifier that achieves an AUROC of 0.9566 with only these linguistic features. Combining its output with the state-of-the-art detector Sem-Detect raises the TPR at 1% FPR of Sem-Detect on the APRDB benchmark from 0.874 to 0.985, outperforming the current state of the art. Our results show that linguistic features of subjectivity provide an explainable source of information for detecting AI-generated peer reviews and can be readily integrated into existing detection methods. The code and models are publicly available on GitHub (https://github.com/Arys02/generated\_ text\_detection\_hub) and Hugging Face (https://huggingface.co/Arys02), respectively.

0.39

Beyond a Single Story: Meta-Reviewing Sparse and Incomplete User-generated Contents for Recommendation

cs.IR ·2026-08-27 ·Hongren Wang, Tianjun Wei, Yingpeng Du, Jie Zhang, Yin-Leng Theng
top venue similarity 0.57
Abstract

Data sparsity remains a long-standing challenge in recommender systems, and it becomes more severe for methods relying on user-generated content (UGC) such as textual reviews, which capture fine-grained preferences but require more user efforts to produce. As a result, UGC exhibits (1) missing reviews, where interactions lack any review, and (2) incomplete reviews, where available reviews cover only a subset of relevant attributes. Existing approaches often overlook these UGC-specific issues, leading to degraded accuracy. Motivated by meta-review in academic peer review, we propose MOSAIC (Meta-review On Sparse And Incomplete user-generated Content), which constructs a meta-review for each target user by aggregating attribute-sentiment evidence from neighbor users' reviews. A multi-gate mixture-of-experts (MMoE) architecture jointly optimizes rating prediction and meta-review attribute-sentiment prediction, while an attention module personalizes the aggregated meta-review signals to each target user, yielding both refined rating predictions and attribute-level explanations. Experiments on four real-world datasets demonstrate that MOSAIC consistently outperforms state-of-the-art baselines in both recommendation accuracy and explanation quality, mitigating UGC sparsity and incompleteness while delivering consistent gains for users with limited interaction history.

0.34