CS Paper Radar

Who Pays for Open Review? Visible Author Reputation and Its Effect on Ratings

Qinghua Zhao, Xinyu Chen, Yanhui Yang, Tengfeng Sun, Junfeng Liu, Zhongfeng Kang

cs.DL cs.AI ·submitted 2026-09-07 05:16 UTC · v1

Abstract

An OpenReview bug in November 2025 broke anonymity at several conferences and prompted calls for open review, which motivate us to ask what shifting from blind to open would mean for authors. Analyzing over 18,000 reviewed submissions to ICLR 2026, split into de facto open and blind groups by arXiv preprint timing, we find that ratings rise with author reputation under both mechanisms, with a steeper slope under open review that is statistically significant, and that the open-blind difference is concentrated at the borderline ratings. The pattern holds across five reputation proxies (including institution, h-index, and citation count), three author-aggregation rules, and five definitions of the open window. A controlled simulation with five AI models as reviewers, holding the manuscript fixed and varying the author reputation, reproduces the effect. With claude-opus-5 as the reviewer, for example, rating rises by 0.5 points as the author moves from low to high reputation.

Comment
11 pages, 6 Tables

Closest papers

Nearest neighbours by meaning across the whole index.

Paying for Publication with Peer Review Design and Simulation of a Fee-Free, Credit-Based Open Access Journal for Information Technology

cs.DL ·2026-09-27 ·Murat Ozer
similarity 0.61
Abstract · 14 pages and 5 figures

Open access publishing has moved much of the cost of scholarly communication from readers to authors, while the labor of peer review remains unpaid and unevenly shared. This paper describes the design of Quorum: An Open Journal of Information Technology (jquorum.org), a journal that charges neither readers nor authors and uses peer review as its currency instead. Every first complete review a scholar writes on a manuscript earns one credit, and one credit pays for one submission. Each manuscript receives three double-blind reviews from reviewers matched automatically by expertise, and editorial decisions follow published rules rather than editor discretion. The paper documents the credit ledger, a bootstrapping mechanism called the genesis block, a reviewer search that widens after seven days and opens the paper to volunteers, the decision rules, and the integrity safeguards. An agent-based simulation, run for two simulated years with 40 replications per scenario, evaluates these choices. Without the genesis block, no paper was submitted in any replication. With it, a community growing from 40 to about 330 members submitted about 603 papers, and papers waited a median of 7 days for a full review panel. The day-7 escalation had a modest effect in a large community but a decisive one in a small community of about 60 members, where removing it left 48.8% of papers without a panel after 60 days, compared with 4.6% with it. Unspent credits also accumulated to about 3.7 per member after two years, which points to a policy choice the journal will face as it grows.

0.27

When Reviews Gate Access: Collusion, Sabotage and the Scale of Peer Review in the AI Submission Surge

cs.MA ·2026-11-14 ·Yuchen Li, Xintong Li, Ken-Tye Yong, Nguyen H. Tran
similarity 0.61
Abstract

AI tools have helped push submissions to the largest machine-learning venues to tens of thousands per cycle. Venues have answered with rules that tie access to review: per-author caps, mandatory reciprocal reviewing, reviewer eligibility earned through accepted papers, and assignment based on reviewer bids. Under these rules a reviewer is also a competitor of the authors. We study two deviations that the rules make attractive. Sabotage lowers a competitor's score. Collusion raises an ally's score. We build an agent-based conference calibrated to public statistics of NeurIPS and ICLR and run 6,220 simulated conferences. Four results stand out. First, the two deviations do different harm and amplify each other. Collusion changes who is accepted, sabotage erodes what scores say about quality, and rings that may also sabotage gain more than rings that only collude. Second, under random assignment the chance that collusion changes a decision is set by one exposure index, $E = Rm/A$, the expected number of allies among a paper's reviewers. Growth therefore protects random assignment, but bidding removes the protection. In a pool of 3,000 reviewers a 20-member ring turns 0.8% of its papers into acceptances under random assignment and 71% when 30% of allies' bids are granted. Third, when agents choose their allies every cycle, caps at the inner layers of the social brain keep collusion nearly harmless. Without a cap agents ally with half of their area and accepted value falls by 35%. Fourth, reviewer eligibility tied to acceptances turns sabotage into a weapon of incumbents, and a longer eligibility window widens the gap between incumbents and newcomers only when sabotage is possible.

0.18

ECHO: Revealing Review-to-Submission Linkage in Anonymous Peer Review

cs.CR ·2026-10-19 ·Rong Wang, Mulong Luo
benchmarkcode/data similarity 0.61
Abstract · 14 pages, 3 figures

Anonymous peer review protects reviewers from retaliation, yet reviewer anonymity may also be weakened by artifacts released through the review process itself. This risk is growing as many venues require submitting authors to review, creating substantial overlap between authors and reviewers within the same review cycle. We identify review-to-submission linkage, where an anonymous review can be associated with the reviewer's own submission, substantially shrinking the reviewer anonymity set. To assess whether this linkage is practically exploitable, we develop ECHO, which combines technical-expertise and writing-style signals across reviews, papers, and rebuttals, and evaluate it on R2S-Bench, a ground-truth benchmark we construct. ECHO correctly identifies the reviewer's own submission in Top-1 for 34.6% of reviews, in the Top-5 for 54.0%, and in the Top-10 for 63.4%, compared with 0.5%, 2.6%, and 5.3% under random ranking, from the candidate submissions. With a stronger Stage-2 reranker, ECHO further reaches 56.4% Top-1 and 66.4% Top-10. These results demonstrate that review-to-submission linkage is feasible using only routinely released peer-review artifacts. Code and benchmark are available at https://github.com/kaimu11111/ECHO-Revealing-Review-to-Submission-Linkage-in-Anonymous-Peer-Review.

0.47

Why LLM Agents Favor Their Group: Stakes, Observed Norms, and Reputation

cs.AI physics.soc-ph ·2026-10-07 ·Yujiao Chen
similarity 0.58
Abstract

Language-model agents favor their own group because they have watched their members favor each other. The group label alone does little once the decision has a cost; what drives favoritism is observed behavior, and an individual's own record can override it. We test this in small societies with arbitrary group labels, ten rounds of point sharing, and matched one-shot decisions across fifteen OpenAI models and three Claude models, about 4,400 societies and 3.3 million audited model calls. First, the large effect of a bare group label reported in earlier work appears only when giving others points costs the agent nothing; once the agent can keep points for itself, that effect collapses on every model that shows it. Second, under a stake, interaction history becomes the main source of favoritism: the history effect is statistically positive on 13 of 15 models, reaches about 3.5-8 points out of 10 on 11, grows with the number of rounds played, and extends to labeled strangers the agent has never met. Third, with scripted histories, favoritism falls to near zero under an egalitarian norm and reverses when the agent's own group is seen favoring the other side; stronger models side with an individual's record when it conflicts with the group. Group favoritism is thus conformity to observed group behavior, carried to strangers by the label and overridden by individual reputation. The same account predicts responses to betrayal, scandal, and a free offer to change group: public reprimand repairs betrayal better than apology or restitution, allocation punishment remains confined to the offending member, and a formed group cannot be bought but can, on weaker models, be invited away.

0.10

Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer Review

cs.CL cs.AIcs.CYcs.DL+1 ·2026-11-16 ·Jizhou Guo
published similarity 0.57
Abstract · 19 pages, 5 figures. Results of the registered tests that need the ICLR 2027 decisions will be added after 16 December 2026

Venues and benchmarks read LLM review scores as measurements of a paper, while providers replace their models within months. We audit twenty frontier models from OpenAI, Google and Anthropic on ICLR 2027 submissions while they were under review, with tests registered before the decisions. Between an earlier and a later version of Claude Opus and of Gemini, the share of papers placed on the accept side falls by 52 and 53 percentage points on 183 and 60 randomly drawn submissions, and by 43 to 79 points in six further paired comparisons, with no paper moving up in any of them. Stating a 30% acceptance rate in the prompt leaves a gap of 79 points between the two Opus versions. When score-stripped critiques are swapped between versions, the later critiques alone lower the accept share by 35 points for Opus and 51 for Gemini; for Opus the drop is 35 points under a third-party scorer and 24 after a rewrite that keeps each technical point and removes evaluative wording. On 57 papers, the percentiles of Opus 4.6 and 5.5 correlate at 0.73 while their accept shares differ by 44 points. On 99 to 119 submissions, the self-estimated percentiles of three models correlate with two human reviewers more strongly than a third reviewer does (point estimates 0.17 to 0.21, 95% intervals between -0.04 and +0.42). All twenty models give a 2 or a 3 in 473 of their 475 reviews, against 60 of 72 human reviews. Retrospective tests on decided papers can draw on memory. Claude Opus 5.5 names the true first author of 27 decided ICLR 2026 papers from their titles alone, with no wrong name, and Claude Opus 4.6 accepted all 11 decided papers it said it recognized, against 5 expected. Thresholds on LLM-reviewer scores therefore have to be validated against human decisions for each model version; our tests against the ICLR 2027 decisions follow on 16 December 2026.

0.43

Can We Believe AI Peer Review?

cs.AI ·2026-10-14 ·Zongrui Yang, Haoyuan Li, Zhongsheng Wang, Gillian Dobbie, Jiamou Liu
similarity 0.57
Abstract · 8Pages and Long appendix.(include supplementary material)

As large language models become widely used in scientific review and manuscript revision, AI scores are increasingly serving as feedback for assessing paper quality and guiding edits. Yet whether a single AI score reliably reflects the reviewer's judgment, and whether revision gains based on that score are endorsed by other reviewers, remain understudied. Using 200 papers from ICLR 2025 and 2026 and approximately 14,000 AI reviews, we study score reliability, score-distribution estimation, and cross-model agreement on revision gains. Repeated reviews fluctuate even under identical conditions. Finer-grained scoring protocols reduce score variance but also systematically shift mean scores, while the gains assigned to manuscript revisions depend strongly on the reviewer.

0.34

Winning Reviews, Losing Discovery? The Co-Evolutionary Game of AI Research and Peer Review

cs.GT econ.GN ·2026-11-07 ·Hongze Liu, Mengru Wang, Zhiyuan Zhang, Yao Liu
similarity 0.55
Abstract

Autonomous research systems can produce convincing manuscripts whose scientific value is difficult to verify, while automated reviewers can reward the appearance of completeness. We examine how this gap can become a self-reinforcing threat to the research community. In a two-population evolutionary game, authors choose between acceptance-oriented and exploratory research, and reviewers choose between proxy evaluation and careful assessment. Cheaper acceptance-oriented production can increase review pressure, making proxy assessment more attractive; proxy assessment can in turn strengthen the incentive to produce reviewer-compatible work. We derive conditions under which this feedback sustains a stable regime that rewards publication success while reducing the scientific value of accepted work. An analytical basin boundary explains why the same institutional rules can support either this regime or exploratory research with careful review. Numerical illustrations show how publication counts can conceal scientific loss, how faster adjustment can reinforce existing practices, and why partial reform may fail once the cycle is established. Careful audits and reviewer rewards can reverse the feedback, while temporary support can produce lasting recovery if it moves the community across the basin boundary. The model identifies a warning mechanism rather than forecasting a particular conference: a review system can remain operational and privately rewarding while becoming less effective at supporting discovery. Its institutional implication is to reward independently validated contributions and consequential checking, rather than accelerate paper production and scoring alone.

0.06

CABAL: Multi-Agent Simulacra for Tracing the Effects of Collusive Bidding in Peer Review

cs.AI cs.CY ·2026-09-04 ·Jicheng Zhou, Kemou Li, Kahim Wong, Zheyuan Li, Zhuan Shi +3 more
hot direction similarity 0.55
Abstract

Recent reports during the AAAI-27 review cycle highlight the risk of reviewers coordinating bids for reciprocal assignment advantage. Prior work treats bidding, reviewer assignment, and review manipulation as separate stages, leaving the lifecycle effects of collusive bidding unclear. Real-world analysis is further constrained by typically unobservable collusive intent and the lack of counterfactuals for the same conference. Motivated by this gap, we introduce \alg, an end-to-end multi-agent simulacra framework for studying reviewer assignment integrity by holding the conference environment fixed and configuring LLM-driven reviewer agents with honest or collusive policies. We further develop an affinity-guided collusive bidding strategy that uses mutual reviewer-paper affinities to construct collusion rings and select target papers, producing expertise-consistent rather than arbitrarily targeted attacks. Controlled experiments show that collusive bidding more than doubles target-paper capture and that assigned colluders score target papers about two points higher than honest co-reviewers, while conference-wide effects remain comparatively modest. Evaluated bid-phase detectors provide only limited evidence of collusion: in a fixed-triplet detector stress test, native positive-bid graphs are confounded by benign affinity, while a Very-High-only diagnostic view enables precise but low-coverage local recovery.

0.49