CS Paper Radar

Understanding Suggested Edits in Stack Overflow: A Large-Scale Empirical Study and Review Prioritization Framework

Mehedi Hasan Shanto, Muhammad Asaduzzaman, Md. Ahasanuzzaman, Alioune Ngom

cs.SE ·submitted 2026-11-11 03:59 UTC · v1

Abstract

Suggested edits allow low-reputation users to improve Stack Overflow posts and contribute to the platform's knowledge base. However, we still know little about why these edits are accepted or rejected and how reviewers can efficiently identify problematic edits. In this study, we analyze 3,888,805 suggested edits submitted between 2011 and May 2023. First, we compare accepted and rejected edits using 35 features. We find significant differences between them, with the largest differences related to users' editing history. Second, we manually analyze rejected edits and identify 17 fine-grained rejection reasons. These reasons provide a more detailed understanding of why edits are rejected than Stack Overflow's predefined rejection reasons. Third, we evaluate three models for prioritizing edits based on their likelihood of rejection. The Feature Model, built using the 35 studied features, provides the best trade-off between review rate and rejected-edit recall. At a decision threshold of $τ= 0.40$, it prioritizes 52.3% of edits for human review while capturing 80.8% of rejected edits. In comparison, the Embedding Model captures 78.8% of rejected edits while prioritizing 60.9% of edits, and the EditEx Model captures 72.5% while prioritizing 59.0%. We also conduct a complementary zero-shot evaluation on a separate set of 500 edits. The Feature Model achieves 78.0% recall at $τ= 0.40$, compared with 23.6%-31.2% for GPT-4o, Qwen2.5-32B, and Llama-3.3-70B. Overall, our findings provide a detailed understanding of suggested-edit rejection and show that rejection-risk prediction can help prioritize potentially problematic edits for human review.

Comment
45 pages

Closest papers

Nearest neighbours by meaning across the whole index.

Edit Recommendation: Can LLMs Anticipate Edit Requests from Users, and When?

cs.CL cs.AIcs.HC ·2026-11-16 ·Daisuke Kikuta
benchmarkcode/data similarity 0.66
Abstract · Blog: https://ntt-dkiku.github.io/llm-edit-recommendation, Code: https://github.com/ntt-dkiku/llm-edit-recommendation

After a large language model (LLM) agent generates an artifact, users often send edit requests for it. This paper studies whether an LLM can anticipate these requests before they are sent. Such anticipation is useful in practice because it reduces the effort of user review and feedback. We formalize this task as edit recommendation and build EditRecBench, a benchmark for evaluating the performance of LLMs on this task, from real-world conversation logs in WildChat-4.8M. Through benchmarking and failure analysis, we identify a practical challenge that some cases are inherently difficult to predict in real-world settings. To address this, we introduce decision model-based gating to filter out cases where predictions are unlikely to be correct before they are presented to the user. Our trained model outperforms existing decision models and raises pass@5 among shown suggestions by 12--25 points at a gating threshold. Our code is available at https://github.com/ntt-dkiku/llm-edit-recommendation.

0.35

Edit Discovery in Human-AI Interactions

cs.LG ·2026-10-15 ·Alireza Mirrokni, Shayan Kiyani, George Pappas, Hamed Hassani
similarity 0.63
Abstract

Human-AI interactions are becoming a rich source of data for improving AI systems: while many model responses are already satisfactory, others expose failures that could become valuable supervision if identified and corrected. We study this setting through the notion of an editor: a more capable evaluation process that can inspect a response and either confirm it or modify it, but whose invocation is costly. The editor may be a human expert, a stronger AI system, or the same model equipped with additional computation, context, tools, or information. The key challenge is to decide which interactions to send to the editor before knowing whether an edit will actually be made, since invoking the editor itself consumes limited resources such as expert time, inference cost, latency, or tool-use overhead. We formulate this problem as Edit Discovery Optimization (EDO), which seeks to minimize edit opportunities missed among skipped interactions while controlling unnecessary confirmations and respecting an editor-cost budget. We characterize the population-optimal policy and show that it takes the form of a cost-adjusted threshold on the posterior probability of an edit. Building on this structure, we develop EDIT (Edit Discovery via Iterative Thresholding), an online algorithm for the partial-feedback setting in which edit labels are revealed only for selected interactions. EDIT combines an adaptive edit score, a confirmation-control threshold, and a cost-aware selection rule; it satisfies the editor budget at every round and controls the average confirmation rate up to a vanishing finite-horizon term, without distributional assumptions on the interaction stream. Across four reasoning and factual question-answering benchmarks, EDIT achieves a better trade-off between editor cost, Type-I error, and Type-II error than direct LLM-based batch selection.

0.29

Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

cs.DL cs.AIstat.AP ·2026-10-14 ·Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman
similarity 0.62
Abstract

Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22.

0.20

Ranking Agreement Does Not Reveal Paragraph-Edit Agreement in Two Released Paper Acceptance Predictors

cs.CL cs.LG ·2026-11-04 ·Shaoliang Yang, Jun Wang
hot direction similarity 0.62
Abstract · 38 pages including supplementary methods and results; 10 figures and 22 tables

A fast acceptance classifier could select revisions for a slower evaluator. We ask whether agreement on existing papers supports that use. We compare a classifier of word and document features with two released text-only acceptance predictors fine-tuned from the same language model with three billion parameters. For 198 papers from the International Conference on Learning Representations, we compare model rankings of the original papers and of paragraph rewrites screened automatically for unchanged facts, claim strength, and attribution. The classifier ranks original papers closely with one predictor and moderately with the other, but shows no detectable agreement with either on which rewrites raise scores. The tests specified in advance used reduced numerical precision. Rounding error dominated the small score responses. We therefore rescored the released weights in full 32-bit precision as a post hoc check; edit agreement remained near zero. At this precision, when each predictor chooses among five rewrites per paper, its average raw score is under two points out of 100 higher than with random choice. The evidence covers presentation rewrites and two open predictors from one family; transfer to costly evaluators remains untested. Revision selectors should therefore be checked against the intended evaluator on the revisions they rank.

0.31

Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories

cs.CL ·2026-11-01 ·Yaochen Wang, Chenguang Wang, Ming Li, Haomin Zhuang, Xinyue Zeng +6 more
similarity 0.58
Abstract · 43 pages including appendices, 8 figures

AI reviewers are increasingly used to assess scientific manuscripts, yet it remains unclear whether their judgments reflect the improvements authors make through revision. We investigate this question using PaperTrajectory-2026, the 2026 release of PaperTrajectory, a dataset of 6,959 paper trajectories across five AI conferences, with automatic annotations of changes along 14 dimensions and separate judgments of overall quality. The annotations show that major improvements emphasize empirical evidence and framing, while later revisions tend toward incremental improvement. Using independent assessments of manuscript versions from eight AI reviewers, we examine whether annotated major improvements receive higher scores, which revision dimensions are associated with score changes, and whether user opinions can outweigh observed revision gains. Every reviewer leaves scores unchanged on most major improvements, and agreement on the direction of score changes is weak. Revision dimensions show some reviewer-specific associations with score increases, but provide limited predictive discrimination and no consistent guide across reviewers. Even when a revision earns a higher neutral score, a brief negative user opinion can erase or reverse that advantage, with substantial variation across reviewers. These findings show that substantive revisions are not consistently reflected in review scores and that observed scoring gains can be vulnerable to user framing, motivating assessments that connect manuscript evidence to overall judgments and remain stable when no scientific evidence changes.

0.33

AppliedScientist: Automated Scientific Revision Through Iterative AI Reviewing

cs.AI ·2026-09-13 ·Vidushee Vats, Karun Sharma, Shengzhi Li, Shichao Pei
similarity 0.58
Abstract · v2: Updated Table 3

Automated reviewing systems are increasingly evaluated based on the quality of the reviews they produce. Yet a review is only useful if acting on it leads to a measurable improvement in the paper. We present AppliedScientist, a closed-loop system that couples an autonomous AI scientist with an AI reviewer, and evaluate it by iteratively revising rejected papers from a range of research subfields. To mirror how human authors build on earlier drafts, the AI scientist has access to its previous versions during revision. To avoid bias from prior judgments, however, each review is generated independently, with the reviewer having no memory of earlier feedback or scores. We compare three revision settings: one initialized with the original venue reviews, one initialized with AI-generated reviews, and autonomous self-revision using the same fixed prompt in every round. Because the reviewer both guides and evaluates the revision, we also assess the human-initialized revisions using Stanford Reviewer as an independent evaluator. Reviewer-guided revision consistently improves more than fixed-prompt self-revision, and Stanford Reviewer also assigns higher scores to later revisions. AppliedScientist resolves 128 of 150 execution-related weaknesses (85.3%), but only 2 of 18 idea-related weaknesses (11.1%), suggesting that iterative revision is effective at improving experiments and implementation, but rarely changes concerns about novelty or significance.

0.24

From Counterfactual Edits to Human Narratives: LLM-Driven Interpretations for Recommender System Explanations

cs.AI cs.IR ·2026-11-10 ·Vojtech Cahlik, Amir Reza Mohammadi, Rodrigo Alves, Pavel Kordik, Eva Zangerle
top venuepositioncode/data similarity 0.58
Abstract · Published at CIKM 2026 (Rome, Italy). Code: https://github.com/vcahlik/from-counterfactual-edits-to-human-narratives

Recommender systems increasingly influence what users discover and consume online, yet the decision logic behind their outputs often remains opaque. Counterfactual explanation (CFX) methods provide an effective way to show which changes to a user's interaction history would alter a recommendation. In recommendation settings, however, CFXs often require many edits, producing long, low-level lists that are hard for users to interpret. Moreover, most evaluations emphasize edit counts while largely overlooking whether the resulting CFXs are coherent and actionable for users. We argue that this gap between generating valid CFXs and communicating model behavior stems from the lack of user-centered abstraction and limited qualitative evaluation. To bridge this gap, we investigate large language models as mediators that summarize raw CFX interaction edit lists into concise natural language explanations describing higher-level preference patterns. We optimize these mediators for correctness, informativeness, and linguistic quality through preference optimization, and evaluate the resulting explanations with human-calibrated metrics derived from separate human-labeled datasets. On the MovieLens-1M dataset, across multiple counterfactual generators and language models, the resulting summaries achieve high overall linguistic quality, while the strongest semantic results are obtained with high-success CFX methods such as ACCENT and SPINRec. Direct Preference Optimization improves the semantic metrics in almost all evaluated settings.

0.35

Actionable Feedback: Improving Model Capabilities from User Edit Data on Clinical Documentation

cs.CL cs.AIcs.LG ·2026-10-14 ·Daniel P. Jeong, Yewon Byun, John Giorgi, Peng Xu, Zachary C. Lipton +1 more
similarity 0.57
Abstract · Preprint

Large language models (LLMs) are increasingly being deployed in clinical documentation workflows, where clinicians review and edit their outputs before use. In principle, these clinician edits represent a scalable source of supervision for model training. In practice, not all edits represent actionable feedback: for instance, on data drawn from real-world deployment of an AI clinical documentation product, our review suggests that over 45% of edited outputs contain user-added information that is not directly supported by the inputs to the model. Naively fine-tuning on such edits risks teaching the model to hallucinate. In this paper, we develop a method to address this challenge. Given that real clinician edits on deployed documentation tools are not publicly available, we first develop a taxonomy of clinician-editing behaviors, informed by data from a deployed AI clinical documentation product. Then, we develop a method for generating semi-synthetic clinician-edit data, informed by our taxonomy, on a discharge-summary generation task based on MIMIC-III, a public dataset. Finally, we introduce simple filtering methods for identifying actionable edits, with accuracy evaluated by expert review. We show empirically, on our semi-synthetic dataset, that naive fine-tuning on raw edited output increases ungrounded content, while our approach of progressively filtering toward grounded edits enables improvement on clinician-alignment metrics while retaining faithfulness to the input context.

0.23