CS Paper Radar

Replication Studies: Not Just a Copy

Yiheng Liang, Kim Marriott, Helen C. Purchase

cs.HC ·submitted 2026-09-16 06:13 UTC · v1

Abstract

Replication studies revisit previous experimental findings for multiple purposes, including assessing reliability, exploring generalisability, and evaluating research methods. Such studies entail design decisions that a single replication label cannot fully capture, making their designs difficult to describe and compare. We present REPVIS2, a validated design space that describes how a replication study relates to a reference study across eight practical dimensions, each coded as identical, similar, or different. We refined our initial design space, REPVIS1, by applying it to a corpus of replication studies and validated REPVIS2 against a separate corpus. We then characterised 86 replication studies from 51 papers in visualisation through paired reading of the replication and reference reports. Studies often retained the task while changing the procedure, interface, environment, participant population, evidence, or analysis. Additions beyond the replication core were also common. REPVIS2 makes replication design explicit for characterisation, reporting, and planning. An interactive visualisation and supplementary materials are available at https://replication-study.github.io/.

Closest papers

Nearest neighbours by meaning across the whole index.

RaivenTracks: Branching Provenance for Conversational Visualization Workflows

cs.HC ·2026-08-14 ·Ella Hugie, Alexandra Irger, Grace Guo, Kenneth Moreland, David Pugmire +2 more
similarity 0.61
Abstract · *Ella Hugie and Alexandra Irger are co-first authors

As AI agents increasingly participate in scientific workflows, scientists are shifting from direct authorship toward oversight, inspection, and steering. LLM-driven visualization systems are a promising interface for this hand-off, yet they remain largely stateless, forcing users to reconstruct context across refinements and offering little support for revisiting prior decisions or exploring alternatives. We present RaivenTracks, a workflow-aware extension of the Raiven DSL-mediated visualization pipeline that treats validated visualization specifications as persistent, branchable checkpoints. Because each checkpoint is a verifiable RaivenDSL specification rather than a dialogue transcript, restoring a node recompiles a known artifact rather than re-interpreting prior context. RaivenTracks contributes a two-level state management architecture that pairs a persistent, branchable version tree with a fine-grained undo/redo stack over runtime visualization settings, across both InfoVis and SciVis backends. A formative pilot study with three visualization researchers shows early promise, with all participants adopting the version tree for branching and recovery, and surfaces design directions for tree navigation, node labeling, and scalability that inform a planned controlled comparison against Raiven without version history. We frame branchable conversational visualization history as a step toward provenance support for future scientist-in-the-loop oversight of AI-driven scientific workflows.

0.04

ProLiVis 2.0: Literature-Centric Visualization of Protein--Protein Interaction Networks, with a Citation-Trust Model for Interaction Evidence

cs.IR cs.SIq-bio.MN ·2026-09-14 ·Melih Sözdinler, Yalçın Doksanbir, Gökhan Akpınar, Ege Aktan
code/data similarity 0.61
Abstract · 8 pages, 6 figures, Github Repo: https://github.com/melihsozdinler/CenterLayout, Supplement/Guide is available on repo

Protein-protein interaction databases record evidence without weighing it. In BioGRID, an interaction asserted once by a single high-throughput screen and one confirmed by twenty laboratories across a dozen assays are the same kind of row in the same file. Tools built on such databases inherit that flattening: they draw every reported interaction as an edge, and the resulting picture states that two proteins interact without stating how much anyone should believe it. We present ProLiVis 2.0, a rewrite of the literature-centric visualization system of arXiv:2111.12794. It contributes three things. First, a citation-trust model that scores each interaction from seven terms, including a term for the number of independent laboratories behind the supporting publications, obtained by clustering those publications over shared institutional affiliations; a plain count of publications cannot distinguish five confirmations from one group publishing five times. Second, a deterministic reformulation of the center layout, closed-form and $O(n \log n)$, which replaces the force-directed placement of the original and makes published figures regenerable from a session manifest. Third, an implementation that runs entirely in a web browser, with an embedded analytical database, requiring no installation and uploading no data. On BioGRID release 5.0.260 restricted to SARS-CoV-2, 24,344 of 34,540 reported interactions (70%) rest on a single publication, and raising the trust threshold to 0.2 leaves 11,320 of them. That the large majority of a curated interaction network is unreplicated is a fact no existing view of the database makes visible.

0.52

Gricea: An Open Science Platform for Conversational AI Research

cs.HC cs.AI ·2026-09-18 ·Nikhil Sharma, Yunlin Gong, Xinyang Cheng, Ziang Xiao
similarity 0.60
Abstract · 19 pages, 3 figures, 4 tables. Pre-print

We need studies on conversational AI (CAI) at scale to understand human behavior and shape CAI design. However, fragmented reporting of systems and study configurations hinders replication, extension, and knowledge accumulation. We present Gricea, an open-science platform representing studies as configurable, deployable research artifacts that researchers can run, inspect, share, and reuse. Informed by a formative analysis of prior CAI research, Gricea couples study procedures, participant-facing systems, and conversational task behavior in. In a replication study using Gricea, we replicated configurations 93% of eligible CUI 2026 papers; while also flagging missing information in 96% of papers that hinder faithful replication --- further motivating Gricea's need. In a user study, researchers and practitioners from diverse backgrounds successfully constructed runnable studies addressing various open-ended research questions. Together, these findings demonstrate Gricea's support for constructing, reproducing, and extending CAI studies through shared research artifacts, enabling cumulative knowledge building through open science.

0.27

Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill

cs.CL ·2026-08-12 ·Zhuoyang Qian, Biao Wu, Yiran Wang, Chris D Yan, Desan Dai +4 more
similarity 0.59
Abstract · 24 pages, 10 figures

Turning a research idea into a complete paper requires more than text generation: the system must retrieve literature, design and execute experiments, revise claims according to evidence, produce publication-ready figures, and maintain consistency across a long generation process. We present Spark-to-Paper, an end-to-end research paper generation system implemented as thirteen composable skills inside an existing coding assistant, without requiring a separate agent platform or orchestration service. Spark-to-Paper separates model-based judgment from deterministic operations that can be directly executed and checked. It further separates experiment planning from reporting, so that required evidence is specified before results are observed and manuscript claims are revised according to measured outcomes. To improve reliability over long research trajectories, the system combines deterministic integrity checks with self-critique and bounds a failure mode we call the Self-Refutation Loop, in which repeated experiments continue to reject the original research objective. Spark-to-Paper also produces editable vector figures through programmatic plotting for experimental results and code-based reconstruction for generated method diagrams. Across eight controlled research topics, Spark-to-Paper achieves 99.5% citation validity and 96.4% figure editability. A controlled ablation increases fabrication detection from 14% for a single-pass draft to 92% with the full integrity and review stack, while adversarial review achieves 74% precision. The full system uses 11.9M tokens, costs $8.1 per manuscript, and requires 3.2 hours on average. These results show that end-to-end research paper generation can be implemented as a lightweight, composable workflow inside existing coding assistants while keeping experimental evidence central to how claims are accepted, revised, or abandoned.

0.10

Considering Contribution Statements in Visualization and HCI Research

cs.HC ·2026-08-13 ·Mara Solen, Wesley Willett, Andrew M McNutt
similarity 0.58
Abstract

Contribution statements are an increasingly common way to make research labor visible, reduce academic malfeasance, and provide broader transparency. Despite this potential value, they remain uncommon in visualization and HCI. To explore this gap, we conducted an online study with (N=21) visualization and HCI researchers. We find a range of differing opinions about the utility of contribution statements, which are set against a background of tensions relating to contribution frameworks that inadequately fit contribution types in HCI and especially visualization, power dynamics between authors, bias in authorship perceptions, and the tedium of providing yet another form of documentation. From these factors, we offer a modest recommendation to authors: consider contribution statements. There are contexts when they may usefully explicate work, and others where they can cause author-team conflict or become a burdensome chore. Regardless of whether they are used by readers or not, we suggest that scaffolded mechanisms for reflecting on contribution roles are valuable both for public accountability and internal alignment. To institutions, we recommend that contribution statements be exempt from page or word limits, and that flexible templates and examples be provided to authors, but that they continue to not be required. By surfacing these perspectives, we seek to open a dialogue about what constitutes authorship, how our community might move toward more equitable and transparent attribution practices, and where the visualization and HCI communities might be uniquely equipped to help.

0.04

Do the Numbers Match the Plot? Statistical-Visual Verification for Automated Exploratory Data Analysis

cs.HC cs.CV ·2026-08-12 ·Zixun Fu, Jian Huang, Maojun Sun, Yancheng Yuan
similarity 0.58
Abstract

Automated exploratory data analysis (EDA) systems can execute statistics correctly while producing misleading interpretations. For example, correlations may be driven by influential observations, and mean differences may arise from skewness, overlap, or sparse subgroups rather than broad distributional shifts. We present EDAgent, a sequential framework that performs targeted visual verification before report synthesis. For each finding, the system records the statistical result, generates a visualization request identifying relevant distributional features, creates the corresponding chart, and uses the observed pattern to retain, qualify, or reconsider the interpretation. We call this process statistical-visual verification. EDAgent includes VisPlan, an 8B-parameter model that converts visualization rationales into detailed chart plans. We evaluate 87 end-to-end reports on EDA-Bench using a six-dimension rubric and GPT-5.1 multimodal evaluation. Maximum-score proportions reached 87.4% for factuality and visual consistency, 93.1% for statistical-visual cross-validation, and 96.6% for analytical narrative. On 166 open-ended VisEval tasks, VisPlan improved the mean overall score from 4.55 (Qwen3-8B) to 4.62, with the largest gain in aesthetics. These descriptive results indicate that visual verification can make automated EDA reports more transparent about how statistical findings relate to underlying data structure.

0.04

Projecting the Public: Reflections and Recommendations for Empirical Visualization Studies of General Audiences

cs.HC ·2026-10-31 ·Mandi Cai, Jack McNiel, Erik C. Nisbet, Matthew Kay
similarity 0.57
Abstract

Calls for shifting visualization research to general audiences outside of professional contexts have been put forward since the early 2000s, but expanding outwards means contending with a diversity in visualization's communicative purpose beyond transmitting insights from data. After our own experiences studying public-facing visualizations, we sought a set of considerations to help early-career visualization researchers reason through designing empirical visualization studies for the general public that confronts the various ways visualizations are interpreted. We examined literature on guiding paradigms of research, reviewed 73 visualization papers that specify a focus on audiences broader than domain experts, and conducted 15 semi-structured interviews with authors of such papers, to derive the following: (1) a conceptualization of any experiment as 3 elements-casting audience, constructing experimental setting, and projecting audience impressions and interactions into measurable constructs (2) an account of interviewees' aspirations for each element of study design, versus the real challenges that occurred. We detail their approaches to navigate the gaps between aspirations and reality when working with general audiences and summarize them as actionable recommendations. The conceptual framework is intended to help junior researchers scope and outline experimental designs of visualizations "for the masses", while the recommendations provide concrete guidance for putting those designs into practice. The 73 papers are filterable and visualized on our study website, and our code to sample the papers is on GitHub.

0.23

How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices

cs.AI ·2026-10-08 ·Liulei Zhang, Dejing Zhou, Chuyue Huang, Guanhua Chen, Yutong Yao +3 more
publishedsurvey similarity 0.57
Abstract · Accepted to Findings of AACL-IJCNLP 2026

Automated research systems support literature synthesis, ideation, experiments, writing, and peer review, but their evaluation is dispersed across tasks, benchmarks, and studies that are difficult to compare directly. We review this literature from the perspective of evaluation design and evidence, covering six targets: literature synthesis, research ideation, executable workflows, scholarly writing and communication, automatic peer review, and end-to-end research. We compare task construction, evidence sources, evaluators, and scoring procedures to explain the capabilities assessed by different designs. Our synthesis highlights three recurring lessons: output checks, process checks, and human studies provide complementary information; evaluator calibration is specific to the property being assessed; and resource budgets and attempt selection are integral to interpreting performance comparisons. We identify diagnostic evaluation designs and documented gaps in supporting evidence, and translate these comparisons into reporting and audit recommendations for specific evaluation settings. The survey helps readers navigate existing evaluations, select appropriate benchmarks, and design subsequent studies.

0.24