CS Paper Radar

A Meta-Study on Replication Papers in Usable Security & Privacy

Christian Mack, Benjamin Berens, Hanna Algedri, Tobias Hilt, Daniela Reimer, Peter Mayer, Melanie Volkamer

cs.CR ·submitted 2026-08-20 14:39 UTC · v1
survey

Abstract

The field of usable security and privacy research is a young and expanding field, which is still developing standards for its research, e.g. regarding replications. We used a mixed-method approach, in order to get a better understanding of the current state of replications in the field of usable security and privacy: (1) we examine the Call for Papers of 13 venues spanning security, privacy, and human-computer interaction; (2) we conduct a systematic search for papers reporting replicated user studies published across these venues between 2016 and 2025, yielding 24 relevant publications; (3) we categorized these 24 papers employing the replication taxonomy proposed by Olszewski et al. (2025); (4) we distributed a survey to the authors of these papers to understand their motivations for conducting replications. Our analysis reveals four key insights: (A) Calls for Papers would benefit from clearer guidelines for authors and reviewers regarding replication work; (B) determining what modifications were made relative to the original study proves difficult when reading replication papers; (C) strict exact replications do not exist in our sample. Approximately two-thirds of the 24 studies altered multiple aspects of the original work; (D) temporal and contextual changes affecting results emerged as one of the most frequently cited motivations for replication. Based on these findings, we offer practical recommendations for venues, researchers, and peer reviewers to strengthen replication practices in usable security and privacy research.

Closest papers

Nearest neighbours by meaning across the whole index.

Public, Identifiable, and Effectively Immutable: Ethical Lessons from Research on arXiv

cs.CR ·2026-11-09 ·Jan Pennekamp, Johannes Lohmöller, Peter Mayer, Arjun Arunasalam
published similarity 0.58
Abstract · 11 pages, accepted at the 19th Workshop on Cyber Security Experimentation and Test (CSET 26), co-located with the 42nd Annual Computer Secur…

Ethical decision-making is central to safe cybersecurity research. Recent studies, one of which is our own, analyzing the source files of arXiv papers bring this challenge to light - arXiv publicly distributes the source files, to the surprise of many authors. Unique aspects reveal ethical challenges and dilemmas that extend beyond established ethical frameworks, institutional processes, and community best practices. In this work, we examine this case and provide insights for similar research on other public live resources, such as Zenodo or the Wayback Machine. While such empirical studies are particularly realistic, they must account for personally identifiable information, leaked secrets, and other sensitive details that could expose, shame, or otherwise negatively affect individuals, possibly leading to both reputational and practical harm. sDrawing on our collective experience and discussions among researchers, affected authors, and readers, we make three contributions: First, we define the ethical setting and highlight that even intentionally-public data with deliberate real-name attribution can raise ethical concerns when repurposed for empirical analysis. Second, we discuss the ethical issues surrounding the use of arXiv data and derive four higher-level dilemmas that researchers may face when working with similar data. Third, we derive practical guidance for ethically navigating such settings and, in doing so, distill four lessons for research on arXiv and similar public, live datasets: public availability should serve as the starting point of ethical reasoning; research should minimize the amplification it induces on exposed information; disclosure should be calibrated to the harm, confidence, and actionability of a finding; and effective immutability must be accounted for. Together, these contribute to community resources for ethically-sound cybersecurity experiments.

0.49

SoK: From Generation to Consumption of Privacy Documents in Software Systems

cs.CR cs.CL ·2026-08-12 ·Shidong Pan, Clark LaChance, Zhen Tao, Sepideh Ghanavati
top venuesurvey similarity 0.56
Abstract · This SoK paper has been accepted by NDSS 2027

Privacy documents (e.g., privacy policies) are a central mechanism through which digital services disclose data practices and seek user consent. Over the past decades, research on privacy documents has expanded significantly, encompassing not only traditional privacy policies but also short notices (e.g., privacy labels) and interface-level transparency mechanisms. As this research area continues to grow, it has become increasingly difficult to obtain a coherent view of how privacy documents are created, analyzed, evaluated, and maintained across their lifecycle. This SoK provides a unified, lifecycle-oriented view of privacy documents from a software engineering perspective. We systematically review and analyze 290 papers published between 2010 and 2025, organizing them around five research questions that examine how privacy documents are (1) defined and scoped, (2) generated, (3) analyzed and extracted, (4) checked for inconsistencies and noncompliance, and (5) evaluated and improved for usability. Building on our findings, we identify 15 key research trends and 21 open opportunities. We further chart four broader research directions that highlight (i) emerging challenges in AI-centric platforms, (ii) the need for diverse and up-to-date data foundations, (iii) LLM-based unified policy-code analysis, and (iv) dual usability for end-users and developers. We hope this SoK provides a shared foundation for future research on privacy policies and privacy documents.

0.64

Protection Is Not One Event: A Survey of RAG Protection Research by Measured Endpoint

cs.CR cs.CLcs.IR ·2026-10-21 ·Peiyang Liu, Xi Wang, Di Liang, Youru Li, Zhi Cai +1 more
top venuesurveydataset similarity 0.56
Abstract · ACL Rolling Review (August 2026 cycle) recommends that the paper be accepted at an *ACL conference

Retrieval-augmented generation (RAG) systems face security, privacy, and copyright risks including content leakage, source reconstruction, unauthorized access, data poisoning, and prompt injection. Yet RAG protection studies use incompatible evaluation metrics, datasets, and threat models. We present a systematic survey and human-coded evidence map of 118 full-text-confirmed RAG defense papers, organized by measured endpoint rather than claimed goal or defense mechanism. Our seven-family taxonomy covers content disclosure, formal privacy proxies, access control and suppression, corpus-use detection, and pipeline integrity and attribution. Thirty-two annotators double-coded the papers kappa=0.867. The survey identifies three gaps. First, retrieval reranking and context rewriting each span five or six endpoint families, so mechanisms do not identify protection outcomes. Second, only 17.8% of studies use content disclosure or reconstruction as their primary endpoint; under the frozen copyright/ownership relation label, 73.3% of records have corpus-use detection as their primary endpoint, and none have an emitted-expression primary endpoint. Third, most evaluations use general-purpose question-answering datasets and non-adaptive attacks rather than sensitive data and adaptive, multi-turn extraction. Our controlled RAG testbed shows that preventing verbatim copying does not prevent private-fact recovery: a normalized 8-gram guard reduces 20-token copying to 0.0%, but a three-query adaptive attacker recovers 48.3% of hidden facts on Qwen3-8B. We release the taxonomy, dataset, coding protocols, and testbed to enable comparable, claim-aligned RAG security evaluation.

0.45

Enhancing Privacy, Neglecting Harms: An Analysis of Real-World Digital Privacy Incidents

cs.CR cs.CY ·2026-09-07 ·Shannon Veitch, C. Shem, Lena Csomor, Oleksandr Dudiy, Naone Kim +5 more
similarity 0.55
Abstract

Privacy-enhancing technologies (PETs) have emerged as a technical means for providing individuals with greater control over their information. Yet despite the growing deployment of PETs, people continue to experience privacy harms. In this work, we revisit our understanding of privacy incidents and the realities of those experiencing privacy harms, to assess whether the goals and abilities of PETs are misaligned with the harms people face. For our study, we collect news articles that correspond to a sample of 257 real-world privacy incidents. We employ content analysis over the articles to develop a new information flow model that encompasses the complexity of data flows and their relation to resulting harms. We demonstrate that our model captures both established and novel aspects of privacy incidents and their mitigations. In particular, it captures why consent is often insufficient to prevent privacy violations, how harms emerge from complex interactions among multiple entities and actions, and reveals a flaw in our understanding of PETs: a focus on enabling functionalities still permits the harms inherent in those functionalities. Moreover, we find that the entities best positioned to implement harm-preventing measures for the incidents in our sample are the least incentivized to do so. Overall, our model and analysis identify limitations of privacy technology research for harm prevention and further identifies paths for transforming how we approach the advancement of these technologies.

0.30

Replication Studies: Not Just a Copy

cs.HC ·2026-09-16 ·Yiheng Liang, Kim Marriott, Helen C. Purchase
similarity 0.55
Abstract

Replication studies revisit previous experimental findings for multiple purposes, including assessing reliability, exploring generalisability, and evaluating research methods. Such studies entail design decisions that a single replication label cannot fully capture, making their designs difficult to describe and compare. We present REPVIS2, a validated design space that describes how a replication study relates to a reference study across eight practical dimensions, each coded as identical, similar, or different. We refined our initial design space, REPVIS1, by applying it to a corpus of replication studies and validated REPVIS2 against a separate corpus. We then characterised 86 replication studies from 51 papers in visualisation through paired reading of the replication and reference reports. Studies often retained the task while changing the procedure, interface, environment, participant population, evidence, or analysis. Additions beyond the replication core were also common. REPVIS2 makes replication design explicit for characterisation, reporting, and planning. An interactive visualisation and supplementary materials are available at https://replication-study.github.io/.

0.31

Available but Not Usable: Dark Patterns and Interaction Cost in Social Media Privacy and Safety Settings for Teens

cs.HC ·2026-09-23 ·Jingxin Dong, Lingyun Chen, Chen Ling, Colin M. Gray
similarity 0.54
Abstract · 23 pages, 3 figures. Submitted to CHI 2027

Social media platforms are central to teenagers' lives, and their designs can expose users to privacy, safety, and wellbeing harms. Platforms increasingly offer protective settings, though the presence of a control reveals little about whether teenagers can find, use, and benefit from it over time. We paired an expert evaluation of six privacy and safety tasks across TikTok, Instagram, Snapchat, and YouTube with moderated think aloud sessions in which 11 teenagers aged 14 to 17 attempted the tasks. Interaction cost and dark patterns analysis allowed us to compare the complexity designed into each task with the effort participants incurred as they located, configured, and interpreted controls. Recurring dark patterns appeared across tasks, and most participant attempts exceeded the expert baseline. Protective settings therefore risk being insufficiently usable or durable in practice, and we propose a wayfinding audit that integrates expert evaluation, usability testing, interaction cost, and dark pattern analysis.

0.37

Replication Certification at Publication: Checkable by Anyone, Anywhere, in Seconds, Forever

cs.DL cs.CRstat.ME ·2026-11-06 ·Gary King, Yiqing Xu, Leo Yang Yang
similarity 0.54
Abstract

Authors increasingly deposit replication datasets---the data and code behind publications---in archives, and journals increasingly require them to, under progressively onerous rules. But a deposit alone proves little, so authors keep paying: months for the journal's reanalyses, years answering readers' questions, and rancor as failed replications become accusations because no record establishes what actually ran. We introduce a replication certificate: a cryptographically signed record of the deposited files and logged output. It replaces the journal's reanalyses with evidence anyone can check in seconds, which no one, including the archive, can backdate, and no one can fake without risking detection. Certification asks little of anyone and benefits everyone: The author deposits, then reruns the analysis once under a random seed the archive computes from the deposit's hash and a blockchain block; the archive signs, timestamps (in that blockchain), and stores the record---confidential data included---without rerunning anything; the journal merely checks a website for the certificate; and anyone can check or replay any step decades later.

0.19

The Privacy Paradox Is a Misnomer: Data Under Structural Uncertainty

cs.CY cs.HC ·2026-10-21 ·Ignacio Cofone
published similarity 0.54
Abstract · 2 figures, 4 tables

The infamous privacy paradox refers to the apparent inconsistency between people's stated concern for privacy and their readiness to disclose personal information. This phenomenon has sparked two largely disconnected literatures: one offering experimental evidence of inconsistent behavior, and another providing qualitative accounts and defending the importance of privacy. The Article presents an online field experiment that bridges those literatures and shows that the so-called paradox arises from a mischaracterization of the underlying behavior. The Article finds that it is structural uncertainty about risk that drives seemingly paradoxical privacy decisions. It does so by isolating discounting mechanisms and empirically testing whether observed privacy choices reflect temptation or rational responses to uncertainty. The results suggest that privacy behavior is not paradoxical but, rather, consistent with choices shaped by incomplete information. The Article then discusses the policy implications of this reframing. As privacy decisions stem from structural uncertainty, which operates as a market failure, regulation should aim to reduce that uncertainty. This supports regulation that prioritizes transparency (for people to assess the risks of data collection) and flexibility mechanisms that accommodate evolving contexts. Such reframing provides a new argument for the right to be forgotten, which allows people to revisit prior disclosures as new risks become apparent. By shifting the focus from individual inconsistency to structural uncertainty, the findings call for privacy law to better reflect the reality of people's decision-making environments.

0.74