CS Paper Radar

Taxonomizing Synthetic Data for Law

Ignacio Cofone, Katherine J. Strandburg, Nicholas Tilmes

cs.CY cs.CRcs.LG ·submitted 2026-11-11 13:50 UTC · v1
publishedposition

Abstract

Synthetic data is increasingly important in data usage and AI design, creating novel legal and policy dilemmas. All too often, discussions of synthetic data treat it as entirely distinct from "real," collected data, overlooking the risks posed by different kinds and uses of synthetic data. We build on Michal Gal and Orla Lynskey's argument that synthetic data will transform information privacy, market competition, and data quality. While the risks posed by synthetic data depend on its connection to collected data, we argue that background knowledge and assumptions about ground truth used to create it are at least as important. We bring that focus to Gal and Lynskey's taxonomy of synthetic data, arguing that this focus is essential to grasp synthetic data's legal and policy implications. As such, we divide synthetic data into (1) transformed data, which modifies collected data to preserve certain statistical properties for an end use; (2) augmented data, which relies on assumptions to bolster a collected dataset's fidelity to the ground truth; and (3) simulated data, which relies almost entirely on background knowledge and ground-truth assumptions. Applying this taxonomy, we show that no category of synthetic data prevents privacy harms from group-based inferences, that augmented and simulated data can amplify bias while making a dataset look more complete, and that synthetic data rarely lets smaller competitors overcome incumbents' data advantages. As policymakers weigh whether to incentivize, mandate, or discourage the use of synthetic data, they should consider the validity of the ground-truth assumptions used in producing that data.

Comment
29 pages
Journal ref
Iowa Law Review Online 110 (2025) 217-245

Closest papers

Nearest neighbours by meaning across the whole index.

Position: Privacy Is a Claim, Not a Property of Synthetic Data

cs.LG cs.CR ·2026-09-01 ·Jiachen Zhao, Antonia Januszewicz, Taeho Jung
top venueposition similarity 0.72
Abstract

Synthetic data has become a common component of machine learning research. While widely adopted, its use in privacy-sensitive contexts has quietly shifted from a claim of residual inference risk under stated assumptions to an appearance-based property inferred from data generation itself. In this position paper, we argue that this shift reflects an implicit change in community standards for what counts as sufficient privacy evidence, rather than a misunderstanding of well-established privacy principles. Drawing on an empirical analysis of recent publications across major ML venues, we show that synthetic data is frequently used in privacy-sensitive settings without explicit articulation of threat models, inference risks, or falsifiable privacy claims. As a result, privacy assurance often remains implicit, difficult to verify, and unevenly distributed, with heightened exposure for rare and minority records. We argue for treating privacy as an explicit, evidence-based scientific claim and recommend that ML venues adopt norms requiring privacy-relevant assertions to be clearly scoped, testable, and contestable.

0.46

Exploring The Empirical Privacy Landscape of Synthetic Data

cs.CR cs.CL ·2026-10-15 ·Zihao Zhao, Avi Schwarzschild, Casey Meehan, Katherine Lee, Zhili Feng +1 more
similarity 0.67
Abstract

Methods for training language models on sensitive data differ in how they balance capability gains against the risk of leaking private information. In this paper, we present a controlled comparison of data-level and training-level interventions for navigating the capability-privacy tradeoff on synthetic medical records with canaries to probe for leakage. Direct finetuning performs best on capability metrics but leads to the most privacy leakage, while DP-SGD reduces leakage at substantial cost to capabilities. Generating synthetic data from these records offers a more favorable balance, particularly when guided by a uniqueness-aware instruction to replace private details. Relative to naive paraphrasing, privacy-aware rewriting improves privacy protection across a variety of different rewriting models. We also find that more capable rewriting models generally reduce the risks of downstream privacy leakage, although held-out fit does not improve monotonically and increased test-time compute yields inconsistent gains. Overall, these results highlight privacy-aware rewriting as a promising direction for achieving a more favorable trade-off between capabilities and empirical privacy compared to other training-level methods, and our evaluation framework supports comparisons of future models and methods.

0.34

PEARL: A Task-Aware Framework for Evaluating Differentially Private Synthetic Educational Data

cs.CR cs.LG ·2026-09-08 ·Xianghui Meng, Yujing Zhang, Jionghao Lin
similarity 0.60
Abstract

Personalized learning systems rely on real learner data, including performance, behavior, and demographic information, but these data are highly privacy-sensitive. Differentially private (DP) synthetic data can support system development and educational research while reducing exposure of individual learners. Existing evaluations, however, assess privacy and predictive usefulness separately, without determining whether synthetic learner data remain usable for the intended personalized learning task. We introduce PEARL (Privacy-Equivalence Audit and Release Ledger), which approves a DP synthetic educational dataset only when it passes all required checks of validity, privacy protection, predictive usefulness, and suitability for the intended educational task, while recording why each rejected dataset fails. Across 96 study settings, each defined by a dataset, data-generation method, privacy budget, and random seed, only 12 produced synthetic datasets that passed all applicable PEARL checks. Many privacy-protected datasets were rejected for omitting important outcome groups, such as withdrawn students, or for failing to preserve the order of learning activities. Fairness analysis further showed that some datasets passing privacy and predictive-usefulness checks still yielded unequal at-risk prediction performance across groups defined by disability and socioeconomic background. Moreover, Deep Knowledge Tracing and Self-Attentive Knowledge Tracing learned no meaningful next-response patterns from any tested synthetic knowledge-tracing dataset, showing that privacy protection alone does not guarantee usefulness for dropout prediction, knowledge tracing, or adaptive tutoring.

0.13

When Is Content "AI-Generated Enough"? Labelling Synthetic Media under the Digital Services Act and the AI Act

cs.CY ·2026-09-07 ·Marie-Therese Sekwenz
published similarity 0.59
Abstract · 13 pages, 7 figures, 1 table. Accepted and presented at the Fifth European Conference on Algorithmic Fairness (ECAF 2026), Ghent, Belgium, 2…

European platform and AI governance increasingly relies on transparency duties to address synthetic and manipulated media. Under the DSA, very large online platforms and search engines may use prominent markings and recipient-facing indication tools as systemic-risk mitigation measures. Under the AI Act, providers must support machine-readable marking, while deployers must disclose deepfakes and certain AI-generated or manipulated public-interest text, subject to statutory qualifications. This extended abstract examines when labelling is a meaningful regulatory response to synthetic media and when it risks becoming over-inclusive, under-inclusive, or ineffective. It argues that the central challenge is not only whether content should be labelled, but how legal thresholds, technical provenance systems, platform interfaces, and reporting practices determine when content is sufficiently generated, manipulated, or authentic-looking to trigger transparency obligations. Drawing on the emerging Article 50 AI Act implementation framework and a snapshot of the DSA Statement of Reasons database, the paper identifies four governance tensions: definitional ambiguity, interface and responsibility design, communicative effectiveness, and fairness and contestability. It conceptualises labelling as a socio-technical classification practice that distributes responsibility among AI providers, deployers, platforms, uploaders, and recipients.

0.56

Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text

cs.CL cs.AI ·2026-08-31 ·Minkyung Cho, Jihyo Kim, SeungWoo Song, Junghun Yuk, Minjoon Kee +2 more
top venue similarity 0.59
Abstract · To be published in EMNLP 2026

Synthetic data is increasingly used to train large language models (LLMs), yet its security implications remain poorly understood. Prior work on subliminal learning suggests that models can inherit behavioral traits from seemingly unrelated training data. In this work, we investigate whether such mechanisms can be exploited to inject targeted social biases into aligned models through semantically benign synthetic data. We construct a pipeline in which a misaligned teacher model generates filtered synthetic datasets across domains such as creative writing and code generation, which are then used to fine-tune aligned student models. Our experiments show that benign-looking synthetic data can act as a covert channel for transmitting targeted biases while largely preserving the student model's general task capabilities. These results reveal a previously underexplored security risk in synthetic data-driven LLM training pipelines and highlight the need for improved safeguards. As one possible step toward this goal, we suggest that log-linearity-based scoring may provide a useful signal for screening seemingly benign synthetic data.

0.36

Who's Asking? Audience-Conditioned Privatization

cs.CR cs.AIcs.LG ·2026-11-06 ·Daniil Filienko, Anusha Bagalkotkar, Xan Vongsathorn, Sikha Pentyala
similarity 0.58
Abstract

The abundance of user-generated conversational data makes the question of how to responsibly derive insights from confidential data without putting the individuals in it at risk ever more important. However, the majority of existing differentially private (DP) synthetic text generation (SDG) and anonymization approaches ignore the context of the data's downstream usage, which results in loss of important information. DP-SDG provides the same guarantee regardless of the recipient, and anonymization relies on predefined attribute categories. They both ignore the observation that an attribute that is harmless to one audience can be sensitive to another, and each audience typically needs only some attributes preserved. We propose Ext-Classify-Priv, a framework that (1) extracts the information types present in a corpus, (2) classifies which of them are private for a given audience following contextual integrity, and (3) protects only those attributes, using either anonymization or SDG with Feature-DP approaches. On Reddit data with 3 audiences, our audience-specific releases are more faithful than a single audience-agnostic release for every audience. On bioRxiv data, our SDG method improves MAUVE from 0.097 to 0.420 at epsilon=1 and on the proprietary real estate data, our anonymization method improves MAUVE from 0.698 to 0.959 over the best baselines.

0.12

Neighborhood Watch: Privacy Risks in Seeded Local Combination Synthetic Data

cs.CR ·2026-08-27 ·Hadrien Lautraite, Tristan Allard, Anne-Sophie Charest, Jean-François Rajotte, Sébastien Gambs
hot direction similarity 0.58
Abstract · 16 pages main body, 20 pages with Appendix, 6 figures (including appendix), 7 tables (including appendix)

Synthetic data is seen as a promising solution for sharing data in sensitive contexts. However, recent work on privacy attacks have shown that there are still significant residual risks, especially for synthetic data generations methods that are not based on formal approaches such as differential privacy. In this paper, we investigate the privacy risks associated with local combination approaches for generating synthetic data in which synthetic profiles are built by combining real neighbouring profiles. More precisely, we focus on three methods from this family, namely SMOTE, Simulant and Avatar, which have been recently used as a way to share 'anonymised data' in the healthcare domain. In particular, we conduct an extensive privacy analysis through a diverse set of attacks: membership inference, linkage and reconstruction attacks. Our results demonstrate substantial privacy leakage for all three methods, raising serious doubts about whether their outputs should be regarded as anonymous in practice.

0.36

PrivCert: Certifying Statement Support under Differential Privacy

cs.CR cs.LG ·2026-09-30 ·Tsubasa Takahashi, Takumi Hiraoka
similarity 0.58
Abstract

Differentially private (DP) text generation can protect individual records, but privacy alone does not specify what evidence a released statement carries about the underlying data. We identify this as an evidence gap: a private report may contain plausible claims without indicating whether they are strongly supported by the private dataset. We introduce PrivCert, a framework for privacy-preserving reporting that makes statement support explicit through privacy-preserving certificates and emit-or-abstain decisions. As a canonical instantiation, PrivCert-PF (Proposal-and-Filter) separates data-independent candidate discovery from private support certification, emitting only statements whose support passes a private evidence test. We provide theoretical grounding for this framework by characterizing the limits of implicit evidence under DP, deriving a sharp privacy--honesty frontier for single-statement certification, and establishing a worst-case cost for fine-grained multi-statement certification. Experiments on synthetic tasks and TAB, WildChat, and Yelp show that explicit certification maintains low unsupported emission, while free-text DP baselines frequently produce low-support claims under the same declared support semantics. We further show that the PrivCert contract can be realized with histogram, sparse-vector, and Gaussian mechanisms, and use DP synthetic data to illustrate an important boundary: support in a private proxy does not automatically certify support in the original data. Together, these results position privacy-preserving reporting as an evidence-design problem: not only how to generate private text, but what a private report can substantiate about its underlying data.

0.14