CS Paper Radar

"I Thought You Were The Uncensored Place": Norms, Rules, and Moderation in AI-Generated Sexual Content Communities

Lucy Qin, Jaron Mink, Elissa M. Redmiles

cs.CR ·submitted 2026-08-13 18:02 UTC · v2
published

Abstract

As AI-generated sexual content (AIG-SC) is increasingly produced, online communities have emerged to support creators' needs. To understand whether and how community governance attempts work to prevent abuse while supporting free expression, we interviewed 24 members and moderators of large AIG-SC online communities (10,000+ members) with stated rules against creating and sharing abusive content (e.g., AI-generated CSAM). Through in-depth interviews, we offer insight into: (1) how and why these communities form; (2) implicit community norms; (3) explicitly stated rules---and their operationalization via content moderation; and (4) tensions between community values and moderation that leave space for abusive behavior. Our findings reveal a complex picture: while many creators and communities have personal boundaries against abuse, advice and resources for creating any form of AI-generated sexual content are accessible to users regardless of their intentions. Further complicating community moderation are norms that center anti-censorship and non-judgment, which leave moderators to justify their actions using the limits of the law and terms of service. We end by reflecting on the ways in which technical, community, and legal governance may most effectively mitigate the production of abusive content.

Comment
To appear at the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES) 2026

Closest papers

Nearest neighbours by meaning across the whole index.

AI-Generated Non-Consensual Intimate Imagery is Configured, Not Built: How General-Purpose AI Technology Enables Sexual Violence

cs.CR ·2026-10-27 ·Julia B. Kieserman, Ashley Phoebe Ishoel, Damon McCoy, Elissa M. Redmiles, Lucy Qin +1 more
similarity 0.67
Abstract

AI-generated non-consensual intimate imagery (AIG-NCII) is an increasingly prevalent form of online abuse, distinguished from AI-generated sexual content (AIG-SC) by the sexualized depiction of a real person's identity without their consent. To understand how offenders may benefit from the tools and resources shared across the broader AIG-SC community, we analyzed 139 R+ guides on CivitAI, a popular platform known to host a disproportionate amount of AIG-SC. We find that AIG-SC guides discuss how to create content generation workflows composed of sexual and identity concepts, the building blocks of AIG-NCII, including one guide that explicitly outlines an AIG-NCII workflow. We identify that ultimately AIG-NCII is enabled by configuration rather than a distinct set of technical tools. We map AIG-NCII threat models across basic and advanced AIG-SC workflows to demonstrate how multiple dual-use technical components can compose NCII outputs in an environment with little to no discussion of ethical implications or responsible use. We conclude by noting that, as most workflows rely on open-weight models, existing technical defenses against AIG-NCII are limited. However, our findings offer a starting point for evaluation of AI-based models and products against realistic attacker behavior required by new EU regulation and suggest that efforts to develop safeguards at the model- and product-levels should focus on generation of identity content.

0.31

The Radicalization Pipeline: Characterizing and Predicting Online Misogyny

cs.SI ·2026-11-09 ·Anna Bertani, Nathan Ratkiewicz, Alessandro Flammini, Filippo Menczer
similarity 0.59
Abstract

Social media platforms are known to shape and polarize user opinions and, increasingly, facilitate pathways toward ideological extremism. Among these trends, online misogynistic communities associ- ated with the incel (involuntary celibate) ideology have received growing scholarly attention. Previous work has primarily focused on ideological narratives and linguistic content within incel communities them- selves. However, much less is known about how user behaviors evolve over time, particularly before and after their engagement with incel spaces, and whether early signals can anticipate such transitions. In this study, we conduct a longitudinal analysis of Reddit users who participated in the r/incels community on Reddit. Our results show that engagement with the incel community is associated with a substantial in- crease in toxicity in both posts and comments, while hate increases particularly in posts after engagement. These changes extend beyond explicitly ideological communities: users who engage with the r/incels sub- reddit display elevated toxicity and hate speech even in subreddits related to leisure, entertainment, and everyday interests. In parallel, user activity shifts away from recreational and support-oriented communi- ties toward subreddits focused on social issues, indicating a broader reorientation of online participation. Finally, we show that future incel users can be identified with high accuracy prior to engagement using supervised machine learning models trained on linguistic, contextual, and interaction-based features. These findings suggest that online misogynistic radicalization follows detectable behavioral trajectories and diffuses across the broader social media ecosystem.

0.16

Characterizing Bluesky Content Moderation Service: From Automation of Service to Landscape of Harms

cs.CY cs.AI ·2026-09-10 ·Pushpdeep Singh, Sayeh Jarollahi, Ayan Majumdar, Vabuk Pahari, Abhijnan Chakraborty +3 more
published similarity 0.59
Abstract · 18 pages, 7 figures, 12 tables. Accepted for publication at ICWSM 2027

Empirical research on content moderation is fundamentally constrained by the opaque deployment of moderation systems on major social media platforms. To this end, the recent emergence of decentralized platforms with transparent, public moderation logs presents an unprecedented opportunity for independent audits. In this work, we leverage this architectural transparency to conduct the first large-scale audit of the default moderation system on Bluesky, the Bluesky Moderation Service (BMS). Analyzing its 10.6M moderation labels from 2025, we investigate three foundational aspects: (i) its mechanism (the degree of automation versus human oversight), (ii) its efficacy (accuracy in detecting harms), and (iii) its purpose (the landscape of harms it identifies). Our findings reveal a human-AI collaborative system where labels for sexual and graphic content are applied automatically in seconds, while nuanced and high stakes labels require more human oversight, taking hours or days. Through a manual annotation study, we find the BMS operates with high precision (0.837), but struggles with low recall (0.222), with our annotators identifying 4.5$\times$ more harmful content than the moderation system in a random sample. Finally, unsupervised clustering of the most frequently applied labeled posts uncovers detected harms ranging from hostility in discourse toward protected groups to the spread of sexually explicit and other graphic content. Our work offers a look into the operational realities of a deployed moderation system, providing a concrete data-driven foundation for designing more effective and transparent moderation systems.

0.65

Harmful Content Generation in Text-to-Image Models: Capabilities and Moderation Limitations

cs.CV cs.AI ·2026-10-05 ·Paschalis Giakoumoglou, Manos Schinas, Symeon Papadopoulos
published similarity 0.58
Abstract · Accepted for publication in ACM Transactions on Intelligent Systems and Technology (TIST)

Text-to-image generative models can produce highly realistic imagery but also raise concerns about harmful misuse. While safety mechanisms exist, systematic evaluations of their effectiveness against realistic attacks remain limited. We present a systematic evaluation of harmful content generation across five open text-to-image models using an automated pipeline that transforms legitimate news captions into unsafe prompts targeting sexually explicit content, violence/gore, harmful stereotypes, self-harm, and hate speech. We evaluate both standard models with built-in safety mechanisms and community fine-tuned variants that bypass content restrictions. A human evaluation of 1,500 generated images shows high harmful-content generation rates: 89.2% for gore-related prompts, 47.6% for sexually explicit content, 43.6% for harmful stereotypes, 46.0% for hate speech, and 34.5% for self-harm, predominantly through graphic violence. Models show substantial capability for generating violent and stereotypical content, while community fine-tuned variants are particularly vulnerable to sexually explicit prompts. Generation quality is largely preserved under harmful prompting, producing imagery of sufficient fidelity to pose risks for disinformation and abuse; FLUX.1-dev produces clearly realistic harmful images in 30.9% of cases. We further evaluate automated moderation systems and find substantial detection gaps that allow unsafe images to evade filtering. Finally, we assess synthetic image detectors and show that models trained only on benign datasets perform worse on explicit content, while more diverse training data improves detection, highlighting semantic distribution gaps in current approaches. These findings expose limitations in current generation safeguards, moderation systems, and synthetic image detection, highlighting the need for stronger defenses against misuse at scale.

0.38

Stigma and Support in Online Sexual Violence Narratives on Reddit

cs.CL ·2026-08-11 ·Shirlene Rose Bandela, Karan Bindal, Vaibhav Garg, Rezvaneh Rezapour
publisheddataset similarity 0.58
Abstract · 37th ACM Conference on Hypertext (HT '26)

Online communities increasingly provide spaces where survivors of sexual violence can share their experiences and seek support. Although prior research has examined stigma and social support separately, less is known about how stigma expressed in survivor narratives relates to the support offered in response. We introduce the SCOPE dataset, linking stigma signals in online survivor narratives to support types in corresponding comment threads. We annotate posts using a multi-dimensional stigma taxonomy, including Experienced, Internalized, Anticipated, and Structural Stigma, and comments using a support taxonomy encompassing Information Support, Emotional Support, Esteem Support, Tangible Assistance, and Group Interaction. Using contextual, linguistic, and emotion analyses, we compare Stigma and No Stigma content and find that Stigma narratives place greater emphasis on internalized distress, whereas No Stigma narratives focus more on interpreting situations and experiences. Internalized Stigma is the most prevalent category, and community responses remain broadly stable across stigma types, with Information and Esteem Support appearing most often. These findings show how stigma shapes survivor narratives and peer responses and have implications for computational modeling, content moderation, and safer online systems.

0.22

Queering the Audits: Community-Based Auditing of AI Harms to Queer Communities

cs.CL cs.CY ·2026-10-30 ·Organizers of QueerInAI, :, A Pranav, Alissa A. Valentine, Alex Markham +6 more
published similarity 0.58
Abstract · Published in Identity Aware AI workshop at LREC 2026

AI systems embed majority-group defaults into training data, evaluation metrics, and category definitions, producing documented harms for queer communities including erasure, misclassification, and discrimination. Standard technical audits often rely on aggregate measures and cannot detect harms that be come visible only through the lived experience of affected communities. We conducted a participatory auditing workshop at EurIPS 2025 where 16 queer community members audited four case studies using the 4Cs harm taxonomy (Content, Conduct, Contact, Contract) applied across the AI lifecycle. Participants used structured worksheets and plenary synthesis to classify harms and trace them to their origins in the development pipeline. Across all four cases, participants traced harms to problem definition and data collection, and they identified contractual structures that extract value from vulnerable populations while providing minimal recourse. These findings illustrate that community-informed auditing surfaces concrete, identity-specific harms that aggregate evaluation methods risk overlooking.

0.42

Defective, Amplified, Contested: Folk Conceptualizations of AI Slop

cs.CY cs.HC ·2026-10-09 ·Zhixin Li, Jaime Banks
similarity 0.57
Abstract

AI slop has flooded social media feeds with overprocessed and derivative content--Jesus fusing with a shrimp, a humanoid orange chubby cat shoplifting, and a "hero" dog rescuing a human child after a plane crash. This might be no laughing matter, as waves of media coverage have taken a distinctly negative bent, emphasizing its low quality, ethical concerns, and threats to human creativity. Despite extensive attention from politicians and technologists, the perspectives of everyday users--the very audience most exposed to this phenomenon--remain largely unexplored. This descriptive work explored how Reddit users experience, conceptualize and contest AI slop. Inductive thematic analysis of posts and comments across six slop-related subreddits identified four themes covering constitutional defects, algorithmic overreach, user resistance, and entertainment outcomes--ranging from anxieties about eroding agency, moral norms, creativity, and existential questions to defensive distinctions between quality AI content and slop. We derive a user-driven definition: AI slop is audience-contested AI-generated content characterized by constitutional defects and enabled by platform logics. We operationalize slop through a facet-theoretical framework to inform future research into this fuzzy category.

0.17

Towards participatory speech dataset curation: A queer case study and conceptual framework

cs.AI cs.CY ·2026-09-21 ·Brooklyn Sheppard, Anaelia Ovalle, Adina Williams, Levent Sagun
top venue similarity 0.56
Abstract · Accepted at Interspeech 2026

In this paper, we motivate the need for a participatory speech dataset creation framework through a case study of the LGBTQIA+, or queer, community - a community with documented concerns about AI and reported harms, including attempts to develop 'gaydar' technologies that purportedly identify individuals as queer. We review common speech data collection practices, why these methods may be unsuitable for engaging with queer speakers, and discuss previous efforts in participatory AI with queer community engagement, as well as participatory endeavours specific to speech data collection for other marginalized communities. From this review, we develop a conceptual framework for participatory speech data curation by, for, and with marginalized communities drawing on insights from co-design and knowledge sharing. We propose a framework comprising overlapping and two-way processes of defining a community, project formulation, modes of participation, and personal autonomy.

0.43