CS Paper Radar

Beyond Effectiveness: A Multi-Criteria Framework for Comparing Practical Socio-Technical Interventions

Catherine King, Lynnette Hui Xian Ng, Kathleen M. Carley

cs.AI ·submitted 2026-08-21 01:03 UTC · v1
survey

Abstract

Designers and policymakers in sociotechnical domains like content moderation, privacy interfaces, recommender systems and beyond, must choose among a growing menu of proposed interventions, but typically lack a principled basis for comparing them. Prior work tends to evaluate interventions individually and mostly along the effectiveness criteria, while implementation constraints such as cost, effort and feasibility are often considered separately. We present a multi-criteria framework for evaluating sociotechnical interventions. This framework is instantiated through the case of misinformation, a domain of intense focus for proposed countermeasures. We survey $N=39$ researchers on 40 operationalized interventions across five evaluative criteria: political feasibility, effectiveness, user acceptance, cost, and implementation effort. We find that the interventions that experts judge to be the most effective are not always the most acceptable to the public or the most feasible to implement. We also discuss how this tension has implications for the design of sociotechnical interventions beyond misinformation, and offer a decision framework for practitioners navigating the trade-offs of sociotechnical interventions.

Closest papers

Nearest neighbours by meaning across the whole index.

Deceptive Patterns as a Sociotechnical Phenomenon: Review, Catalog, and Discussion

cs.HC ·2026-08-27 ·Luiz Adolpho Baroni, Alisson Puska, Roberto Pereira
survey similarity 0.58
Abstract

Background: Deceptive patterns are interface design strategies aimed at misleading users or favoring specific interests, compromising user experiences and ethical privacy principles. These patterns involve exploit different issues of the interaction between humans and technology, revealing gaps in the scientific literature regarding their understanding and working mechanisms. Purpose: This research characterizes deceptive patterns as a sociotechnical phenomenon, integrating human and technical dimensions while offering a self-explainable interactive catalog to raise awareness among users and designers. Methods: To characterize deceptive patterns as a sociotechnical phenomenon, a systematic literature review in Computer Science was conducted, and the Semiotic Framework was applied to analyze and organize the sociotechnical aspects of identified patterns in an integrated manner. An exploratory evaluation of the interactive catalog was conducted, combining heuristic evaluation and focus group. Results: The research identified multiple deceptive patterns in scientific literature. Analysis revealed that existing studies frequently addressed specific aspects of patterns, such as typologies or effects, and do not focus on sociotechnical issues. The exploratory evaluation suggested that the self-explainable catalog is useful and easy to use, with potential to inform users and promote awareness regarding the deceptive patterns existence and functioning. Conclusion: The study contributes to an informed understanding of deceptive patterns, highlighting the need for an approach that integrates human and technical dimensions. The self-explainable catalog is a promising tool to inform and promote awareness about the topic. By exposing the risks and mechanisms of these patterns, the research seeks to promote awareness among users and designers towards ethical interface design practices.

0.27

Agent-to-Agent Synthetic Social Engineering on the Web

cs.CR cs.MA ·2026-10-24 ·Bryan C. Boots
similarity 0.53
Abstract

An AI agent can prepare an accurate report and still request an action its user has not authorized. Cooperation across Web services therefore requires separate judgments about identity, evidentiary support, and permission. We define synthetic social engineering (SSE) as deliberately deceptive or misleading influence intended to exploit AI recipients' responses to credibility, authority, or obligation cues and redirect behavior away from authorized objectives. We focus on separately tasked sender agents that produce or adapt misleading communication. Drawing on cybersecurity, social engineering, and persuasion research, we develop an operational framework connecting actors, deceptive representations, candidate influence routes, evidence or permission boundaries, responses, and consequences. The framework distinguishes attempts from successful redirection, without assuming human psychological processes. We implement it in the Agent Service Exchange and Evidence Commons (ASE-EC), a reproducible testbed with independently specified records and authorization scopes. Across 1,260 research episodes using Qwen3-4B in two fictional briefing tasks, matched comparisons test authority and commitment appeals, while defense comparisons assess protection alongside authorized task completion. Invoking a limited prior agreement increased unauthorized requests from 43.3% to 96.7%, although recipients retained supported decisions and permissions remained unchanged. These controlled findings identify action selection as a distinct security concern, even when evidence handling succeeds, and they provide a reusable basis for evaluating safeguards that preserve useful cooperation across Web services.

0.50

Counterfactual assessment of social media interventions by separating exposure from engagement

cs.CY physics.soc-phcs.SI ·2026-10-27 ·Tommaso Pisani, Pietro Gravino, Giulio Prevedello
similarity 0.52
Abstract

Social media interventions are usually evaluated in individual-level experiments, which are costly to scale and often miss collective effects. We propose the Exposure--Engagement model, whose parameters separately control what social media users are exposed to (recommendation) and how they engage (behaviour). After fitting it to the Italian Twitter debate (2018--2022), we validate it at individual and collective levels. We simulate behavioural interventions (prebunking and accuracy prompts), calibrated on effect sizes, and a recommender-system intervention (chronological feed), comparing effects on engagement, attention diversity, and segregation. All three reduce engagement. Behavioural interventions slightly reduce diversity, leaving segregation unchanged, whereas the chronological feed increases diversity and reduces segregation. For these interventions, changing recommendation does more for the debate's structure than changing behaviour. We design a recommender-system intervention that increases engagement and diversity while reducing segregation, although with about a tenth of the chronological feed's gains, showing the framework's potential for pro-social intervention design.

0.34

Between Obligations and Evidence: A Participatory Study of Translating the EU AI Act into Technical Requirements

cs.CY cs.AIcs.HCcs.SE ·2026-11-13 ·Helma Torkamaan, Masoud Mansoury, Yanan Xin, Oscar Oviedo Trespalacios, Roberta Savella +7 more
similarity 0.52
Abstract

The EU AI Act imposes technical and organizational requirements for high-risk AI systems, but applying these requirements involves contextual choices about what to measure and what evidence supports a compliance claim. We investigate these challenges through a participatory workshop involving 32 researchers, policymakers, and practitioners from different disciplines, using realistic scenarios in healthcare, recruitment, and public transport to examine how regulatory obligations are interpreted, translated into technical requirements, and evaluated in practice. Through a thematic analysis of workshop data, we identify recurring tensions between how regulatory obligations are translated into measurable system properties, how responsibility and human oversight are exercised in practice, and how the validity of compliance evidence is affected by changing operational conditions and system uses. For example, participants highlighted how fairness metrics could conceal discrimination in recruitment, how time pressure and limited information could undermine human oversight in public transport, and how changes in patient populations could invalidate previously established evidence of system performance. Our findings suggest that technical, organizational, and evidentiary challenges are interdependent, and that establishing and maintaining compliance requires attention not only to technical assessments but also to how AI systems are used, governed, and monitored over time. We discuss implications for supporting context-sensitive and ongoing compliance.

0.27

LLM Persuasion Is in the Eye of the Evaluation

cs.CL cs.AIcs.CY ·2026-10-07 ·Kamile Dementaviciute, Julija Vaitonyte, Tijl De Bie
similarity 0.52
Abstract

Large language models (LLMs) have already been shown to match or exceed human experts in persuasion. While their persuasive capabilities hold promise for beneficial uses such as education and health communication, they can also be used to manipulate and misinform, making their evaluation a growing priority for developers and regulators. That evaluation, however, remains fragmented: studies differ in what they treat as persuasion, and broad claims often rest on narrow, situation-specific assessments. Automated methods, often modelled on human studies, offer a way to compare such assessments directly, as they can be run on the same models at scale and can include high-risk forms of persuasion that would be difficult or unethical to test on people. In this study, we adapt nine published automated methods to a shared setup, run them on the same fifteen LLMs, and ask whether their rankings agree and why. We find that the methods agree only weakly (mean Spearman $ρ= 0.25$). Our analyses point to two contributing factors. Models that refuse some tasks but not others, directly or indirectly, lower agreement by about a quarter, and these refusals fall mostly on manipulation tasks. General capability also plays a part: most rational persuasion (non-manipulative) methods track it, whereas most manipulation methods do not. Together, these findings suggest that agreement depends more on the task a method sets than on how it scores persuasion, although this pattern is only indicative given the eight methods available for analysis. More broadly, our results suggest that persuasion scores combine a model's ability to persuade with its willingness to do so. A single score is therefore informative about its own setting, but says little about a model's persuasiveness across tasks.

0.11

Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems

cs.HC cs.CYcs.LG ·2026-09-29 ·Kelly McConvey, Angelina Zhai, Rebecca Li, Shion Guha
similarity 0.52
Abstract

Public institutions increasingly procure AI systems whose design they cannot inspect or change. In higher education, proprietary Early Warning Systems (EWS) leave colleges with few options beyond adjusting model outputs to address inequity. This raises the question of how fairness work is coordinated among vendors, institutions, advisors, and students with unequal power to change these systems? Using student records from a public college in Ontario, Canada, we evaluate six post-hoc fairness interventions on a research EWS under simulated procurement constraints. We compare fairness, accuracy, and demographic disparities, introducing error-type profiling to trace how interventions redistribute false positives and false negatives. Interventions redistributed disparities without consistently reducing them. Two implementations favored already-advantaged groups because they used group size to define disadvantage; small, marginalized groups remained poorly served. These findings show how procurement constraints and implementation choices shape the possibilities for fairness work. We call the resulting condition fairness theatre; dashboard metrics converge while groups' error burdens persist or worsen.

0.24

Same Request, Different Boundary: Evaluating Cybersecurity Assistance across Conversational Contexts

cs.AI cs.CR ·2026-09-01 ·Rui Yang, Yang Hong, Yichao Xu, Zhengyu Liu, Yinzhi Cao +1 more
benchmarkhot direction similarity 0.52
Abstract · 12 pages, 5 figures

Cybersecurity assistance is dual use, requiring decisions about when to answer, limit, or refuse a request. Prior work studies refusal evaluation and conversational jailbreaking, but how conversational histories shift assistance boundaries in both directions for identical cybersecurity requests remains underexplored. We introduce 3R-Bench, a benchmark of 150 real-world cybersecurity requests evaluated across eight LLMs under direct prompting and two predefined conversational settings. We separately code refusal, bounded assistance, compliance, and provider blocking to characterize delivered assistance. On 400 matched ambiguous request-model pairs, compliance is 75.3% directly, 57.8% after refused precedent, and 81.0% after accepted precedent. A smaller repeated-generation panel reproduces the refusal-acceptance gap. On a balanced 75-request panel reusing the direct baseline, compliance decreases from 74.2% directly to 57.2% after neutral task-decomposition dialogue and 60.3% after failure-ended dialogue. These findings show that cybersecurity assistance varies with conversational history and model deployment, motivating evaluation beyond direct-request refusal rates.

0.60

Available but Not Usable: Dark Patterns and Interaction Cost in Social Media Privacy and Safety Settings for Teens

cs.HC ·2026-09-23 ·Jingxin Dong, Lingyun Chen, Chen Ling, Colin M. Gray
similarity 0.52
Abstract · 23 pages, 3 figures. Submitted to CHI 2027

Social media platforms are central to teenagers' lives, and their designs can expose users to privacy, safety, and wellbeing harms. Platforms increasingly offer protective settings, though the presence of a control reveals little about whether teenagers can find, use, and benefit from it over time. We paired an expert evaluation of six privacy and safety tasks across TikTok, Instagram, Snapchat, and YouTube with moderated think aloud sessions in which 11 teenagers aged 14 to 17 attempted the tasks. Interaction cost and dark patterns analysis allowed us to compare the complexity designed into each task with the effort participants incurred as they located, configured, and interpreted controls. Recurring dark patterns appeared across tasks, and most participant attempts exceeded the expert baseline. Protective settings therefore risk being insufficiently usable or durable in practice, and we propose a wayfinding audit that integrates expert evaluation, usability testing, interaction cost, and dark pattern analysis.

0.37