CS Paper Radar

DOCKET: A Pluralistic Benchmark for Evaluating AI-Assisted Policy Feedback in U.S. Rulemaking

Alexander Spangher, Charlotte Peale, Sílvia Casacuberta, Victor Y. Wu, Mirac Suzgun, Olivia H. Martin, Dan Bateyko, Cassandra Handan-Nader, Diyi Yang, Sanmi Koyejo, Daniel E. Ho

cs.CY ·submitted 2026-11-13 08:55 UTC · v1
benchmark

Abstract

As AI models advance and impact deliberative democracies, there is a growing need to assess their support of pluralistic viewpoints. Most pluralism benchmarks are insufficient for this: they are not grounded in diversity (i.e., stakeholders across different frames, skills and needs), quality (i.e., rigorous and technical debates), and impact (i.e., invested deliberators and real-world outcomes). To address this, we first (i) curate DOCKET, the most comprehensive dataset of the U.S. Federal Notice & Comment process to date. DOCKET consists of 42.3M public comments on 22,400 regulations across 44 government agencies between 2016 and 2026 and contains millions of policy debates; we link public comments to particular policy items proposed by regulators to track the public's attention and the responsiveness of regulators to public input. (ii) Next, we demonstrate the pluralistic utility of DOCKET - we show that public commenters have a wide breadth of perspectives with significant depth in policy expertise and that regulators are often responsive. (iii) Finally, we present DOCKETBench, the first systematic evaluation of LLMs' ability to represent human policy perspectives in federal rulemaking. Only 18-30% of LLM-generated feedback aligns with real human comments. LLMs critique proposals broadly but generally, lacking the specific grounding, citations, and lived-experience arguments that humans bring to bear; in addition, most models we test systematically agree with government perspectives (overlapping with up to 68% of their claims) and agree least with labor unions (as little as 26% of their claims), demonstrating a fundamental skew in perspectives that even tools like persona modeling do not solve. Taken together, we reveal fundamental gaps in how language models navigate real policy issues, and DOCKETBench serves as a testbed for closing those gaps in a crucial real-world domain.

Closest papers

Nearest neighbours by meaning across the whole index.

Who Gets Heeded? An Obligation-Level Audit of Responsiveness in EPA Rulemaking

cs.CY cs.CL ·2026-08-11 ·Jianing Fan, Yue Yao
publishedcode/data similarity 0.71
Abstract · Camera-ready version, ACM EAAMO '26 (oral). Code and data: https://github.com/brucefjn/who-gets-heeded

Notice-and-comment rulemaking gives any affected party the same formal right to influence federal regulation, but formal access is not substantive capacity to shape rule text. Existing measurement strategies operate at the rule or aggregate-corpus level, too coarse to capture the discrete regulatory obligations where commenters seek change. We introduce obligation-level responsiveness auditing, an auditable, AI-assisted framework for measuring whether public-comment engagement co-occurs with changes to specific regulatory obligations. The framework extracts proposed and final-rule obligations, matches comments to the obligations they address, and classifies proposed-to-final outcomes; we test each of these AI-assisted components against blind human judgment. We apply the framework to 70,075 comments across 36 EPA anchor rulemakings, drawn from a corpus of 786,197 comments across 6,145 dockets from 2010-2022. Three descriptive findings emerge. First, engagement is associated with revision across rulemakings, but not clearly within the same rulemaking. Second, support-versus-opposition direction does not clearly differentiate outcomes, an informative null inconsistent with simple preference aggregation. Third, under a permissive reconstruction of commenter type, organizational-majority engagement concentrates in editorial-refinement rather than substantive-modification outcomes across rulemakings. A blind human audit of the outcome labels behind this finding preserves it under corrected labels and reveals that text-similarity methods are insufficient for distinguishing editorial from substantive regulatory change. Together, these findings locate the equity asymmetry upstream of agency response: in which rulemakings and obligations different commenter populations have the capacity to identify, interpret, and contest. Obligation-level auditing makes this upstream inequity visible.

0.48

Honest Brokers in AI Regulation: How Public Comments Introduce Evidence to Regulators

cs.AI ·2026-11-16 ·Sarah Tran, Anna-Maria Gueorguieva, Eva Maxfield Brown, Nicholas Weber
similarity 0.66
Abstract

The regulation of Artificial Intelligence is susceptible to various forms of capture by industry actors that hold undue influence over the policymaking process, due in part to their information advantage in developing these technologies. In the following paper, we investigate the roles of public commenters in providing evidence to potential regulation in the USA. We build a corpus of 40 potential regulations, and 2,664 public comments submitted by a variety of stakeholders. We computationally process these texts to extract evidence meant to inform regulators, in the form of formal citations and informal references. We find significant differences in the type and amount of evidence supplied: Holding comment length constant, business commenters introduce altogether less evidence, and government, university, and civil society commenters provide significantly more research and legal sources. The publication status of cited research is nearly evenly split, with preprints and peer-reviewed articles being introduced by commenters of all types. We further explore patterns of self-citation and find that while it is prevalent amongst all groups, a small cluster of organizations account for over half of all self-citations in our corpus. Our findings complicate a narrative that public comments alone are means to information capture. We interpret these results in terms of the role that commenters attempt to play in the policymaking process-being either an Issue Advocate or an Honest Broker. These findings on the supply of evidence in public comments shed light on an often overlooked form of policymaking that has implications for how and through what sources information is made accessible to regulators of AI.

0.23

Public Perceptions of Public Participation in Local Government Mediated through a GenAI Tool

cs.CY ·2026-11-05 ·Jason Yan, Yanna Krupnikov, Nicholas Weber, Sabina Tomkins
hot direction similarity 0.66
Abstract

City council meetings offer a site for direct democracy through public comment periods. Yet, this process can lack transparency, and research on local political participation has documented that people who speak at public meetings are often unrepresentative of their communities. However, a lack of representation does not establish whether other people view these participants as advocates for their concerns. In fact, little is known about how they are perceived. To address this gap and improve transparency into this democratic process, we built a generative AI (GenAI) tool, VoicesView, that allows individuals to retrieve and read public comments. We deploy this tool, along with a novel survey, in a study with 72 participants in Ann Arbor, Michigan, and Seattle, Washington, in order to understand people's perceptions of public commenters. We find that participants generally viewed public commenters favorably, and that these views became more positive after using VoicesView. For example, the share who agreed that public commenters advocate for issues they care about rose from 49% to 62%. Knowledge of how public comment works in their city was moderate and also improved: the proportion of participants who correctly answered a procedural knowledge question about council meetings increased from 53% to 74%. Participants viewed public commenters more favorably than local government, yet their sense of being heard by local government also grew: agreement that their own voice is heard rose from 28% to 46%. These findings suggest that giving residents opportunities to engage with and learn from records of local political participation can foster positive civic outcomes.

0.32

Scaling public input on AI policy with pluralistic agents

cs.CY cs.AIcs.CLcs.LG ·2026-11-11 ·Jacy Reese Anthis, Ken Burke, David Madras, Crystal Qian, Meredith Ringel Morris +3 more
similarity 0.64
Abstract

AI is being deployed across a vast range of application domains, cultural environments, and human stakeholders. Safe and responsible AI development requires continuously accounting for diverse perspectives. We develop and test pluralistic agents, AI agent representatives that deliberate on behalf of individuals to align AI systems with diverse human perspectives. Pluralistic agents could help people contribute to many more discussions than they could directly participate in, on technical topics that would require training, and on policy choices that rely on private information that corporations and governments cannot readily share. Our experiments identify opportunities, risks, and perceptions of pluralistic agents. Agents were able to reliably shift policies in the direction of their represented persona's preferences, but the extent of each agent's influence was sensitive to the instructions from its persona, such as having more influence when instructed to be more verbose or less cooperative. This suggests that influence in AI-assisted deliberation could vary based on how participants prompt their agents and how the system is designed. In a preregistered study, 746 humans participated in the pluralistic agents process. Participants viewed pluralistic agents as useful and fair, preferring agents over the baseline of providing input to a single AI reasoning model. Most participants preferred participating themselves over either pluralistic agents or the baseline, but pluralistic agents recovered 68% of the baseline-direct gap in reported satisfaction and 82% in fairness. Because pluralistic agents can be used in many real-world contexts where direct discussion is not, they could be a promising tool to scale AI safety policy development. We encourage cautious exploration of this method, playing to its strengths as an amplifier of human input rather than a replacement.

0.37

Templated or fully synthetic? Prompt construction as a confound in measuring LLM political stance beyond writing assistance

cs.CL cs.CY ·2026-08-11 ·Ilias Chalkidis
position similarity 0.63
Abstract

Political stance detection in LLMs has long been dominated by closed-ended, multiple-choice political survey questions---originally designed for humans, and thus lacks the realism and nuance of human-AI interactions in the wild, while also being susceptible to sandbagging. The recent IssueBench framework substantially mitigates these limitations with templated prompts anchored in real-world chat logs. Given the rise in non-work-related use of GenAI assistants, we extend IssueBench beyond writing assistance to include two additional tasks, information seeking and opinion sharing. We argue that templated prompts still lack the nuance of real ones, especially for open-ended tasks, and remain recognisable as evaluation artefacts. We propose the use of fully synthetic (LLM-generated) prompts, produced under detailed instructions with real prompts as seeds. We assess the ecological validity of real, templated, and LLM-generated prompts in a small-scale study covering 3 highly contested policy issues and 3 recent geopolitical conflicts. Human and LLM annotators rank LLM-generated prompts as no less realistic than real ones and clearly more realistic than templated ones, and find that they carry their intended intent and stance more clearly; the LLMs separate templated prompts from the other two far more sharply than the humans do. In a case study, templated and LLM-generated prompts yield systematically different stance estimates for the same model, most visibly under neutral framings, where templated prompts overstate the model's leaning in the direction encoded by the topic-and-stance text (filler) slotted into their templates.

0.04

AI-Voyager: A Public-Interest System for Human-AI Civic Reasoning to Contest and Shape the Rules That Govern AI

cs.AI cs.CLcs.HC ·2026-11-12 ·Snehalkumar `Neil' S. Gaikwad, Dhruv Dixit, Bryen Alvarez, Pratik Patekar, Anika Sharma +2 more
benchmarkhot direction similarity 0.62
Abstract

Governments are writing rules for artificial intelligence that decide what counts as an automated decision or a harm, often before the people they affect have a chance to question them. We introduce AI-Voyager, a public-interest system for human-AI civic reasoning, in which people and AI work together to understand, contest and shape the rules that govern AI. A resident can find their city's rules on AI in hiring or upload their country's AI strategy, read it as an affected worker would, open the passage behind any answer, and draft a revision. Its architecture turns 3,653 governance documents from city, state, federal and national governments into passages that keep their page and edition, and retrieves evidence through bounded agentic search that verifies each statement. On 360 new questions that require finding the relevant documents, it reached 83.0% macro recall, against 80.5% for the strongest baseline, 73.7% without iterative search and 99.1% on a benchmark with fixed documents. In a study with 24 participants new to two AI bills, most proposed changes asked how provisions would apply. Participants chose lenses for the kind of explanation they wanted, and citations signaled grounding but rarely led them to check the source. By binding AI assistance to the exact text of public documents, this work introduces a grammar of operations and a design space that inform how such systems are designed and evaluated, and how people take part in governing AI.

0.26

Beyond accuracy: what AI assurance needs to establish in public services

cs.CY cs.CL ·2026-11-04 ·Louise Yorke
similarity 0.62
Abstract · 20 pages, 6 tables, 2 figures

Public bodies increasingly rely on AI-produced evidence, including classifications, summaries and assessments. These outputs can inform policy decisions and service delivery, making their reliability a matter of public accountability. Establishing whether this evidence can be relied upon requires testing its outputs and examining how organisations decide whether those tests are sufficient. We combine technical evaluation with analysis of AI governance requirements, informed by measurement theory and management-based regulation. We compare eight GPT-4o configurations classifying 1,200 UK Prime Minister's Questions into 21 policy topics, against human-coded labels and across unchanged repeats. Accuracy ranges from 68.2% to 70.3%, yet 24.3% of questions receive different labels, with changes in topic shares and rankings. Configurations agree more with one another than with human coding: agreement does not guarantee correctness. Under the implemented score-extraction procedure, the proportion flagged for human review ranges from 0.9% to 41.0%. We analyse 37 UK and EU policy documents. Both sets expect testing and oversight, but reporting expectations do not consistently cover changing answers, shared errors, selective evaluation, substantive conclusions or review routing. Some UK guidance explicitly addresses stability. The UK sample emphasises public bodies checking their own use; the EU places substantial testing duties on providers while leaving them important choices about measures and acceptance thresholds. Without appropriate checks, evidence could misrepresent policy issues, redirect attention or resources, and leave errors undetected. Public policy should require organisations to justify reliance through checks beyond aggregate accuracy, clear responsibility, access to expertise and scrutiny of results.

0.24

OmouAI: Argumentative Human-AI Policy Deliberation with Simulated Personas

cs.AI ·2026-09-25 ·Stylianos Loukas Vasileiou, Antonio Rago, William Yeoh, Georgina Curto
similarity 0.61
Abstract

Debates amongst agents driven by large language models (LLMs) have demonstrated vast potential in various applications, but when these interactions include humans and take place in high-stakes environments, e.g., in public policy deliberations, they are beset with issues such as sycophancy and a lack of faithful explanations. To tackle these issues, we present OmouAI, an interactive and inclusive deliberation system that uses LLMs in combination with computational argumentation, a field which excels in representing and reasoning within debates. OmouAI allows a human user to deliberate policy claims for real-world challenges with simulated personas, e.g., representing stakeholders, domain experts or devil's advocates, towards reducing sycophancy. Each persona generates its own arguments, and the arguments of all parties form a shared argumentation framework. Users can then contest, add and revise arguments, providing crucial human oversight. Then, arguments are evaluated using deterministic argumentative semantics against external goals, such as the UN Sustainable Development Goals, guaranteeing faithful explanations. The advancement or worsening of the goals thus serve as indicators for the policy recommendations.

0.06