CS Paper Radar

Who Gets Heeded? An Obligation-Level Audit of Responsiveness in EPA Rulemaking

Jianing Fan, Yue Yao

cs.CY cs.CL ·submitted 2026-08-11 00:05 UTC · v2
publishedcode/data

Abstract

Notice-and-comment rulemaking gives any affected party the same formal right to influence federal regulation, but formal access is not substantive capacity to shape rule text. Existing measurement strategies operate at the rule or aggregate-corpus level, too coarse to capture the discrete regulatory obligations where commenters seek change. We introduce obligation-level responsiveness auditing, an auditable, AI-assisted framework for measuring whether public-comment engagement co-occurs with changes to specific regulatory obligations. The framework extracts proposed and final-rule obligations, matches comments to the obligations they address, and classifies proposed-to-final outcomes; we test each of these AI-assisted components against blind human judgment. We apply the framework to 70,075 comments across 36 EPA anchor rulemakings, drawn from a corpus of 786,197 comments across 6,145 dockets from 2010-2022. Three descriptive findings emerge. First, engagement is associated with revision across rulemakings, but not clearly within the same rulemaking. Second, support-versus-opposition direction does not clearly differentiate outcomes, an informative null inconsistent with simple preference aggregation. Third, under a permissive reconstruction of commenter type, organizational-majority engagement concentrates in editorial-refinement rather than substantive-modification outcomes across rulemakings. A blind human audit of the outcome labels behind this finding preserves it under corrected labels and reveals that text-similarity methods are insufficient for distinguishing editorial from substantive regulatory change. Together, these findings locate the equity asymmetry upstream of agency response: in which rulemakings and obligations different commenter populations have the capacity to identify, interpret, and contest. Obligation-level auditing makes this upstream inequity visible.

Comment
Camera-ready version, ACM EAAMO '26 (oral). Code and data: https://github.com/brucefjn/who-gets-heeded
Journal ref
Proceedings of the 6th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO '26), Munich, Germany, 2026
DOI
10.1145/3846167.3848608

Closest papers

Nearest neighbours by meaning across the whole index.

DOCKET: A Pluralistic Benchmark for Evaluating AI-Assisted Policy Feedback in U.S. Rulemaking

cs.CY ·2026-11-13 ·Alexander Spangher, Charlotte Peale, Sílvia Casacuberta, Victor Y. Wu, Mirac Suzgun +6 more
benchmark similarity 0.71
Abstract

As AI models advance and impact deliberative democracies, there is a growing need to assess their support of pluralistic viewpoints. Most pluralism benchmarks are insufficient for this: they are not grounded in diversity (i.e., stakeholders across different frames, skills and needs), quality (i.e., rigorous and technical debates), and impact (i.e., invested deliberators and real-world outcomes). To address this, we first (i) curate DOCKET, the most comprehensive dataset of the U.S. Federal Notice & Comment process to date. DOCKET consists of 42.3M public comments on 22,400 regulations across 44 government agencies between 2016 and 2026 and contains millions of policy debates; we link public comments to particular policy items proposed by regulators to track the public's attention and the responsiveness of regulators to public input. (ii) Next, we demonstrate the pluralistic utility of DOCKET - we show that public commenters have a wide breadth of perspectives with significant depth in policy expertise and that regulators are often responsive. (iii) Finally, we present DOCKETBench, the first systematic evaluation of LLMs' ability to represent human policy perspectives in federal rulemaking. Only 18-30% of LLM-generated feedback aligns with real human comments. LLMs critique proposals broadly but generally, lacking the specific grounding, citations, and lived-experience arguments that humans bring to bear; in addition, most models we test systematically agree with government perspectives (overlapping with up to 68% of their claims) and agree least with labor unions (as little as 26% of their claims), demonstrating a fundamental skew in perspectives that even tools like persona modeling do not solve. Taken together, we reveal fundamental gaps in how language models navigate real policy issues, and DOCKETBench serves as a testbed for closing those gaps in a crucial real-world domain.

0.31

What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models

cs.AI ·2026-08-17 ·Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth
benchmark similarity 0.65
Abstract

Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy. Such monitoring is meaningful only if a detector's verdict depends on the stated rule rather than on surface features of the scenario. We show this condition fails across the current class of compliance detectors, a failure we call rule blindness. Deleting, permuting, or substituting the governing rule leaves detection accuracy unchanged for every guard and activation probe we test, including a policy-conditioned guard that correctly cites the governing clause yet barely changes its verdict when that clause is swapped for its permissive counterpart. A purpose-built benchmark crossing two rules with two scenarios, so that neither alone predicts the label, confirms the failure under a design no prior benchmark rules out, and shows that step by step reasoning, not any fast detector we test, is what escapes it. Auditing at scale requires a retraining-free detector, so we introduce the Internal Compliance Score (ICS): a training-free activation readout calibrated from ten labelled pairs and scored by a single projection. We hold ICS to the same scrutiny as the guards it audits: a pre-registered criterion for beating trivial baselines is not met, and a bag-of-words model matches its pooled generalisation exactly. It remains useful because it is inexpensive, letting us audit four deployed guard models, an 8B zero-shot judge, and thirteen benchmarks, and it raises the mechanically verified pass rate when used to rank candidate responses, though an adaptive white-box attack removes this gain. We release the counterfactual protocol and crossed-rule benchmark so rule blindness can be tested in future probe and guard claims.

0.33

ARCCS: An Automated Regulatory Compliance Checking System

cs.CL ·2026-10-01 ·Giorgos Filandrianos, José Menezes, Chrysoula Zerva, Alessandro Gianola
top venuebenchmarkcode/data similarity 0.64
Abstract · This is the extended version of a paper accepted to EMNLP 2026 (System Demonstrations)

Regulatory compliance checking - deciding whether a target document satisfies the obligations of a regulation - requires interpreting dense legal text, identifying which provisions apply, and grounding each decision in explicit evidence. We present ARCCS, an end-to-end, automated, agentic, and regulation-agnostic Legal NLP system for compliance checking. ARCCS decomposes raw regulatory text into atomic, traceable requirements and evaluates a target document against them using retrieved evidence, confidence scores, and human-interpretable justifications. This design decouples compliance assessment from any fixed regulatory template or predefined rule set, enabling the pipeline to operate over regulations of varying size and structure. We evaluate ARCCS in two complementary settings. First, in a GDPR policy-document evaluation, LLM-based judges find its decisions and justifications legally and evidentially consistent in up to 96.67% of the assessed cases. Second, on an EU public-procurement benchmark comprising more than 1,200 individual rule checks, the system attains 98.8% accuracy in violation detection. ARCCS is, to our knowledge, the first fully open-source system for end-to-end regulatory compliance checking and auditable report generation.

0.45

Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluations

cs.CL ·2026-09-25 ·Yonghong Zhang, Yong Xie, Isabel M. Parra, Ricardo Correia
benchmarkcode/data similarity 0.63
Abstract · Accepted at ClimateNLP 2026, the 3rd Workshop on Natural Language Processing meets Climate Change (EMNLP 2026). 9 pages plus appendix (21 pa…

Difference-in-differences (DID) studies are widely used to evaluate climate policy, but assessing the evidence supporting their identification assumptions remains challenging. We introduce ARGUS, a structured language-model pipeline that audits reported evidence against an eleven-dimension assumption-implication-evidence rubric and abstains when relevant evidence cannot be retrieved. We evaluate ARGUS using injected flaws, economics papers, and a small pilot with reconciled labels. On the 11-flaw benchmark, ARGUS detects 73% of planted flaws, compared with 18% for a keyword-based pipeline. Across 26 economics papers, ARGUS abstains on about 40% of paper-dimension assessments for lack of retrievable evidence. In a five-paper pilot with labels reconciled by two annotators, it assigns a higher risk level than the labels on 25 of the 33 assessments it completes. A rule fixed before the labels arrived removes most of this in-sample; weighted agreement stays low. ARGUS provides evidence-linked risk reports that localize potential weaknesses for expert review, without adjudicating causal claims. Code and data: https://github.com/yonghongzhang-io/ARGUS

0.46

Honest Brokers in AI Regulation: How Public Comments Introduce Evidence to Regulators

cs.AI ·2026-11-16 ·Sarah Tran, Anna-Maria Gueorguieva, Eva Maxfield Brown, Nicholas Weber
similarity 0.62
Abstract

The regulation of Artificial Intelligence is susceptible to various forms of capture by industry actors that hold undue influence over the policymaking process, due in part to their information advantage in developing these technologies. In the following paper, we investigate the roles of public commenters in providing evidence to potential regulation in the USA. We build a corpus of 40 potential regulations, and 2,664 public comments submitted by a variety of stakeholders. We computationally process these texts to extract evidence meant to inform regulators, in the form of formal citations and informal references. We find significant differences in the type and amount of evidence supplied: Holding comment length constant, business commenters introduce altogether less evidence, and government, university, and civil society commenters provide significantly more research and legal sources. The publication status of cited research is nearly evenly split, with preprints and peer-reviewed articles being introduced by commenters of all types. We further explore patterns of self-citation and find that while it is prevalent amongst all groups, a small cluster of organizations account for over half of all self-citations in our corpus. Our findings complicate a narrative that public comments alone are means to information capture. We interpret these results in terms of the role that commenters attempt to play in the policymaking process-being either an Issue Advocate or an Honest Broker. These findings on the supply of evidence in public comments shed light on an often overlooked form of policymaking that has implications for how and through what sources information is made accessible to regulators of AI.

0.22

Policy-Conditioned AI-Use Detection: An Evidentiary Framework for Academic Publishing

cs.CL cs.AIcs.CY ·2026-09-29 ·Jairo Diaz-Rodriguez, Mumin Jia
publishedposition similarity 0.61
Abstract · Accepted at NeurIPS 2026 AI-Native Academia Workshop

Major venues now publish detailed rules about how authors, reviewers, and area chairs may use AI, and those rules differ by role, by task, and by what must be disclosed. AI detection, the instrument usually proposed to enforce them, estimates something else: whether an AI model wrote the text. We argue that this target is misaligned with the decisions conferences and journals face, and propose policy-conditioned AI-use detection, an evidentiary framework for assessing whether a human--AI workflow complied with a stated rule. Policy makes the governing rule an explicit input. Inference reports hypotheses, evidence, calibration regime, and uncertainty in place of verdicts such as "AI detected". Evaluation builds benchmarks from reproducible pipelines that generate compliant and non-compliant workflows, and reports true positive rate at a false positive rate the venue fixes in advance. We work the framework through peer review, where at plausible violation rates a detector at a strong operating point still flags more compliant authors than violating ones. The framework therefore also names what a venue must instrument: structured disclosure, approved-tool routing that respects reviewer confidentiality, and a path by which a finding can be contested. Under this framing a detector is not an authorship classifier but an auditable procedure with an error rate the venue fixes in advance and can defend.

0.26

Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error

cs.CL ·2026-09-13 ·Hongliu Cao
benchmark similarity 0.61
Abstract · Accepted at REALM EMNLP 2026

Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two $τ^2$-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.

0.26

Policy Over Purpose: Auditing LLM Content Moderation of Reclaimed Slur Usage

cs.CL ·2026-11-06 ·Christina Chance, Kai-Wei Chang
position similarity 0.61
Abstract

As Perspective API is deprecated, platforms are turning to large language models for content moderation, encoding policy as natural language that the model must interpret at inference time. Reclaimed slurs test this design directly. The same term can express in-group pride or target the community that reclaimed it, and misreading it silences the speech moderation is meant to protect. We audit four general-purpose LLMs and four moderation-specific systems, with Perspective API as a legacy baseline, on IYKYK, a corpus of reclaimed and derogatory usage of three slurs annotated by community members. Across the policy, decision, reasoning, disparity, and steering layers of this audit, we find that whether a system was purpose-built for moderation explains little about how it handles reclaimed language. No system reliably agrees with community judgment, and all fail most on language reclaimed within Black communities. What separates systems is instead how their policy is represented. Only systems whose policy is expressed in the prompt can be steered at all, and even then, explicit definitions change how models justify a decision more than how they make it. We argue that policy transparency determines whether moderation failures can be corrected, but that correcting them requires contextual judgment that definitions supplied at inference time cannot provide.

0.25