CS Paper Radar

Agentic Web Accessibility Auditing: A Criterion-Specific Framework for Translating WCAG Requirements into Assessments

Arjun Mishra, Pranav Karthik, Byungjun Bae, Dongwook Yoon

cs.HC ·submitted 2026-09-08 19:27 UTC · v2
dataset

Abstract

Web accessibility auditing requires interpreting diverse requirements and examining interface behavior. Rule-based checks and noninteractive model assessments can miss barriers requiring contextual or interactive evidence. We present an agentic framework that assigns a vision-language agent to each accessibility requirement. Guided by tailored instructions, agents inspect webpages, operate controls, and record evidence supporting their findings. We implement the framework for 40 requirements from the Web Content Accessibility Guidelines (WCAG). To compare detection and cost, we construct a dataset of 250 page-criterion records derived from expert audits across 11 scholarly platforms. Agents recover 67 of 78 reported positive cases (86% recall), compared with 36% for axe-core, a rule-based checker, and 67% for an uncued, noninteractive vision-language model, at lower precision (56%). They recover nine of ten Keyboard and No Keyboard Trap cases missed by both baselines. Together, the framework, implementations, and dataset support automated accessibility auditing grounded in inspectable evidence.

Comment
Preprint. 26 pages, 7 figures. Revised title, abstract, conclusion, and exposition; clarified methods, dataset construction, and interpretation of findings. Supplementary material and supporting research records included as ancillary files

Closest papers

Nearest neighbours by meaning across the whole index.

InterA11y: Interactive Web Accessibility Error Detection with LLM Agents

cs.HC ·2026-10-29 ·Mingyuan Zhong, Mengqi Shi, Davin Win Kyi, Ajit Mallavarapu, Xia Chen +2 more
code/data similarity 0.86
Abstract · We release our code and data at https://github.com/jszh/InterA11y

Large Language Models (LLMs) have shown potential to expand the coverage of web accessibility evaluation. However, current LLM-based checkers neglect much of the interactive and dynamic web contexts, resulting in limited coverage, high false discovery rates, and inconsistent performance across models. We present InterA11y, an agentic accessibility evaluation system that combines rubrics and interactive tools to support LLM agents in evaluating accessibility effectively. The rubrics decompose WCAG success criteria (SC) into testable aspects and guide agents in collecting and interpreting evidence. The interactive tools inspect or operate a webpage deterministically to collect evidence, such as focus appearances and error messages. We developed these rubrics and tools on 682 curated test cases grounded in established accessibility guidelines that target 10 commonly violated SC. We evaluated InterA11y on 197 expert-annotated cases sampled from 54 real webpages. Results show InterA11y achieves higher recall than the strongest evaluated LLM-driven baseline (79.0% vs. 44.4%), while maintaining comparable precision to best rule-based checkers (93.1% vs. 90.3-97.6%), with little variability between models. Further, when equipped with our interactive tools, general-purpose web agents detect more accessibility errors with fewer false positives and less token usage than without these tools.

0.33

Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories

cs.CL ·2026-10-01 ·Chengguang Gan, Zimeng He, Yoshihiro Tsujii, Ken-ichiro Kobayashi, Hiroki Itoh +1 more
published similarity 0.68
Abstract · 13 pages, 1 figure, 10 tables. Accepted as a poster at the NeurIPS 2026 Workshop "Who Verifies the Agents? Toward Reliable Agent Development…

Web agents are an important application of large language models, yet their evaluation often depends on rule based or language model evaluators that inspect only the final outcome. Human verification of task completion and detailed analysis of failed trajectories remain limited. We audit all 165 WebArena Lite tasks under six evaluation conditions built from GPT 5.5 and an untrained Qwen3.5 9B model. The audit retains the original score, corrects false negatives from the automatic evaluator, identifies the first consequential error, and examines progress across the trajectory. We also study a Memory and Analysis Support Mechanism (MASM), which maintains explicit execution state, and Guide Text, which provides task relevant procedural guidance. Across four GPT 5.5 settings, human review recovers 5.45 to 8.49 percentage points of success missed by the evaluator. With a 25 step budget, Guide Text raises corrected success with MASM from 34.55% to 38.18%. On the untrained Qwen3.5 9B model, MASM raises the evaluator score from 13.90% to 18.80%. Review of 102 failed GPT 5.5 trajectories reveals frequent scrolling loops, unfinished exploration, premature answers, invalid actions, and incomplete form workflows. Step level evidence further shows that substantial early progress can coexist with a final failure. These results show why final scores alone provide an incomplete account of web agent behavior and motivate human grounded, trajectory aware verification.

0.38

A11yLTLNav: Automatic Detection of Accessibility Navigation Failures

cs.HC cs.SE ·2026-09-16 ·Chenming Ge, Kewen Peng, Chengyang Shi, Ben Greenman, Yue Jiang
similarity 0.65
Abstract

For blind and low-vision (BLV) screen-reader users, a website that appears accessible in a static snapshot can become difficult or impossible to navigate once interaction begins. Yet, most automated accessibility checkers miss failures involving focus, interface state, and accessible feedback across interactions. We present A11yLTLNav, a property-based approach for automatically detecting accessibility navigation failures. Through a structured review of prior research, we organize accessibility navigation failures into a failure taxonomy and formalize a browser-observable subset as executable Linear Temporal Logic properties over action-state traces. A11yLTLNav combines random keyboard exploration with runtime property monitoring to detect these failures during interactions. We evaluate A11yLTLNav on 31 generated websites based on real-world websites and tasks. It reported 309 accessibility failures, of which 274 were confirmed, achieving 88.7% precision and identifying more confirmed failures than the comparison checkers. Our results show that A11yLTLNav transforms accessibility knowledge into reusable checks of interface behavior over time.

0.20

From Subjective Judgments to Auditable Standards:Protocol-Guided AI Auditing of Website Redundancy

cs.SE cs.CV ·2026-08-21 ·Ge Kong, Yongtong Cao
benchmark similarity 0.62
Abstract

Website redundancy does not have a single fixed meaning. The same repeated element may distract during one task and provide backup during another. We introduce CORA (Counterfactual, Observable Redundancy Audit), which measures repetition load, normal-use tax, and failure-domain recovery reserve separately. Each run retains screenshots, stable element identities, and task traces. A versioned vision-language model proposes the annotations. Typed validation and release checks then determine whether a calibrated dimension can be reported; failed or malformed outputs stay in the fixed denominator. On a transparent mechanistic testbed, the factorized CORA representation separated reserve from normal-use tax and predicted perturbed success more accurately than scalar-load baselines. The model studies then showed why repeatability is not enough: two small local vision-language models produced recurring outputs, but neither instrument met all release requirements. CORA therefore withheld automated scores from both instruments while retaining the raw responses and failure records. Separate checker fixtures confirmed that the typed validator and hardened release gates implement their specifications; these tests do not establish semantic grounding or accuracy on production sites. Taken together, the results position CORA as an auditable candidate procedure for the controlled benchmark studied here rather than a general standard. Human agreement, AI-versus-human accuracy, and validation on independent production sites remain open empirical questions.

0.21

Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications

cs.HC cs.AI ·2026-09-01 ·Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan +1 more
top venue similarity 0.60
Abstract · Accepted to the EMNLP 2026 Main Conference

Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.

0.38

AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

cs.CL cs.AIcs.IRcs.SE ·2026-10-15 ·Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala +2 more
benchmarkcode/data similarity 0.60
Abstract · 25 pages, 2 figures, 13 tables. Code: https://github.com/Lexsi-Labs/AuditKIT. Shreeyans Arora and Utsav Avaiya contributed equally; work don…

Evaluating language models and agents requires combining benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality. Differences in interfaces, metric semantics, and trace formats complicate these comparisons and can hide failures in agentic workflows. We present AuditKIT, a source-available evaluation library that records these measurements in one result representation, with direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends. For agents, it provides turn-aware tool-call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory-level diagnostics. Missing or unobservable evidence is marked unscored rather than counted as failure. For retrieval-augmented generation, it combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability show how answer extraction, runtime behavior, and metric selection change evaluation outcomes.

0.32

Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2

cs.CL ·2026-10-22 ·Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan
publishedhot direction similarity 0.60
Abstract · Accepted to WaC Workshop at EMNLP 2026

Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. We use an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API and compiles each finding into an executable test, which a native speaker then validates. On the 26.5M-document Slovak portion of FineWeb2, three runs costing \$2.21-\$2.84 each found seven findings over four phenomena. The largest is a machine-generated content farm of 1,408,437 documents (5.32%) that passed the pipeline's filters, 86.6% of it passing even stricter thresholds. Blinded labels support 6 of the 7 findings (defect rates .53-1.00 against .08 in a random control) and do not support one. The compiled suite scans the full corpus in 0.7 s of wall-clock time without a GPU, against \$12k-\$90k per LLM classification pass. The best static sampling baseline validates more selectors than any agent run, at under half the cost. The agent contributes exact prevalence, a precision estimate checked against sampled documents, an account of why the pipeline missed each slice, and a reusable test. We release the framework, Slovak test suite, validation data and agent trajectories.

0.41

Measuring the Inaccessible: Can AI Evaluate Accessibility in Heritage Tourism?

cs.CY ·2026-10-28 ·Sabrina Armenio, Manuele Leonelli
position similarity 0.59
Abstract

Accessibility assessment underpins the right to participate in heritage tourism, yet evaluation of archaeological sites remains artisanal: fieldwork does not scale, is often carried out without accessibility training, and is reported in forms that reach neither the sector nor disabled visitors. AI appears to offer a shortcut. We argue that assessment is measurement and inference under sparse, expert-mediated evidence: AI transforms how that evidence is captured and communicated, whereas evaluation, which turns evidence into defensible judgements, demands properties current generative systems lack. In a five-park field study in Spain and Italy, an AI desk audit scored only 66% of dimension-site combinations, could not observe embodied and organisational dimensions, and mirrored sites' online self-presentation. Simple Bayesian models instead quantify the confidence rankings deserve and calibrate desk evidence against fieldwork. We propose a separation of roles between AI and interpretable statistical evaluation as the basis for a reliable accessibility atlas.

0.31