CS Paper Radar

Measuring the Inaccessible: Can AI Evaluate Accessibility in Heritage Tourism?

Sabrina Armenio, Manuele Leonelli

cs.CY ·submitted 2026-10-28 10:53 UTC · v1
position

Abstract

Accessibility assessment underpins the right to participate in heritage tourism, yet evaluation of archaeological sites remains artisanal: fieldwork does not scale, is often carried out without accessibility training, and is reported in forms that reach neither the sector nor disabled visitors. AI appears to offer a shortcut. We argue that assessment is measurement and inference under sparse, expert-mediated evidence: AI transforms how that evidence is captured and communicated, whereas evaluation, which turns evidence into defensible judgements, demands properties current generative systems lack. In a five-park field study in Spain and Italy, an AI desk audit scored only 66% of dimension-site combinations, could not observe embodied and organisational dimensions, and mirrored sites' online self-presentation. Simple Bayesian models instead quantify the confidence rankings deserve and calibrate desk evidence against fieldwork. We propose a separation of roles between AI and interpretable statistical evaluation as the basis for a reliable accessibility atlas.

Closest papers

Nearest neighbours by meaning across the whole index.

Agentic Web Accessibility Auditing: A Criterion-Specific Framework for Translating WCAG Requirements into Assessments

cs.HC ·2026-09-08 ·Arjun Mishra, Pranav Karthik, Byungjun Bae, Dongwook Yoon
dataset similarity 0.59
Abstract · Preprint. 26 pages, 7 figures. Revised title, abstract, conclusion, and exposition; clarified methods, dataset construction, and interpretat…

Web accessibility auditing requires interpreting diverse requirements and examining interface behavior. Rule-based checks and noninteractive model assessments can miss barriers requiring contextual or interactive evidence. We present an agentic framework that assigns a vision-language agent to each accessibility requirement. Guided by tailored instructions, agents inspect webpages, operate controls, and record evidence supporting their findings. We implement the framework for 40 requirements from the Web Content Accessibility Guidelines (WCAG). To compare detection and cost, we construct a dataset of 250 page-criterion records derived from expert audits across 11 scholarly platforms. Agents recover 67 of 78 reported positive cases (86% recall), compared with 36% for axe-core, a rule-based checker, and 67% for an uncued, noninteractive vision-language model, at lower precision (56%). They recover nine of ten Keyboard and No Keyboard Trap cases missed by both baselines. Together, the framework, implementations, and dataset support automated accessibility auditing grounded in inspectable evidence.

0.16

MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places

cs.AI ·2026-08-28 ·Jason Armitage, Ioannis Tsochantaridis, Linda Mazzone, Chuqiao Yan, Srini Narayanan +1 more
benchmark similarity 0.58
Abstract

We introduce MAP, the first benchmark to evaluate multimodal AI systems as assistants for users with accessibility requirements when planning visits to places in the real world. In our evaluation, systems are presented with requests to verify or recommend a point of interest meeting an accessibility requirement. MAP contains two novel assessments: Claim verification for accessibility planning assesses if information on places and stated accessibility features is supported and identifies places that satisfy requested accessibility features. Visual evidence retrieval for accessibility planning checks if a multimodal AI system can select visual evidence for the requested place and accessibility feature. Our methodology supports comparison of AI systems in a setting where place information and accessibility information can change over time by evaluating systems and refreshing ground truth data at scheduled times. The benchmark is based on automatic rating and human rating for a proportion of responses.

0.27

InterA11y: Interactive Web Accessibility Error Detection with LLM Agents

cs.HC ·2026-10-29 ·Mingyuan Zhong, Mengqi Shi, Davin Win Kyi, Ajit Mallavarapu, Xia Chen +2 more
code/data similarity 0.57
Abstract · We release our code and data at https://github.com/jszh/InterA11y

Large Language Models (LLMs) have shown potential to expand the coverage of web accessibility evaluation. However, current LLM-based checkers neglect much of the interactive and dynamic web contexts, resulting in limited coverage, high false discovery rates, and inconsistent performance across models. We present InterA11y, an agentic accessibility evaluation system that combines rubrics and interactive tools to support LLM agents in evaluating accessibility effectively. The rubrics decompose WCAG success criteria (SC) into testable aspects and guide agents in collecting and interpreting evidence. The interactive tools inspect or operate a webpage deterministically to collect evidence, such as focus appearances and error messages. We developed these rubrics and tools on 682 curated test cases grounded in established accessibility guidelines that target 10 commonly violated SC. We evaluated InterA11y on 197 expert-annotated cases sampled from 54 real webpages. Results show InterA11y achieves higher recall than the strongest evaluated LLM-driven baseline (79.0% vs. 44.4%), while maintaining comparable precision to best rule-based checkers (93.1% vs. 90.3-97.6%), with little variability between models. Further, when equipped with our interactive tools, general-purpose web agents detect more accessibility errors with fewer false positives and less token usage than without these tools.

0.33

(Whose defaults?) Is artificial intelligence reorienting archaeological methods?

cs.CY cs.AIcs.CLcs.HC ·2026-09-10 ·Lorenzo Cardarelli, Roberto Ragno
similarity 0.56
Abstract

Generative AI and the practice of "vibe coding" are changing how archaeologists carry out computational research, but their effects on the discipline's range of methods is still understudied. In this paper, we evaluate whether large language models (LLMs) are narrowing the variety of methods archaeologists use. We first analysed approximately 119,000 archaeology abstracts from Scopus, covering publications from 2010 to 2025. Using a locally run LLM, we identified the computational methods reported in each abstract and organised them into 25 broad categories (L2) and 241 finer clusters (L3). A Bayesian Dirichlet-multinomial model of method composition within sub-disciplines found a small but credible shift in method use after 2023. However, this shift was smaller than the variation already present across the full study period. No individual technique showed a significant change, and overall methodological diversity increased rather than declined. We then ran a controlled experiment to see whether LLMs recommend a narrower set of methods than archaeologists have used in practice. Two different open-weight models were asked to suggest methods for 28 archaeological research problems, with prompts providing three levels of methodological guidance: novice, intermediate, and expert. Recommendation diversity was much lower than in the published literature, particularly without methodological guidance. The models also tended to favour methods that were widely used before 2023, and their recommendations more closely resembled the post-2023 literature. Taken together, these results are consistent with LLMs pushing methodological choice towards convergence, although our study cannot establish a causal effect. They raise a broader question: how can archaeology retain methodological diversity as LLMs become more involved in research?

0.25

Choose your own adventure: How data source selection shapes urban accessibility analyses

cs.CY stat.APecon.GN ·2026-10-19 ·Geoff Boeing, Jiyoon Kim, Daniele Simon-Thompson, Jaehyun Ha, Ruoyu Chen
published similarity 0.52
Abstract

Urban park accessibility shapes mental health, physical well-being, and community cohesion. Measuring it is thus central to urban and landscape planning. Although researchers have debated the impacts of different accessibility measures on analysis results, scant attention has been paid to variability from data source selection. Do the results of accessibility analyses vary depending on the dataset used to define park locations and attributes? We compare four widely used geospatial data sources - OpenStreetMap, Esri, SafeGraph, and The Trust for Public Land - by validating their data and measuring park access across two cities. These datasets purport to cover the same spatial features, but our findings reveal wide discrepancies in accessibility results, driven by differences in dataset completeness, coverage, and accuracy between the sources. We provide guidelines for selecting and carefully validating data for accessibility analyses, as these decisions directly impact the accuracy and equity of downstream urban park planning and decision-making.

0.41

Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

cs.CL cs.AIcs.CYcs.ET+1 ·2026-10-13 ·Mir Tafseer Nayeem, Davood Rafiei
benchmark similarity 0.51
Abstract · Preprint

Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define successful performance. We introduce a user-centric audit framework that distinguishes users who contribute benchmark content from those whose needs evaluation results are intended to represent. The framework operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures. Across 16 resources, we find that American-centered conventions account for 62.4-94.4% of matched signals in 11 text benchmarks, while geographic visibility in five image-text resources is concentrated and strongly associated with national GDP share (mean Pearson r = 0.780; Spearman p = 0.778). Documentation reveals a systematic source-target gap: all 16 resources operationalize source-user context, but none operationalize target-user context. Paired scoring experiments further show that meaning-preserving answer variants reduce DROP exact-match accuracy from 100.0% to 33.3%, while TruthfulQA exhibits metric-dependent reductions in reference similarity. Our framework and documentation checklist make user assumptions auditable, clarifying what benchmark scores represent and where evaluation claims are supported.

0.26

Passable for Whom? Few-Shot Learning for Personal Calibration of Sidewalk Accessibility Perception for Human-Centered Autonomous Driving

cs.CV cs.LG ·2026-11-15 ·Wesley Maia, Angel Martinez-Sanchez, Maitrayee Keskar, Ross Greer
similarity 0.51
Abstract

Whether a sidewalk is passable depends on who is trying to pass: people who use the same mobility aid disagree about the same image, so one label per scene, or per aid, is wrong for many of them. We introduce few-shot personal accessibility perception: from K answers of a person never seen before, predict their judgment of sidewalk scenes never seen before. PassCal separates a shared description of the scene from the person's interpretation of it. The description comes from the internal state of a frozen vision-language model (VLM) and from barrier type, severity and context properties predicted by a detector trained on 14.9k public audits; the interpretation, a calibration of the person's response style and a weighting of the properties, is fit from their answers. On 8,883 judgments in which each of 135 verified mobility-aid users rated the same 50 sidewalk scenes, with people and scenes both held out, eight answers lower the Brier score from 0.550 to 0.479. The same VLM prompted with those answers reaches 0.562, and the same calibration without the scene 0.521. Conformal risk control turns predictions into passable, blocked or "depends on you" statements with a bounded expected rate of wrong statements over people; at 5%, eight answers raise the share of decisive scenes from 27% to 42%. Audit-supervised properties improve all sixteen vision and VLM encoders we test and outperform attributes from a sidewalk navigation VLM.

0.23

Distilling Expert Judgment at Scale

cs.AI ·2026-10-23 ·Robbie Goldfarb, Andrew B. Hall, Jillian Fisher, Kathryn Salam, Matt Wilde
benchmark similarity 0.51
Abstract · 9 pages

Frontier AI models are increasingly deployed on topics where the stakes are high and expert judgment is required, including healthcare, defense, finance, governance, and politics. We present a general methodology for distilling expert human judgment into automated evaluation systems that operate at scale. Our approach assembles networks of domain experts and develops techniques for extracting not just their conclusions but their reasoning processes, encoding those processes into structured editorial standards that automated judges can apply continuously. We ground our approach in a long tradition of research on the nature of expertise and the conditions under which expert judgment is reliable and more effective than other approaches, and argue that independent, expert-calibrated evaluation is necessary not only on epistemic grounds but as a matter of accountability and public trust. We validate the methodology on AI news reporting as a first application domain. On source quality, two independent domain experts achieve Krippendorff's $α= 0.83/0.69$ on a held-out test set when applying the editorial standards, and our automated judge reaches $F1 = 0.81/0.97$ against expert consensus. On neutrality, experts achieve $α= .72$ on the test set, and the automated judge reaches $F1 = 0.86$ against expert consensus. Additionally, a controlled comparison shows that expert-calibrated judges substantially outperform uncalibrated frontier models on every source-quality metric, demonstrating that encoding expert reasoning is the critical step that prompt engineering alone cannot replace. We present this benchmark as a proof of concept for a broader evaluation infrastructure applicable to any domain where judgment matters and expertise is scarce.

0.32