CS Paper Radar

Coral: Contextual Gists for Blind and Low Vision Screen Reader Users' Understanding of Dynamic User Interfaces

Ritesh Kanchi, Jianna So, Krzysztof Z. Gajos

cs.HC ·submitted 2026-09-19 06:04 UTC · v1
hot direction

Abstract

Blind and low vision (BLV) screen reader users construct mental models of user interfaces (UIs) through incremental screen reader interaction, a time-consuming and cognitively demanding process complicated by modern interfaces that may not convey dynamic content accessibly. We interviewed eleven BLV screen reader users about UI understanding and derived design objectives that informed Coral, a context-aware browser extension. Coral synthesizes interface and user context to generate screen reader-narrated gists which holistically notify users of interface states and changes relevant to their likely goals and ongoing interaction. In a comparative evaluation with eight BLV screen reader users, participants used Coral to form initial expectations and interpret interface changes, and reported spending less time and effort manually verifying interaction outcomes. Together, our findings provide deeper insight into how BLV screen reader users navigate gaps in their UI understanding, and how intelligent support that combines interface and user context can bridge these gaps.

Closest papers

Nearest neighbours by meaning across the whole index.

Spatial Organization Is Accessibility Structure: Hierarchical Screen Reading of Mobile UIs

cs.HC ·2026-11-02 ·Yongxiang Hu, Xuan Wang, Qingyang Qiu, Yingjie Liu, Chun Kang +3 more
similarity 0.69
Abstract · 26 pages, 6 figures, 7 tables

Screen readers for blind and low-vision (BLV) users flatten mobile UIs into a linear sequence. Modern mobile UIs, however, contain dozens of elements grouped by spatial layout. Such linear access is ill-suited for BLV users, who must infer each element's meaning and which elements form a functional unit. Drawing on a quantitative TalkBack audit and formative interviews, we present Spatial Screen Reader (SeeR), which builds a function-aware hierarchy from accessibility metadata and provides unit summaries with hierarchical navigation. Compared with TalkBack, SeeR reduces non-informative focus stops by 82.1% and navigation steps by 52.7%. With ten BLV participants, summaries helped them connect controls to their functions and skip nonessential elements, especially on unfamiliar pages, though extra navigation levels added effort when the target was already known. These findings recast spatial organization as accessibility information: apps and platforms should expose it, rather than leaving screen readers to infer it post hoc.

0.17

Q&A or Document-Based? The Effects of Interface Type on How Screen Reader Users Access Interconnected Documents

cs.HC cs.AIcs.IR ·2026-08-26 ·Colleen F. Cipriano, Yichun Zhao, Miguel A. Nacenta, Kotaro Hara, Jaylee Soh
top venue similarity 0.68
Abstract · 17 pages, 12 figures, accepted at ASSETS 2026

Blind and low-vision (BLV) users are increasingly engaging with large language model (LLM) interfaces to access documents, but it is unclear how such systems support or hinder their ability to build interconnected knowledge. To examine this gap, we compared a Question-Answer Interface (QAI) that supports open-ended conversational inquiry, with a Document Interface (DI) based mostly on traditional structured text document navigation. We recruited 16 BLV screen reader users where they used both interfaces to explore two fictional worlds. Data from interaction logs, concept maps, decision-based tasks, and semi-structured interviews provide comparative insights into how interface design supports knowledge construction. Findings show that participants visited more distinct documents with the DI and formed larger and more correct mental models with the DI than with the QAI. They were also more able to apply knowledge they had gained. Simultaneously, many still preferred the QAI and often estimated that they had explored more, formed better mental models and applied their models better when acquiring the information with the QAI, despite this not being the case. Our analysis suggests possible interface design reasons for these differences and highlights some of the risks introduced by using question-answer interfaces to access information spaces.

0.31

Affora: A Design System for Agent-Friendly Interfaces

cs.HC cs.AIcs.SE ·2026-09-16 ·Jin Gao
similarity 0.66
Abstract · 20 pages, 8 figures

Computer-use agents increasingly operate software designed for people, but interfaces often leave actions or task state unclear to machine readers. We present Affora, a design system that supports both readers while preserving visual freedom and familiar human workflows. Three controlled studies examine component implementations, visual variation, and interaction-design principles. Their findings inform guidance from individual components to complete sites, supported by reusable implementations and executable checks. Agent performance depends on the interaction meaning available through its interface representation; substantial visual variation remains possible when that meaning is preserved. Evaluation on independently authored interfaces shows gains where Affora addresses existing deficits, but limited effects where those deficits are absent or outside its coverage. A workflow case provides preliminary evidence of reduced interaction cost. Affora connects user experience and agent experience through a shared interface rather than a separate agent-only surface.

0.24

A11yLTLNav: Automatic Detection of Accessibility Navigation Failures

cs.HC cs.SE ·2026-09-16 ·Chenming Ge, Kewen Peng, Chengyang Shi, Ben Greenman, Yue Jiang
similarity 0.63
Abstract

For blind and low-vision (BLV) screen-reader users, a website that appears accessible in a static snapshot can become difficult or impossible to navigate once interaction begins. Yet, most automated accessibility checkers miss failures involving focus, interface state, and accessible feedback across interactions. We present A11yLTLNav, a property-based approach for automatically detecting accessibility navigation failures. Through a structured review of prior research, we organize accessibility navigation failures into a failure taxonomy and formalize a browser-observable subset as executable Linear Temporal Logic properties over action-state traces. A11yLTLNav combines random keyboard exploration with runtime property monitoring to detect these failures during interactions. We evaluate A11yLTLNav on 31 generated websites based on real-world websites and tasks. It reported 309 accessibility failures, of which 274 were confirmed, achieving 88.7% precision and identifying more confirmed failures than the comparison checkers. Our results show that A11yLTLNav transforms accessibility knowledge into reusable checks of interface behavior over time.

0.20

InSight: A Mobile Assistant Helping Visually Impaired People Navigate Unfamiliar Environment

cs.HC ·2026-10-13 ·Junwei Zheng, Jiale Wei, Di Wen, Yufan Chen, Ruiping Liu +4 more
similarity 0.62
Abstract

Navigating unfamiliar outdoor environments is one of the most demanding everyday tasks for people with visual impairments (PVI). However, existing navigation systems cannot simultaneously satisfy the needs of informative perception and precise localization during navigation. This paper explores the core navigation demands of PVI through a preliminary study. Based on the results of the pre-study, we present the design of InSight, a mobile assistant that combines scene descriptions with precise localization to support PVI navigating unfamiliar outdoor environments. The offline mode of InSight runs entirely on device for real-time navigation, while the online mode streams camera frames to a self-hosted server for arbitrary object detection and accurate localization. Finally, a user study with 21 visually impaired participants validates that InSight could assist users in navigating unfamiliar outdoor environments with increased success rate and confidence compared to mainstream navigation applications like Apple Maps.

0.18

Beyond Scene Description: Multi-Agent Orchestration for Non-visual Access to Virtual Worlds

cs.AI ·2026-09-13 ·Toqeer Ali Syed, Ali Akarma, Adeel Ahmad, Danial Hameed
similarity 0.62
Abstract · Paper submitted in ICAAD Conference 2026

Virtual worlds now host classrooms, meetings, conferences, shops, and social venues, and nearly every interaction they expose assumes a user who can scan a three-dimensional scene, follow avatars, and read floating panels. Blind and visually impaired (BVI) users are left with assistive tools that each solve one task in isolation: naming an object, reading text, describing a scene, or planning a route. A live virtual room defeats that model: obstacles, speakers, gestures, chat, slides, and notifications arrive together, and a tool that narrates all of them trades a visual barrier for an auditory one. This paper presents MetaBlind, an architecture that distributes nonvisual access across eight specialized agents, spanning perception, navigation, social and object interaction, communication, safety and trust, memory, and personalization, and that places an Accessibility Orchestrator between those agents and the user. Agents publish candidate information into a shared accessibility context instead of speaking to the user directly. The orchestrator scores each candidate on safety relevance, goal relevance, urgency, confidence, user relevance, and estimated listening load, then releases only the items it judges relevant at that moment through speech, structured audio, or haptic output. We give the selection step a formal statement, specify the orchestration cycle as an algorithm, and define an evaluation protocol against a single-agent assistant. MetaBlind is reported at the design stage, with no prototype measurement or user study, and the protocol states which outcomes would support the design and which would refute it.

0.31

Look Over There! Dynamic Visual Guidance for AI-Narrated Slides

cs.HC ·2026-10-30 ·Jiongning Allen Lian, Yutong Alison Mei, Ho Ting Cheung, MK Michael Cheung, Xiangzhi Eric Wang +5 more
similarity 0.62
Abstract · 21 pages, 7 figures

Slides present information spatially, but narration unfolds linearly, leaving the viewer struggling to locate what the narration refers to. This split attention between audio and visual channels forces viewers to search the slide manually, imposing extraneous cognitive load and disrupting comprehension. The problem is especially acute on mobile devices, where dense layouts compressed onto small screens make visual search particularly effortful. In teacher-led settings, however, instructors naturally direct attention through pointing, pacing, and emphasis. Inspired by this, we present GuideSlide, a system that automatically generates Dynamic Visual Guidance for slide presentations, providing viewers with narration-synchronized visual cues that direct attention in real time. Given a PDF slide deck, GuideSlide automatically produces region-level zoom and time-aligned highlights without manual annotation. In a within-subjects study with 24 participants, GuideSlide substantially reduced perceived cognitive load and yielded higher immediate comprehension scores compared to static slides with identical narration. Participants strongly preferred the Dynamic Visual Guidance condition, reporting that it made narration easier to follow and reduced uncertainty about where to look. These findings suggest that narration-synchronized visual guidance is a valuable design strategy for slide-based presentations, with potential to extend to other instructional and presentation scenarios.

0.17

ColorA11Y: Enhancing Creative Design Workflows with Just-in-Time Color Accessibility Recommendations

cs.HC ·2026-08-24 ·Alexa Siu, Rajiv Jain, Abhinav Kannan, Jose Echevarria, Mary Ann +5 more
top venue similarity 0.61
Abstract · To appear in ASSETS '26: The 28th International ACM SIGACCESS Conference on Computers and Accessibility (October 25-28, 2026, Vila Nova de G…

Effective color contrast in visual design is essential for content accessibility. While existing tools can identify contrast issues, they often operate in isolation from design workflows or are used as an afterthought. We present ColorA11Y, a system that supports designers in creating accessible content by providing just-in-time feedback and actionable recommendations throughout the authoring process to meet accessibility color contrast guidelines. Our system analyzes the visual properties of text and background elements and offers recommended changes, including text color adjustments, background modifications, and opacity changes. Through two user studies, we evaluate ColorA11Y's effectiveness. A user preference study (n=40) revealed varying effectiveness of different recommendations based on design context, while a qualitative study with designers (n=8) indicated a more seamless workflow experience in comparison to a baseline using a color contrast checker. This work advances a born-accessible approach to design, where accessibility considerations are seamlessly integrated into the creative process rather than treated as an afterthought.

0.30