CS Paper Radar

Beyond Scene Description: Multi-Agent Orchestration for Non-visual Access to Virtual Worlds

Toqeer Ali Syed, Ali Akarma, Adeel Ahmad, Danial Hameed

cs.AI ·submitted 2026-09-13 13:34 UTC · v1

Abstract

Virtual worlds now host classrooms, meetings, conferences, shops, and social venues, and nearly every interaction they expose assumes a user who can scan a three-dimensional scene, follow avatars, and read floating panels. Blind and visually impaired (BVI) users are left with assistive tools that each solve one task in isolation: naming an object, reading text, describing a scene, or planning a route. A live virtual room defeats that model: obstacles, speakers, gestures, chat, slides, and notifications arrive together, and a tool that narrates all of them trades a visual barrier for an auditory one. This paper presents MetaBlind, an architecture that distributes nonvisual access across eight specialized agents, spanning perception, navigation, social and object interaction, communication, safety and trust, memory, and personalization, and that places an Accessibility Orchestrator between those agents and the user. Agents publish candidate information into a shared accessibility context instead of speaking to the user directly. The orchestrator scores each candidate on safety relevance, goal relevance, urgency, confidence, user relevance, and estimated listening load, then releases only the items it judges relevant at that moment through speech, structured audio, or haptic output. We give the selection step a formal statement, specify the orchestration cycle as an algorithm, and define an evaluation protocol against a single-agent assistant. MetaBlind is reported at the design stage, with no prototype measurement or user study, and the protocol states which outcomes would support the design and which would refute it.

Comment
Paper submitted in ICAAD Conference 2026

Closest papers

Nearest neighbours by meaning across the whole index.

More Than Just Access: Generative AI as Communication Intermediary for Blind and Low-Vision Users

cs.HC cs.AIcs.ET ·2026-09-14 ·Protik Dey, Mohd Saifuzzaman, Taslima Akter
published similarity 0.67
Abstract · Proceedings of the CSCW 2026 Workshop on Broader Impacts of GenAI in Communication:Building Agendas for Research, Design, and Policymaking, …

Generative AI (GenAI) tools are increasingly woven into how blind and low-vision (BLV) people communicate, not only with digital information, but with the physical world and with other people. Tools such as ChatGPT, Google Gemini, Be My AI, and Seeing AI translate visual and textual content into accessible form, and are beginning to substitute for interpersonal requests for help, such as asking a family member to read a label or describe a scene. Drawing on semi-structured interviews with 19 BLV participants, we examine GenAI as a communication intermediary and how it succeeds and fails as an alternative for reading, describing, and even asking another person for help. We also investigated what BLV users gain and risk when these tools take over that role. We conclude with design and policy implications for GenAI systems that communicate uncertainty honestly, protect information, and support BLV users' independence rather than substitute for it unsafely.

0.38

"I didn't know how to read a map, but now I can": TouchingSpace, an Audio-Haptic Map for Blind and Low-Vision Readers

cs.HC ·2026-09-28 ·Li Liu, Yihe Wang, Jiaming Qu, Ashmita Dua, David T. Lee +1 more
similarity 0.65
Abstract

Accessible map systems either make a layout explorable by hand or convey information through speech; few combine both to support pre-travel spatial understanding for blind and low-vision (BLV) people. We present TouchingSpace: a system that retrieves map data for an outdoor place and renders its surroundings as bounded regions at fixed trackpad positions. During exploration, users receive audio and haptic feedback and can ask a conversational agent open-ended questions. We conducted a user study with fourteen BLV participants who explored a place using TouchingSpace and reflected on the experience. We found participants used sound and vibration to locate places and speech to identify and describe them; the bounded surface supported discovery, revision, and spatial checks by hand; they expected this awareness to support future travel. TouchingSpace demonstrates how a laptop trackpad can support self-directed spatial exploration. These findings suggest accessible AI maps should ground conversation on bounded, user-controlled spatial surfaces.

0.09

Spatial Organization Is Accessibility Structure: Hierarchical Screen Reading of Mobile UIs

cs.HC ·2026-11-02 ·Yongxiang Hu, Xuan Wang, Qingyang Qiu, Yingjie Liu, Chun Kang +3 more
similarity 0.62
Abstract · 26 pages, 6 figures, 7 tables

Screen readers for blind and low-vision (BLV) users flatten mobile UIs into a linear sequence. Modern mobile UIs, however, contain dozens of elements grouped by spatial layout. Such linear access is ill-suited for BLV users, who must infer each element's meaning and which elements form a functional unit. Drawing on a quantitative TalkBack audit and formative interviews, we present Spatial Screen Reader (SeeR), which builds a function-aware hierarchy from accessibility metadata and provides unit summaries with hierarchical navigation. Compared with TalkBack, SeeR reduces non-informative focus stops by 82.1% and navigation steps by 52.7%. With ten BLV participants, summaries helped them connect controls to their functions and skip nonessential elements, especially on unfamiliar pages, though extra navigation levels added effort when the target was already known. These findings recast spatial organization as accessibility information: apps and platforms should expose it, rather than leaving screen readers to infer it post hoc.

0.17

Coral: Contextual Gists for Blind and Low Vision Screen Reader Users' Understanding of Dynamic User Interfaces

cs.HC ·2026-09-19 ·Ritesh Kanchi, Jianna So, Krzysztof Z. Gajos
hot direction similarity 0.62
Abstract

Blind and low vision (BLV) screen reader users construct mental models of user interfaces (UIs) through incremental screen reader interaction, a time-consuming and cognitively demanding process complicated by modern interfaces that may not convey dynamic content accessibly. We interviewed eleven BLV screen reader users about UI understanding and derived design objectives that informed Coral, a context-aware browser extension. Coral synthesizes interface and user context to generate screen reader-narrated gists which holistically notify users of interface states and changes relevant to their likely goals and ongoing interaction. In a comparative evaluation with eight BLV screen reader users, participants used Coral to form initial expectations and interpret interface changes, and reported spending less time and effort manually verifying interaction outcomes. Together, our findings provide deeper insight into how BLV screen reader users navigate gaps in their UI understanding, and how intelligent support that combines interface and user context can bridge these gaps.

0.24

Context Matters! Social, Spatial, and Semantic Coordination with an Embodied Agent in Immersive Analytics

cs.HC ·2026-10-15 ·Andrew Cunningham, Nicolas Barbotin, Adam Drogemuller, Jack Faser
top venue similarity 0.61
Abstract · In 2025 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (pp. 189-194). IEEE

Embodied AI agents in Augmented Reality (AR) could enable more natural collaboration in immersive analytics by engaging users through spatial, conversational, and visual interaction. However, designing such agents requires careful attention to context. We propose a preliminary conceptual framework for context-aware design of embodied agents structured around three dimensions of spatial, social, and semantic. This framework is further categorised into embeddedness (how agents are situated in the contexts) and awareness (how agents perceive and respond to the contexts). To explore this, we present Cobot, an embodied AI agent designed as a design probe. Cobot supports context-sensitive interaction through features such as gaze cues, spatial sound, and task-aware behaviour. We reflect on its design and identify implications for future contextaware embodied agents.

0.28

Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications

cs.HC cs.AI ·2026-09-01 ·Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan +1 more
top venue similarity 0.60
Abstract · Accepted to the EMNLP 2026 Main Conference

Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.

0.38

Affora: A Design System for Agent-Friendly Interfaces

cs.HC cs.AIcs.SE ·2026-09-16 ·Jin Gao
similarity 0.60
Abstract · 20 pages, 8 figures

Computer-use agents increasingly operate software designed for people, but interfaces often leave actions or task state unclear to machine readers. We present Affora, a design system that supports both readers while preserving visual freedom and familiar human workflows. Three controlled studies examine component implementations, visual variation, and interaction-design principles. Their findings inform guidance from individual components to complete sites, supported by reusable implementations and executable checks. Agent performance depends on the interaction meaning available through its interface representation; substantial visual variation remains possible when that meaning is preserved. Evaluation on independently authored interfaces shows gains where Affora addresses existing deficits, but limited effects where those deficits are absent or outside its coverage. A workflow case provides preliminary evidence of reduced interaction cost. Affora connects user experience and agent experience through a shared interface rather than a separate agent-only surface.

0.24

ABRA: Agent-Bound UI for Human-Agent Communication beyond Chat with Reusable Views

cs.HC cs.MA ·2026-10-22 ·Seon Gyeom Kim, Hannah Kim
top venuecode/data similarity 0.59
Abstract · EMNLP 2026 System Demonstrations

Agents can create rich representations, yet these representations are rarely used as a medium for expressing user intent. We present Agent-Bound UI, an interaction model that separates reusable interaction structures from task-specific meanings. Agents bind task-specific meanings to interactions in reusable views and publish the resulting bindings as shared state, enabling humans and agents to understand and act on shared surfaces whose meanings vary across tasks. We implement ABRA, a framework-agnostic prototype of Agent-Bound UI, in which users and agents collaborate through chat and shared interactive canvases. We demonstrate ABRA through a representative use case and evaluate four LLM-backed agents on canvas authoring, owner-side consequence resolution, and cross-agent invocation. The code and demo video are available at https://github.com/megagonlabs/abra.

0.56