CS Paper Radar

ATCion: Exploring the Design of Icon-based Visual Aids for Enhancing In-cockpit Air Traffic Control Communication

Yue Lyu, Xizi Wang, Hanlu Ma, Yalong Yang, Jian Zhao

cs.HC ·submitted 2026-09-21 16:34 UTC · v1
top venueunusual

Abstract

Effective communication between pilots and air traffic control (ATC) is essential for aviation safety, but verbal exchanges over radios are prone to miscommunication, especially under high workload conditions. While cockpit-embedded visual aids offer the potential to enhance ATC communication, little is known about how to design and integrate such aids. We present an exploratory, user-centered investigation into the design and integration of icon-based visual aids, named ATCion, to support in-cockpit ATC communication, through four phases involving 22 pilots and 1 ATC controller. This study contributes a validated set of design principles and visual icon components for ATC messages. In a comparative study of ATCion, text-based visual aids, and no visual aids, we found that our design improved readback accuracy and reduced memory workload, without negatively impacting flight operations; most participants preferred ATCion over text-based aids, citing their clarity, low cognitive cost, and fast interpretability. Further, we point to implications and opportunities for integrating icon-based aids into future multimodal ATC communication systems to improve both safety and efficiency.

Comment
23 pages. To appear at UIST 2025 (The 38th Annual ACM Symposium on User Interface Software and Technology)
Journal ref
UIST 2025 (The 38th Annual ACM Symposium on User Interface Software and Technology)
DOI
10.1145/3746059.3747792

Closest papers

Nearest neighbours by meaning across the whole index.

AdaptType: Accessible and Customizable Multimodal Text Entry in Virtual Reality for People with Upper-Body Motor Impairments

cs.HC ·2026-11-14 ·Junhan Kong, Mingyuan Zhong, Sanjana Pavani, Jacob O. Wobbrock
similarity 0.54
Abstract

Text entry in virtual reality (VR) is cumbersome, especially for people with upper-body motor impairments. Existing techniques often demand both fine and gross motor control, yet users' motor abilities vary widely. To address this range of needs, we present AdaptType, a multimodal VR text entry system that enables users to configure input techniques that best suit their abilities, and scaffolds customization through ability-based recommendations. Through a study with 10 participants, we found that customized multimodal techniques improved text entry speed compared to gaze- or hand-only techniques, and complemented voice for error correction, decreasing failure rate from 15.1% to 2.0%. We found that AdaptType yielded high text entry speed (36.26 WPM) and accuracy (0.36% error rate), with low task load and high usability ratings. We also found ability-based recommendations effectively scaffolded customization. Our work contributes more accessible and personalized VR text entry solutions for people with upper-body motor impairments.

0.24

Cross-Modal StateKV Injection for Communication in Multi-LLM Systems across Temporal Segments

cs.CV ·2026-11-17 ·Xiaoqian Zhang, Zhihao Zhang
hot direction similarity 0.52
Abstract · 9 pages, 4 figures, 5 tables. Equal contribution: Xiaoqian Zhang and Zhihao Zhang

Video understanding with multiple multimodal language models requires a communication interface that can transfer complementary visual observations. Text-based communication is interpretable, but introduces an intermediate generation stage and can omit information relevant to subsequent reasoning. We study Cross-Modal StateKV Injection, a provider--carrier framework that communicates through key--value (KV) states at visual-token positions. Provider models observe complementary video frames; their visual KV states are aligned and projected into the carrier representation space and integrated through gated residual updates. Injection preserves the carrier KV-cache sequence length, while retaining the additional computation required by the providers and the communication module. The reported experiments on MLVU and Video-MME compare carrier-only inference, text-based communication, and StateKV with two provider architectures. Results show modest, task-dependent accuracy gains and lower reported latency than the evaluated text-communication baseline, with some categories remaining unchanged or decreasing. Increasing provider-side frame coverage offers additional gains on selected tasks without extending the carrier cache. These findings motivate further study of visual KV communication under explicitly controlled observation and computation budgets.

0.40

A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs

cs.AI cs.CLcs.CV ·2026-10-27 ·Alexa R. Tartaglini, Zhengxuan Wu, Thaddäus Wiedemer, Martin Wattenberg, Been Kim +1 more
similarity 0.50
Abstract

Do vision-language models (VLMs) share our ability to communicate visually? We share preliminary findings, starting from a surprising observation: simple visual symbols like a red heart can strongly influence model behavior on seemingly unrelated tasks, such as steering models towards recursive solutions on simple coding problems. Notably, this steering effect is largely implicit, as models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior. In visual communication between models, this leads to a generation--interpretation gap: while models tend to send shape primitives that they (and humans) find intuitive, these visual cues often fail to steer other models. In further communication games involving abstract concepts and stories, models can creatively use visuals to transmit meaning to one another, although some communications remain non-intuitive to humans. These findings reveal that VLMs might be capable of rich, creative, and visually-grounded communication, but their interpretation and production of visual meaning diverges from human conventions in surprising ways. Together this work highlights both the promise and challenges of visual communication between humans and VLMs.

0.18

Present Drawing: Exploring Avatar-Mediated Drawing Events with Physical Traces

cs.HC ·2026-10-23 ·Kojiro Tanaka, Takuya Iwamoto, Yuki Okafuji
top venue similarity 0.49
Abstract · Accepted (ACM UIST 2026)

We introduce Present Drawing, an approach for staging an avatar-mediated drawing event that leaves physical traces in the real world. Present Drawing aims to reconfigure conventional drawing events by relaxing their physical constraints. We describe an organizer-oriented design space for such an event along four elements: Time, Site, Control Source, and Avatar Embodiment. To ground this concept, we developed a proof-of-concept prototype in which an observer wears a head-mounted display to view an avatar. The avatar's motion is synchronized with a customized pen plotter whose mechanical parts are hidden from the observer. We also developed five application examples spanning live, playback, multi-site, mascot, and system-controlled configurations. We conducted an initial evaluation of one representative playback autograph configuration, showing the basic plausibility of the core Present Drawing experience as an avatar-mediated drawing event and highlighting factors associated with perceiving the avatar as the drawer.

0.42

Remote Expert Evaluation of Tabletop Avatar Gaze Control in Asymmetric Multi-Scale MR--VR Collaboration

cs.HC ·2026-10-19 ·Carlos Mosquera, Neven Elsayed, Ernst Kruijff, Joseph Newman, Valeriya Lehrbaum +1 more
publishedsurvey similarity 0.49
Abstract · Accepted for ACM Symposium on Virtual Reality Software and Technology (VRST) 2026

This paper proposes an integrated setup for a multi-scale asymmetric collaboration scenario in the context of a remote-expert use case, focusing on the expert side. In this system, the remote expert (with an exocentric view in Mixed Reality (MR)) uses a Tangible User Interface (TUI) to precisely position and rotate their avatar within a scaled tabletop World-in-Miniature (WIM). Inside the WIM, the expert's avatar and the operator share the same scale and egocentric view, while the expert retains a global tabletop overview of the plant to support collaboration. Interviews with industry experts during the participatory design phase revealed the need for an egocentric view inside the plant on the remote-expert side to enable full collaboration. We designed and studied three interaction-visualisation combinations to integrate and control the expert avatar's gaze: VR (fully immersive with direct head control), Head (floating screen with direct head control), and Touch (on-avatar screen with touchpad control). We evaluated these methods with 29 participants across two fault-detection tasks: Same Machine, where both collaborators inspect the same machine, and Twin Machine, where they inspect identical machines in separate locations. The VR condition, which combines immersion, direct head control, and a full-scale view, yielded the strongest overall performance and user experience, while the tabletop avatar setup was well received for collaboration. Our findings inform the design of gaze-control techniques for asymmetric MR-VR collaboration and reveal challenges and opportunities for future work.

0.23

Through Human Eyes and Machine Eyes: Understanding View Mismatch in Video See-Through Extended Reality

cs.HC cs.CR ·2026-09-24 ·Yanming Xiu
similarity 0.49
Abstract

Video see-through extended reality (VST XR) systems commonly use headset screenshots or captured frames as proxies for the user's first-person visual context. However, the system-captured view and the user's effective visible field do not necessarily coincide: a screenshot records a rectangular machine-readable frame, whereas the user's effective visible region can be more constrained and non-rectangular. This paper studies this human-system view mismatch in VST XR. We formalize the relationship between the system-captured region and the human-visible region by defining their co-visible, system-only, and human-only regions. \rev{We then conduct a pilot-level boundary measurement on Meta Quest 3, revealing a clear mismatch between the rectangular screenshot frame and the approximate human-visible boundary. Building on this model, we analyze how view mismatch can affect screenshot-based XR sensing and downstream vision-language model tasks. Through four representative case studies, we illustrate potential risks and failure modes including prompt injection, privacy leakage, human-invisible information bias, and missing human-visible information. Our results show that view mismatch is not only a geometric artifact, but can also introduce security, privacy, and reliability concerns for AI-integrated VST XR systems.

0.25

Exploring Avatar-Based Representations of Desktop Analytical Workflows for Asymmetric Collaborative Visual Analytics

cs.HC ·2026-09-14 ·Tiansu Chen, Yalong Yang, Wai Tong
similarity 0.49
Abstract

Collaborative visual analytics increasingly occurs across asymmetric desktop-VR settings, with desktop analysis offering precision and efficiency and VR providing spatial and embodied affordances. However, maintaining workspace awareness remains challenging because desktop collaborators are often represented in VR only through indirect cues such as perspective sharing, shared visualization state, or lightweight cursor traces, which do not convey their ongoing analytical activity in a VR-native manner. To address this gap, we present Desk2Avatar, which explores avatar-based representations of desktop analytical workflows in VR. We introduce two representation strategies, Distanced and Embodied, inspired by remote pointing and direct manipulation. We conducted a within-subject study with 18 participants comparing these strategies against a depth-adaptive cursor baseline. Our findings show that avatar-based representations improved collaborator awareness and attentional guidance over cursor cues, while excessive embodiment introduced occlusion, distraction, and additional workload. Finally, we discuss design implications for future desktop-to-VR representations, focusing on balancing collaborator presence, attentional guidance, and workspace readability.

0.24

Offline Multimodal Large Language Models for Decision Support in Air Operations

cs.AI cs.CL ·2026-09-18 ·Joao P. A. Dantas, Jelton A. Cunha, Gabriel Dietzsch
published similarity 0.49
Abstract

Air operations rely on complex rules, established procedures, and time-critical analysis under limited connectivity and strict security constraints. In such environments, analysts must combine written doctrine with images, often without access to external computing resources. This paper studies offline large language models as decision support tools, deployed in isolated and restricted environments to give analysts access to doctrinal knowledge that remains traceable to its original sources through natural language interaction. We describe a modular retrieval-augmented architecture suitable for operation without Internet connectivity, supporting both text and image input from technical manuals. As a first step toward evaluating this architecture, we report a pilot study with four image analysts of the Brazilian Air Force, combining (i) a doctrinal knowledge assessment based on their electronic-target identification doctrine, comparing human and proposed system performance on the same test, and (ii) a measurement of the cognitive workload involved in manually producing a reconnaissance target report (Relatório de Missão de Reconhecimento - REMIR) without AI assistance. The results show a demanding manual task, especially in terms of mental demand (6.0/7) and effort (5.0/7), while the proposed system matches the human score (8/10) and completes the assessment in 7.1 minutes (compared to a human average of 26.5 minutes), establishing a baseline for future AI-assisted evaluation. Finally, we describe a future evaluation protocol to systematically compare manual and AI-assisted workflows.

0.46