CS Paper Radar

Trained on What Screen Readers Cannot Read: Accessibility Debt in the GUI Corpora Behind Computer-Use Agents

Obada Kraishan, Kulsawasd Jitkajornwanich

cs.HC cs.AI ·submitted 2026-11-11 23:28 UTC · v1
top venuebenchmark

Abstract

Computer-use agents learn to operate mobile interfaces from corpora of captured screens and accessibility trees. The same trees are what a screen reader consumes, so an agent trained on them inherits whatever labelling gaps the underlying applications carry. This paper measures one of those gaps, content-description coverage, in two Android corpora: Rico, collected in 2017, and AndroidControl, collected in 2024. Across 17,373,614 interface elements, 17.74% of the Rico elements that need a content description under our coding rule carry one, against 57.27% in AndroidControl. The difference of 39.5 percentage points holds under five alternative coding rules and when whole apps are resampled. It describes two datasets that also differ in app selection, capture method and platform version; it does not measure change over time. Input controls are the least labelled role in both corpora (2.20% and 27.21%, or 30.99% in AndroidControl if hint text counts as a name). Descriptions in the older corpus are also five times more likely to be placeholders (10.25% against 1.95%), so the difference is not an artefact of developers typing filler. Resolving 26,205 task actions to the elements they touched shows that, among tappable graphics with no text, demonstrations operate elements described 9.0 points more often than the others on the screens they visit, about 3.5 points of it from the choice of element within a screen. That is a selection effect inside the benchmark. A two-coder reliability check (kappa = 0.91 and 0.96) and the full pipeline accompany the paper.

Comment
8 pages, 2 figures, 6 tables. Accepted at the International Conference on Artificial Intelligence Centric Systems, Services, and Applications (AISSA 2026), San Antonio, TX, USA, December 8-10, 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Spatial Organization Is Accessibility Structure: Hierarchical Screen Reading of Mobile UIs

cs.HC ·2026-11-02 ·Yongxiang Hu, Xuan Wang, Qingyang Qiu, Yingjie Liu, Chun Kang +3 more
similarity 0.66
Abstract · 26 pages, 6 figures, 7 tables

Screen readers for blind and low-vision (BLV) users flatten mobile UIs into a linear sequence. Modern mobile UIs, however, contain dozens of elements grouped by spatial layout. Such linear access is ill-suited for BLV users, who must infer each element's meaning and which elements form a functional unit. Drawing on a quantitative TalkBack audit and formative interviews, we present Spatial Screen Reader (SeeR), which builds a function-aware hierarchy from accessibility metadata and provides unit summaries with hierarchical navigation. Compared with TalkBack, SeeR reduces non-informative focus stops by 82.1% and navigation steps by 52.7%. With ten BLV participants, summaries helped them connect controls to their functions and skip nonessential elements, especially on unfamiliar pages, though extra navigation levels added effort when the target was already known. These findings recast spatial organization as accessibility information: apps and platforms should expose it, rather than leaving screen readers to infer it post hoc.

0.17

From Observations to Authorized Actions: Measuring Source Information in Android Agents

cs.CR cs.OS ·2026-10-11 ·Hyobin Ahn, Dongjae Lee
benchmark similarity 0.63
Abstract · 18 pages, 2 figures

Android agents read content from applications and web pages and act on it with the user's authority, yet they cannot reliably tell where that content came from -- its source, the application or document that supplied it. The WebView accessibility fields we inspected expose the host application but omit the document origins available inside the browser, so an agent reading through this interface cannot check which document supplied an object; a limited reproduction on an unmodified WebView yields the same finding. We build and separately evaluate two prototypes to examine how source information supports authorization. Our application-source monitor withholds side-effecting calls whose signed source record contains an untrusted application. With the source set derived from the OS accessibility log, it blocked every selected attack write in the tested runs, but it also withheld benign sends issued after reading untrusted content, flagged benign tasks that use external content, and carried source records into a later clean task. Our WebView source-reference gate uses records from a modified browser to check object origins and reference lifetime through direct calls; in a separate 24-trial evaluation, the live agent selected no disallowed object. Supporting benchmark and planner measurements characterize textual argument matches and attacker-target selection, without establishing causal dataflow or a controlled model ranking. The prototypes are not an integrated defense: the WebView gate still lacks authenticated browser delivery and atomic capture of selected bytes and origin. Together, the results show which source information the tested policies need and why it must stay linked to the object used in execution.

0.28

SWE-Recreation: Benchmarking Coding Agents for Software Recreation

cs.SE ·2026-11-19 ·Zhiyao Xu, Zhanjie Ding, Hongyao Yu, Jingyu Xiao
benchmarkcode/data similarity 0.61
Abstract

Agents increasingly operate software and even write it, yet they are almost always told what to achieve: given a goal, a mock-up, or a demonstration. We ask whether an agent can instead understand an unfamiliar product by using it, and then rebuild it. The agent sees only rendered pixels and acts through pointer and keyboard, without access to the DOM, source, logs, or back-end. Within this interface, it explores, generates a replica, and checks its own work before evaluation. We present SWE-Recreation, a benchmark of 54 real applications (28 websites and 26 Android apps) with 1,446 hidden requirements describing multistep, stateful workflows and explicit exclusions for behavior outside the reconstruction scope. A fixed judge grades workflows with pixel evidence; a separate post-hoc judge estimates exploration coverage. Across 54 evaluated applications, five frontier systems reach at most 0.530 weighted fidelity and 22.9% fully reproduced requirements under application-equal averaging. These descriptive results expose both exploration-evidence gaps and implementation shortfalls. Code and data are available at https://github.com/nologistic/recreateanything.

0.36

CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

cs.AI ·2026-08-12 ·Linqiang Guo, Li Gu, Zihuan Jiang, Zhixiang Chi, Siobhan Reid +6 more
similarity 0.61
Abstract

Mobile GUI agents remain brittle when deployed to applications absent from source training. We study novel-app generalization under a limited target interaction budget and without target demonstrations. We introduce CoAdapt-GUI, a test-time adaptation (TTA) framework that jointly adapts structured workflow context and policy from the agent's own target-app rollouts and rewards. The workflow context retains transferable procedures, failure modes, and verification rules while excluding app-bound source details. This separation allows reusable workflow knowledge to guide adaptation without transferring source-interface state. For policy adaptation, task-context-matched group-relative optimization updates a LoRA adapter on a frozen vision-language model. Across two unseen-app evaluations, CoAdapt-GUI reaches 45.0% on AndroidWorld-Generalization, compared with 37.5% for the reported Policy-Only TTA baseline, and raises AndroidWorld Plus performance from 38.6% to 52.9%. These results show that transfer-constrained workflow context provides substantial gains and that joint policy adaptation further improves held-out performance.

0.34

When Agents See Differently: Exposing UI Desynchronization Threats in Mobile Agents

cs.CR ·2026-09-15 ·Heng Li, Fulin Zhao, Zhe Geng, Zhiyuan Yao, Wei Yuan +1 more
benchmark similarity 0.60
Abstract · Revised manuscript with updated experiments, figures, and appendices

Mobile agents may act on interface content that users cannot see or easily notice, making their actions harder to oversee. We characterize this gap as human-agent UI desynchronization and investigate how attackers can exploit it through perturbations embedded in applications before deployment. Beyond accessibility metadata, we identify two mechanisms that expose misleading cues to screenshot-based agents: content rendered in display-occluded regions under compatible device configurations, and low-contrast text that remains encoded in screenshots but is difficult for users to notice. We develop an automated framework that searches application-specific perturbation strategies, optimizes their content using representative tasks, and realizes the perturbations in deployable applications. The perturbation content remains fixed at runtime and requires neither access to future user instructions nor detection of agent use. Using a benchmark of 546 tasks across 13 applications, we evaluate five mobile-agent frameworks and three backbone models. The attacks achieve average misleading rates of 77.4% in static observation-space evaluation and 67.2% in dynamic application-level evaluation, where success denotes selection of an attacker-designated decoy control. We invited 195 participants to assess the perceptibility of our visual perturbations and found that none spontaneously noticed the low-contrast text. These findings expose a security gap between human-visible interfaces and agent-observable representations, showing that inconspicuous application content can redirect delegated actions even without runtime adaptation.

0.18

Practice in Text, Act in Pixels: Correcting Where Perception Fails

cs.CV ·2026-11-17 ·Zhongling Xu, Hairong Wang
code/data similarity 0.60
Abstract · 25 pages, 8 figures, 19 tables. Code: https://github.com/Carlxuzhl/practice-in-text ; aligned reader: https://huggingface.co/xuzhl/practice-…

GUI agents can learn cheaply from a text view of an interface and act on screenshots through a vision-language screen reader. Yet on a simulated mobile platform with a low-cost reader, policies that complete most tasks in text fail on many of the same tasks on screenshots. We ask where a small budget of real screenshots is best spent to close this gap. We find that the reader's misreads are systematic and recur on the same interface elements. Noise matched to the reader's error rates, a natural sim-to-real remedy, misses this structure, and in a pre-specified comparison it did not beat training the policy directly on screenshots. We therefore correct the misreads at their source with reader alignment, which fine-tunes the reader on screenshots paired with their text views. The reader learns to output each screen's text view, as in text practice, and the policy stays unchanged. Spent on the reader, the screenshot budget gives 80.9% full-task success on new task instances, against 55.6% when spent on training the policy. Given the same labeled screenshots, adapting the reader also beats adapting the policy, and a single aligned reader improves every text-trained policy we tried without retraining any of them.

0.49

Towards LLM-Enhanced Android Taint Analysis

cs.SE cs.CR ·2026-08-25 ·Nicholas Miazzo, Marco Alecci, Jordan Samhi, Jacques Klein, Eleonora Losiouk
benchmark similarity 0.60
Abstract

Taint analysis is a fundamental technique for detecting sensitive data leaks in Android apps. However, traditional static tools, such as FlowDroid, still face well-known challenges due to the complexity of accurately modeling the Android framework. In this paper, we investigate whether off-the-shelf Large Language Models (LLMs) can effectively reason about taint flows in Android apps. Our preliminary approach relies on an agentic interaction strategy, enabling the LLM to iteratively explore code and reason about data flows. We conduct an initial evaluation on the DroidBench benchmark against FlowDroid, where our approach outperforms the baseline: Gemini-3 Flash achieves an F1-score of 0.96, compared to 0.55 for FlowDroid. In particular, we observe improvements in challenging categories such as inter-component communication (0.95 vs. 0.17), implicit flows (0.94 vs. 0.00), and reflection (1.00 vs. 0.50), where FlowDroid typically struggles. On a small set of real-world apps, the LLM-based approach also identifies additional potential data leaks not reported by FlowDroid. These preliminary findings suggest that LLM reasoning may effectively complement traditional static taint analysis, motivating future research on hybrid LLM-enhanced taint analysis pipelines.

0.13

PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

cs.AI ·2026-09-28 ·Yangqin Jiang, Lingrui Xu, Chao Huang
similarity 0.58
Abstract

Mobile GUI agents operate through a perception--action loop: at each step they screenshot the device, invoke a vision--language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation---and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it deterministically in sub-second time at zero VLM cost; open-ended interaction and every failure of the compiled path fall back to the embedded VLM interpreter, exactly the pure VLM agent, so compilation can only help. On AndroidLab, PhoneCLI improves the task success rate while reducing steps and token consumption, and it transfers to AndroidWorld's official M3A agent with consistent efficiency gains. What PhoneCLI compiles is the app's navigation rather than one run, so it serves new tasks, not only repeated ones.

0.11