CS Paper Radar

Affora: A Design System for Agent-Friendly Interfaces

Jin Gao

cs.HC cs.AIcs.SE ·submitted 2026-09-16 17:46 UTC · v1

Abstract

Computer-use agents increasingly operate software designed for people, but interfaces often leave actions or task state unclear to machine readers. We present Affora, a design system that supports both readers while preserving visual freedom and familiar human workflows. Three controlled studies examine component implementations, visual variation, and interaction-design principles. Their findings inform guidance from individual components to complete sites, supported by reusable implementations and executable checks. Agent performance depends on the interaction meaning available through its interface representation; substantial visual variation remains possible when that meaning is preserved. Evaluation on independently authored interfaces shows gains where Affora addresses existing deficits, but limited effects where those deficits are absent or outside its coverage. A workflow case provides preliminary evidence of reduced interaction cost. Affora connects user experience and agent experience through a shared interface rather than a separate agent-only surface.

Comment
20 pages, 8 figures

Closest papers

Nearest neighbours by meaning across the whole index.

A Decade in a Day: Synthesizing Agent-Native Interfaces from Human-Oriented Software Evolution

cs.SE cs.AI ·2026-10-11 ·Alan Wu, Wei Tao, Yiwen Guo
benchmark similarity 0.70
Abstract

Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were designed for human end users or application developers, leaving agents to interpret visual controls or discover and compose low-level software operations. This mismatch can hinder even capable agents, yet it remains unclear what interface properties make desktop software easier for agents to use. To study this question, we treat software evolution as a natural source of interface variation and introduce DeskIF-Bench, a benchmark of 255 artifact-verified tasks across six desktop applications. By holding the task, agent, and verifier fixed while varying the software release, we measure how interface changes affect agent performance. Across two models and six applications, the newest release consistently outperforms the oldest. Trajectory analysis identifies three relevant interface properties: discoverability, granularity, and observability. Guided by these properties, we develop ANIS, a framework that synthesizes agent-native interfaces from the capabilities of installed applications. On an expanded evaluation of 368 tasks across eight applications, atomic interfaces improve success over native SDK use by 3.8 and 3.5 percentage points with DeepSeek V4 Flash and Claude Haiku 4.5, respectively, while reducing online token usage by approximately 73%. Goal-level commands further raise success to 74.5% and 67.4%, respectively.

0.14

Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications

cs.HC cs.AI ·2026-09-01 ·Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan +1 more
top venue similarity 0.68
Abstract · Accepted to the EMNLP 2026 Main Conference

Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.

0.60

Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

cs.HC cs.CL ·2026-10-30 ·Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang +3 more
position similarity 0.68
Abstract

Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.

0.13

Interface Quilt: Modifying Applications with Computer Use Agents and Generative UI

cs.HC ·2026-11-01 ·Leping Qiu, Jo Vermeulen, Tim Zindulka, George Fitzmaurice, Fraser Anderson
similarity 0.67
Abstract · Preprint, under review

Modifying an existing application is difficult when it does not provide programmatic access for that change. Recent advances in computer-use agents (CUAs) enable richer understanding and operation of existing interfaces, while generative UI (GenUI) makes it practical to generate new ones. We examine how these advances expand software modification through a design space of what can be changed and a framework for how modifications can be implemented. We then introduce Interface Quilt, a system that combines CUAs and GenUI to generate interface patches, which are new interfaces displayed over an existing application to modify its presentation, interaction, and capabilities. Through demonstrations across a variety of applications, we show that interface patches support a range of modifications without source code, APIs, or accessibility data.

0.08

Evolving Agent Interfaces

cs.AI ·2026-10-20 ·Saaket Agashe, Yuzhe Yang, Gonzalo Gonzalez-Pumariega, Bhuvan Chandra Koduru, Chih-Lun Li +3 more
positioncode/datacommunity pick similarity 0.66
Abstract

Computer-use agents are limited by the interfaces humans give them. An agent may type a spreadsheet cell by cell when a single line of code would do, or print all two thousand lines of a file through a shell to find one function. Both the interfaces and the knowledge of when to use each are built by hand, which cannot keep pace with new applications, workflows, and models. We argue that interface design should itself be an agent capability. We introduce Agent S4, an autonomous computer-use scientist that builds Evolving Agent Interfaces from task experience. A meta-agent examines task agents' trajectories on synthetic training tasks, without success labels, and constructs and revises interfaces and usage knowledge in a repository. Population-based evolution then selects which repositories to develop further by their performance on validation tasks the meta-agent never sees, favoring changes that generalize beyond the seen tasks. On OSWorld V2, Agent S4, with Qwen3.8-27B and the evolved repository of interfaces, sets a new state of the art in open-source computer-use agents, reaching 21.3% binary score, 3.8 times the same model with fixed interfaces. This is above both GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%), at about 4.3% and 1.5% of their respective costs. On ScienceBoard, the evolved interfaces improve the success rate from 41.4% to 58.6% with the same model. These results suggest that useful interfaces can be evolved from experience, making autonomous interface design a source of improvement complementary to advances in models and harnesses. We open-source our code at https://github.com/simular-ai/agent-s.

0.60

Coral: Contextual Gists for Blind and Low Vision Screen Reader Users' Understanding of Dynamic User Interfaces

cs.HC ·2026-09-19 ·Ritesh Kanchi, Jianna So, Krzysztof Z. Gajos
hot direction similarity 0.66
Abstract

Blind and low vision (BLV) screen reader users construct mental models of user interfaces (UIs) through incremental screen reader interaction, a time-consuming and cognitively demanding process complicated by modern interfaces that may not convey dynamic content accessibly. We interviewed eleven BLV screen reader users about UI understanding and derived design objectives that informed Coral, a context-aware browser extension. Coral synthesizes interface and user context to generate screen reader-narrated gists which holistically notify users of interface states and changes relevant to their likely goals and ongoing interaction. In a comparative evaluation with eight BLV screen reader users, participants used Coral to form initial expectations and interpret interface changes, and reported spending less time and effort manually verifying interaction outcomes. Together, our findings provide deeper insight into how BLV screen reader users navigate gaps in their UI understanding, and how intelligent support that combines interface and user context can bridge these gaps.

0.24

Adaptation of Computer Use Agents to New Software with User Manuals

cs.AI ·2026-10-16 ·Chulhoon Yoon, Kyungri Park, Jinyeong Yun, Daeyun Jeong, Sukmin Yun +1 more
code/data similarity 0.66
Abstract · 21 pages, 3 figures, 14 tables. Code: https://github.com/ku-autoai/MAGE

Recent advances in multimodal foundation models have significantly improved the ability to operate real-world software through graphical user interfaces (GUIs) using visual observations and natural language instructions. However, maintaining strong performance on unseen applications remains challenging, as practical tasks often require long-horizon procedural knowledge and application-specific operation patterns. In addition, constructing executable tasks and collecting corresponding trajectories for training still heavily rely on human-annotated data, incurring substantial cost. In this paper, we propose MAGE (Manual-augmented computer use AGent Execution and learning), a framework that leverages user manuals as a scalable source of procedural knowledge to improve both inference and training. During execution, MAGE retrieves task-relevant manual content and uses a planner to generate explicit procedural guidance for CUAs. To further account for the discrepancies between manuals and actual GUI environments, such as version differences or user settings, MAGE also adapts the agent policy through environment interaction. During training, MAGE converts manuals into executable task-plan pairs and uses the plans to guide environment rollouts toward task-relevant states; the resulting trajectories are then used to optimize the policy without human-authored demonstrations. This unified design enables effective utilization of manuals for both task execution and training, reducing reliance on human-authored training data. Experimental results across six applications demonstrate improvements in task success rates over strong baselines, highlighting the effectiveness of manual-guided planning and policy adaptation in new software environments. We provide the code and reproduction scripts at https://github.com/ku-autoai/MAGE.

0.29

Inkitiate: Initiating and Steering Agent Assistance through Ink in a Spatial Workspace

cs.HC ·2026-10-29 ·Liangwei Wang, Nan Chen, Luna K. Qiu, Zilong Wang, Fugee Tsung +2 more
similarity 0.65
Abstract · 18 pages, 11 figures

Handwriting and sketching help people develop ideas around information. As this work unfolds, users may seek assistance with finding, understanding, and organizing materials. We investigate how ink can initiate and steer agent assistance in this process. An eight-participant formative study explored desired assistance and interaction expectations, informing three connected activities: bringing materials in, thinking with materials, and shaping the space. We present Inkitiate, a spatial workspace for ink-based agent assistance in knowledge work. Through ink, users retrieve external materials, build on accumulated content and results, and reorganize the canvas, while inspecting and guiding agent actions. In a study with eighteen participants, users reported favorable usability and low perceived workload, and used retained ink and materials to develop further ink queries. An ink-query vocabulary and cases reveal how users combine linguistic and graphical expressions to direct agents. These findings inform ink-based agent interaction design for expressive requests and continued work with materials.

0.31