CS Paper Radar

CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning

Xin Yan, Zhengbo Jiao, Jiaqi Liu, Zhenglin Wan, SiYuan Ma, Xuliang Yu, Tianyi Jiang, Chubin Zhang, Pengfei Zhou, Wangbo Zhao, Xingrui Yu, Bo An, Yang You, Ivor Tsang

cs.AI ·submitted 2026-09-26 16:16 UTC · v1

Abstract

Reinforcement learning enables computer-use agents to improve through interaction with real software environments, including websites and desktop applications. However, conventional deployments replicate an initialized runtime for each independent rollout, even when trajectories use the same software, incurring repeated memory and initialization costs as the number of parallel environments grows. Does an independent computer-use environment require an independent execution runtime? Our key observation is that trajectories require independent mutable state, while initialized application runtimes can be reused across concurrently evolving environments, making state the natural unit of environment independence. Guided by this observation, we introduce CUA-Sandbox, which separates private state capsules from shared runtimes through state-scoped execution and transactional lifecycle operations, including resets and branches, while retaining the original software interfaces and task evaluators. Experiments show comparable or improved task success relative to Docker, while substantially reducing rollout and resource costs. CUA-Sandbox achieves up to a 6.20x increase in rollout throughput, a 9.2x reduction in per-environment memory, and a 504x reduction in incremental storage.

Closest papers

Nearest neighbours by meaning across the whole index.

WeEnv: The Environment for Agentic Reinforcement Learning at WeChat

cs.DC ·2026-09-25 ·Yang Yu, Jing Lei, Shaoxun Zeng, Xinyu Gao, Jindi Shi +2 more
hot direction similarity 0.72
Abstract

Agentic reinforcement learning (RL) differs from conventional RL in that every task executes inside a complex environment, e.g., a virtual machine or a container. We find that agentic RL pays a heavy environment tax: a large share of the iteration time goes to the environment rather than to learning. The root cause is the lack of a full-lifecycle solution to environment management. We present WeEnv, which manages environments across packaging, initialization, and provisioning. WeEnv packages components as independently published layer groups and composes them at initialization, so that updating a component republishes one small group rather than every artifact containing it. To speed up environment initialization, WeEnv launches environments instantly and fetches contents on demand. During task execution, WeEnv provisions CPU and memory elastically, adjusting each environment's quota from its observed usage to fit the varying demands. WeEnv reduces the initialization by 5.6-14.2x over E2B, Docker, and AgentENV, cutting its share of the iteration time from up to 53.4% to 9.1%. WeEnv is deployed for agentic RL at WeChat.

0.31

Sandman: A Sandbox Service for Production Agentic Reinforcement Learning

cs.DC cs.AIcs.LG ·2026-10-29 ·Yiming Yan, Hongyang Wang, Wenbo Chen, Xuan Wang, Yichuan Li +10 more
code/data similarity 0.71
Abstract · 28 pages, including appendix

Agentic reinforcement learning (RL) needs stateful environments for tool execution and evaluation. Training jobs request thousands of environments in bursts that can overwhelm shared cluster services, while cancelled or timed-out requests can leave resources running. We present Sandman, a sandbox service built and operated at Amazon for large-scale agentic RL workloads in production on shared Kubernetes. A small daemon mounted into unmodified task images preserves files and processes across calls. Sandman paces creation requests, checks sandbox readiness, and retains cleanup responsibility after callers time out or cancel. Production deployments created 3.1 million sandboxes for 122 coding-agent training runs in two weeks. Peak production concurrency was 11,858 sandboxes. In one 10,000-request multi-client burst, Sandman made half the sandboxes usable in 116 seconds and all in 308 seconds. Full delivery took 44% less time than a tuned configuration of agent-sandbox, an open-source sandbox manager for Kubernetes. Sandman left no sandboxes at client exit. On software-engineering task images, median no-op command latency was 2.0 ms, compared with 14.3 ms for agent-sandbox and 203.5 ms for OpenSandbox. We also measure provisioning costs and sandbox delays within training runs, and report design trade-offs and operational lessons from production use and comparisons with three open-source systems. We will open-source Sandman's daemon, provider and pool.

0.23

BunshinRL: Whole-Workspace Forking for Agentic Reinforcement Learning Rollouts

cs.AI cs.OS ·2026-11-07 ·Yutong Huang, Vikranth Srivatsa, Alex Asch, Yiying Zhang
similarity 0.71
Abstract

Agentic reinforcement learning (RL) rollouts interleave model reasoning with tool calls in persistent workspaces. Scaling these rollouts incurs substantial memory costs for concurrent environments, while slow tool calls delay entire rollout batches. We present BunshinRL, an agentic RL rollout system that exploits shared environment state and recurring tool calls through three techniques: memory sharing across environments, tool-prefix caching, and speculative tool execution. These techniques build on BunshinFork, a novel live whole-workspace forking mechanism for containers. BunshinFork reconstructs process trees, shares memory copy-on-write, and preserves file, communication, and GUI state, enabling cached-state reuse and rollback after incorrect speculation. We implement BunshinRL and BunshinFork through Linux kernel module and CRIU modifications. BunshinRL reduces peak memory by up to 43%, total memory consumption by up to 51%, and end-to-end rollout latency by 23%.

0.20

PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning

cs.AI ·2026-09-28 ·Jiaan Zhu, Wei Gao, Youhui Bai, Zewen Jin, Ju Huang +4 more
similarity 0.69
Abstract

Multi-turn rollout dominates the cost of agentic reinforcement learning (RL). Asynchronous execution and elastic GPU resources can accelerate this stage, but adding rollout replicas yields diminishing returns while training GPUs remain idle between updates. We observe that effective resource use also depends on the prefill--decode (PD) configuration. Both the choice between colocation and disaggregation and the optimal PD ratio vary with the workload, making resource scaling and PD configuration interdependent. Exploiting this opportunity requires selecting effective configurations and realizing their benefits within transient resource-availability windows despite reconfiguration costs. We present PEARL, an asynchronous agentic RL system that coordinates external resource elasticity, temporary reuse of idle training GPUs, and adaptive PD execution. PEARL maintains a unified GPU--worker--role state and uses runtime profiles to predict rollout batch completion time, accounting for environment-induced reductions in decode concurrency. It selects the PD mode and ratio under the current GPU budget and translates each decision into an incremental transition plan that minimizes worker and role changes. Cost-aware switching and borrowing policies suppress transitions with insufficient expected benefit while ensuring timely return of training GPUs. Our evaluation show that PEARL achieves $2.17$--$2.79\times$ the throughput of fixed-resource ROLL across different LLMs. Compared with RLBoost+, throughput improves by up to approximately 26.9\% for Qwen3-8B and 36.3\% for Qwen3-30B-A3B.

0.13

VenusRL: A Fully Disaggregated Agentic RL System with Priority Scheduling and Scalable Interaction

cs.DC ·2026-10-02 ·Mingjun Zhang, Yucheng Li, Menghao Zhang, Shuyong Zhu, Ping Zhang +13 more
similarity 0.69
Abstract · 18 pages, 19 figures

Agentic Reinforcement Learning (RL) trains LLM agents through multi-turn interactions with external tool environments. Its multi-turn nature exposes two system-level bottlenecks unaddressed by existing agentic RL frameworks. First, end-to-end training throughput is constrained by the slowest trajectories to complete, yet optimizing per-GPU utilization alone scatters rollout progress across many groups, delaying the completion of enough groups to unblock the next training step. Second, tool sandboxes are statically over-provisioned by their declared memory ceilings, leaving most physical memory stranded while replicating near-identical state across sandboxes launched from the same prompt. We present VenusRL, a fully disaggregated agentic RL system that addresses both bottlenecks. VenusRL's priority-aware action scheduler uses length-prediction heuristics to identify sample groups whose completion is most likely to unblock the next training step, and pushes them ahead of others across batch admission, KV Cache residency, and cross-worker request orchestration. VenusRL's environment resource manager combines a memory-aware admission threshold with a template-keyed page-sharing pool, packing more sandboxes per node while preserving strict memory isolation via write-protected page table entry aliasing and copy-on-write. Across representative agentic RL workloads, VenusRL achieves up to 4.24x end-to-end training speedup over state-of-the-art baselines and reduces environment cost by up to 89%.

0.48

TenonAgent: Virtualizing Execution Residency for Memory-Oversubscribed LLM Agents

cs.OS ·2026-10-11 ·Yifan Zhang, Xinkui Zhao, Ruixin Huang, Lezhuo Xuan, Dianyu Wang +6 more
similarity 0.67
Abstract · 18 pages, 14 figures, 4 tables

LLM agents execute as long-lived, multi-stage workflows with substantial resource-demand variation across phases. Provisioning each agent or sandbox for its lifetime peak leads to low resource utilization, particularly for memory, whereas aggressive oversubscription risks memory contention, out-of-memory failures, and costly workflow retries. This tension limits the concurrency and efficiency of agent serving under constrained host memory. We present TenonAgent, a runtime system that virtualizes sandbox residency by decoupling the logical agent's lifetime from its physical execution environment. To manage memory oversubscription, TenonAgent employs a dual-timescale control plane. At action boundaries, it combines predictive admission control with a fail-closed, restart-safe contract, dynamically materializing or dematerializing sandboxes based on global memory pressure. During active tool execution, a sched_ext-based kernel gate paces sandbox execution permissions to reduce unexpected concurrent memory growth. The gate provides feedback control over future execution opportunities. Evaluations across diverse agentic workloads show that, under the tested workloads and parameter settings, TenonAgent substantially reduces observed OOM failures and achieves a 100% task success rate in bursts reaching an 8.6x memory oversubscription ratio. Overall, it delivers up to a 5.5x speedup over unmanaged execution, outperforms existing agent-level and cgroup-based schedulers by 5x to 37x, and provides consistent benefits across both Docker and Firecracker microVM backends.

0.10

SCLATE: a Substrate for Continual-Learning Agent Training and Evaluation

cs.AI cs.DCcs.LG ·2026-09-26 ·Youngmok Jung, Sirajul Salekin, Henry Tran, Javier Movellan, Zhao Huang +1 more
benchmark similarity 0.67
Abstract

Continual-learning agents are systems of models, harnesses, and memory operating over long multi-session horizons. Evaluating and training them requires interleaving tasks with agent-side events such as session stop and start, crons, and memory consolidation. Yet existing benchmarks and training frameworks schedule only the benchmark's own events, leaving each benchmark and agent pair to build a custom scheduling loop. We present SCLATE, an execution substrate where benchmarks and unmodified agents each add their events to one open event scheduler through an adapter. A hybrid simulated clock runs these events on a shared timeline, flowing in real time while the agent works and skipping idle gaps, which compresses a month-long scenario into hours. SCLATE also serves as a rollout engine that runs any agent's harness and memory unmodified, recording the tokens and log probabilities of every model call through an in-container proxy. We port seven benchmarks to SCLATE and compare ten unmodified harness and memory configurations head to head on ten models. The comparison shows that an added memory system does not reliably beat the harness's native memory and that models differ widely in how they use the same harness and memory. We then post-train Qwen3.5-4B through unmodified harnesses and memory systems. The model learns to use both, reading 6.8x fewer file lines with a 16.7-point higher SWE-bench Verified pass rate, and writing richer memory records, while its held-out MetaClaw accuracy rises by up to 11.8 points.

0.25

Serve Now or Improve Later? Scheduling Self-Evolution in Online Agent Systems

cs.DC ·2026-10-05 ·Yangbo Wei, Junhong Qian, Zhen Huang, Zhenyu Su, Qifan Wang +4 more
similarity 0.66
Abstract · 21 pages, 17 figures

Online agents can improve future service by constructing reusable tools, guidance, or model states, but this work competes with current requests for the same GPUs. Exploiting idle compute for self-evolution faces a fundamental systems constraint: benefits arrive only after an artifact is published and used, while pausing evolution leaves service capacity waiting for memory release and runtime recovery. An investment worth completing may therefore be worth postponing. We present LearnSched, a state-aware scheduler that brings the reuse window of a capability and the timely return of compute into a common investment model. LearnSched incorporates candidate progress, checkpoint overhead, and the current recovery path into action values. Under the same information and capacity constraints, it uses one-step counterfactual rollout to choose progress, checkpointing, or waiting relative to a fully costed window policy. We characterize handoff costs through independent A100 component measurements and evaluate action selection in 1920 paired finite-model scenarios. When cold recovery is expensive, waiting for longer execution windows can improve net value by avoiding handoffs; in evaluated warm-recovery scenarios, the strong window policy already achieves the same value. Retaining recoverable state can shorten capacity return and reduce the need for complex evolution scheduling.

0.22