CS Paper Radar

OfficeTown: Generative Organizations for Evolving Agent Intelligence

Hongchen Wei, Daohong Ying, Zhi Hong, Aidan Wu, Bei Liu, Ruichun Ma, Yifan Yang, Qi Dai, Kai Qiu, Rui Xu, Dongdong Chen, Zhenzhong Chen, Chong Luo

cs.AI ·submitted 2026-10-16 09:32 UTC · v1
community pick

Abstract

AI agents increasingly use files and software tools to produce professional deliverables. Such work draws on materials, decisions, and feedback accumulated across assignments. We introduce OfficeTown, a generative simulation framework for professional organizations in which tasks and their context arise from continuing organizational work. From a natural-language description of an organization's purpose and activities, OfficeTown initializes an organization whose members coordinate work, delegate bounded assignments to assistant agents, and review their deliverables. Artifacts, decisions, and unresolved matters persist and shape subsequent work, while responsibilities and permissions determine what each member can see and do. We curate work requests, their context, and review feedback into executable resources for evaluation and learning. We construct OfficeTown Bench with 406 tasks and 4,836 evaluation criteria from 23 simulated organizations across eight professional domains, incorporating domain-expert feedback. Frontier-model evaluations reveal substantial remaining headroom. Supervised fine-tuning and on-policy distillation on OfficeTown data substantially improve Qwen3.5-9B on external professional-work benchmarks, including GDPval. Supplementary evaluations show that the principal--assistant harness achieves higher scores than the single- and multi-agent baselines tested. These results highlight the value of organizational simulation in generating work-grounded data and environments for the continued development of generalist agents.

Closest papers

Nearest neighbours by meaning across the whole index.

AdaPRA: Process Reward Agents with Adaptive Rubrics for Data Science

cs.AI cs.LG ·2026-10-23 ·Atharva Naik, Vivek Chari, Pol Llado, Daniel Fried, Carolyn Rosé +3 more
benchmark similarity 0.66
Abstract

Data science agents can automate end-to-end machine learning workflows, yet they remain far from competitive on human leaderboards. Agents make process errors such as insufficient exploration, unsound validation, coding bugs, and poor use of a fixed budget (re-running the same exploratory data analysis and baseline models in every attempt). We propose AdaPRA to address these issues: a persistent coding agent harness that accumulates knowledge across attempts on the same task, with rubric-guided process supervision that identifies and corrects errors. Our process reward agent audits the agent's trajectory and workspace by executing checks, and adapts its rubrics to each task. We find that rubrics seeded from easier tasks generalize to harder tasks, with rubric adaptation further improving performance. With five attempts per task, AdaPRA improves an 80B Qwen model's leaderboard percentile by 24.5 points on DSPredict-Easy and 14.8 points on DSPredict-Hard while reducing process errors. We also show that supervision with rubrics mined from one base model's runs improves another (DeepSeek-V3.2), and that AdaPRA beats DataPRM, a competitive PRM for data science. We also share our code for reproducibility.

0.29

SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving

cs.CL ·2026-09-23 ·Zhilong Ge, Yuting Shao, Yutao Yang, Yuxuan Cai, Jie Zhou +4 more
similarity 0.65
Abstract

Human-written agent skills encode rich workflows for real-world problem solving, but are typically used as external inference-time instructions rather than internalized as reusable model capabilities. We introduce \texttt{SkillGym}, a framework that transforms these skills into executable, verifiable training environments for large language model agents. Its skill-to-task pipeline instantiates concrete tasks, verifies outcomes with code-based checkers, and assesses empirical skill dependence through contrastive executions. We construct and release 2,756 environments across 12 categories and collect 8,364 successful trajectories from multiple models and harnesses, averaging 49 tool calls and over 60k logged text tokens. These resources support supervised fine-tuning on verified workflows and reinforcement learning with outcome-based rewards. Under Claude Code, supervised fine-tuning improves Qwen3.5-35B-A3B by 199 Elo on GDPval-AA v2, 19.10 percentage points on Terminal-Bench 2.1, and 28.13 and 12.38 points on SkillsBench v1.1 with and without skills, respectively. Our 35B \texttt{SkillGym-Agent} reaches 51.47\% on skill-assisted SkillsBench, exceeding reported scores for Claude Sonnet 4.6, GPT-5.4 Mini, and DeepSeek V4 Pro. Without skills, it also surpasses skill-assisted bases under Codex and Claude Code, suggesting reusable procedural competence.

0.17

WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks

cs.AI ·2026-09-20 ·Yining Hua, Levi Lian
similarity 0.65
Abstract · 4 figures

Many knowledge-work benchmarks are constructed around individual tasks, with the context needed for each task selected together with or after the task has been specified. This design measures performance on workplace-like tasks in an environment assembled for the task. When task specification guides which context is selected, the evaluation can encode task information into the environment and pre-complete part of the information-localization work that workplace performance normally requires. We introduce WorkWorlds, an evaluation infrastructure that separates organizational state from task specification. A world first fixes a revision, date, and employee seat and materializes the organizational state that employee can access; tasks are introduced only afterward. We implement WorkWorlds in a primary synthetic pharmaceutical company with 8 measured tasks across 6 employee seats, and construct additional organizational worlds. Across 192 matched evaluations, task-level curation increased evidence access by 17.6 percentage points, from 72.8% to 90.4%, and criterion pass by 8.7 points, from 68.0% to 76.7%, while pass conditional on evidence access remained nearly unchanged; most of the measured difference occurred before the agent reached sufficient evidence.

0.20

EvoCowork: A Benchmark for Real-World Office Tasks

cs.AI ·2026-10-10 ·Zhou Liu, Xingyan Chen, ZhaoYang Han, Zewei Pan, Zeli Su +4 more
benchmarkcode/data similarity 0.65
Abstract · 32 pages

Office workflows are a demanding testbed for self-improving agents because their deliverables must be correct, natively editable, and mutually consistent. A central question is whether gains on feedback tasks reflect reusable procedures rather than task-specific fixes. We introduce EvoCowork, a 240-task benchmark for evaluating harness-level recursive self-improvement across four office modalities, with model weights held fixed. The benchmark covers presentations, documents, spreadsheets, and cross-office workflows, and scores submitted artifacts for content, native structure, formulas, cross-file consistency, and visual quality where needed. Its protocol separates feedback, revision selection, and final testing: methods learn from single-modality tasks, then are evaluated on unseen single-modality tasks and cross-office workflows under a shared initial harness and explicit budgets. In static evaluations of two models and four harnesses, the best configuration scores 42.25 overall and 30.62 on cross-office tasks. Starting from the same MSA harness with GLM 5.3, ACE and AHE improve in-domain scores by 4.90 and 2.64 points, respectively, but change cross-office scores by only +0.03 and -1.72. Thus, gains on the file types that provide feedback do not necessarily transfer to coordinated workflows. EvoCowork makes this distinction explicit while recording the capability, transfer, and cost of harness evolution. Code and task packages: https://github.com/DeepThinkingZhouLiu/cowork-evolution-benchmark.

0.47

Efficient Test-Time Adaptation through Human-AI Interaction

cs.AI ·2026-09-03 ·Zora Zhiruo Wang, Apurva Gandhi, Rulin Shao, Aspen Chen, Jonas Mueller +20 more
position similarity 0.65
Abstract

AI agents are trained on population-scale data to encode broad capabilities spanning those of many practitioners. Yet the artifacts they produce rarely meet the personal bar professionals need to stake their reputation on. On realistic, open-ended tasks where success criteria are heterogeneous and insufficiently documented, individual expertise lives precisely in the elevation and departure from the average. In practice, iterative human-agent interaction surfaces criteria that users cannot fully specify up front, yet apply repeatedly across tasks. We argue this cross-session interaction data is a rich, underused signal for closing the gap to individual expertise. In this work, we propose test-time adaptation through human-agent interaction (TAHI), which integrates these signals into agent context and weights, and crystallizes each user's training and evaluation criteria via an evolving rubric module. We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks. Our agents improve solo task success by 4.5-20.9% within only tens of tasks. Meanwhile, our evolving rubric module serves as a scalable annotation tool, creating evaluation rubrics that catch 16.0-22.3% more failures than those from LMs or humans alone. While agents are adapted towards individuals, we show these personalized agents also produce improvements in success of up to 8.8% that generalize across users.

0.14

WorkFactory: Turning Professional Intents into Executable Work for Scalable Agent Training

cs.AI ·2026-10-27 ·Ziyang Zeng, Tingyang Chen, Jia Deng, Qiyao Wang, Shuo Tang +8 more
code/datacommunity pick similarity 0.65
Abstract

Scaling supervised fine-tuning for working agents requires not only broad coverage of professional tasks, but also expert demonstrations that faithfully carry out professional workflows to completion. However, missing or inconsistent task information can disrupt expert demonstrations, weakening the resulting supervision. We introduce WorkFactory, a scalable task-construction framework that turns professional intents into executable task environments through hybrid intent sourcing, semantic workflow planning, and execution-based repair. Starting from these intents, WorkFactory constructs a Semantic Workflow Graph that explicitly represents information requirements, dependencies, and locations within the task environment. The graph guides the instantiation of task environments, while an adversarial execution-refinement loop exposes missing or inconsistent information and steers targeted repairs that preserve the original intent. Using WorkFactory, we construct 112,686 tasks spanning 612 occupations and train the WorkPilot family at 4B, 27B, and 35B-A3B scales on the resulting teacher trajectories. All three models improve over their respective base models across four professional-work benchmarks, with WorkPilot-35B-A3B rising from 879 to 1565 Elo on GDPval-AA v2.1. Scaling experiments with up to 110k tasks further show that larger models benefit more from additional task data but exhibit diminishing returns at smaller data scales, while WorkPilot's long-horizon execution evolves from verbose trajectories toward more efficient task completion. Controlled component studies quantify the negative effects of task-environment defects on downstream performance and show how semantic workflow planning and execution-based repair yield downstream gains through improved task-environment quality. The subset of data are available at https://github.com/IQuestLab/WorkFactory.

0.36

Design and Cross-Domain Evaluation of an Agent Execution Harness

cs.AI ·2026-10-18 ·Saurav Joshi, Manas Gupta, Ashwin Venkatesh Raman, Akshay Budhkar
similarity 0.64
Abstract · 25 pages, 11 figures, including references and appendices

Completing complex tasks with language-model agents requires coordinating planning, execution, and review so the final result addresses the user's original objective. We present an agent execution harness that keeps the original objective separate from a revisable plan and an execution record. A planner assigns scoped tasks, workers produce outputs and artifacts, and a reviewer examines results against the original request before submitting or requesting additional work. Reusable domain-specific instructions guide these roles and are refined during construction using task inputs and execution trajectories, without official grading feedback. We evaluate complete harness configurations across terminal tasks, clinical responses, application building, and scientific questions. On Terminal-Bench 2.1, the harness passes 83 of 89 instances (93.26%); on HealthBench Professional, a Fable-configured system achieves a length-adjusted rubric score of 69.63 across 525 tasks. In a 53-task paired HealthBench comparison, the Fable-configured harness achieves an aggregate score 19.89 points higher than direct completion, despite a $-1.46$-point median task-level change; mean inference cost rises from \$0.29 to \$2.13 per task. We combine these outcomes with latency, documented instruction revisions, and reviewer decisions. Evaluating outcomes alongside execution trajectories and resource use clarifies how coordination supports task completion, where it introduces regressions, and at what cost.

0.21

DataPraxis: Scaling Analytics Tasks via Verified Workspace Evolution

cs.AI ·2026-10-17 ·Nan Jia, Wenxuan Shi, Jingqing Ruan, Zhizhao Zeng, Tianyi Zhai +9 more
datasetcode/data similarity 0.64
Abstract

General-purpose agents are increasingly expected to perform professional analytics work, which requires sustained reasoning, iterative investigation, and reliable evidence. Training these capabilities requires diverse and challenging tasks. However, generating tasks directly from raw data often overlooks deeper questions that emerge through analysis and provides limited evidence for verification. To address these limitations, we introduce DataPraxis, an analytics task synthesis framework based on verified workspace evolution. The framework progressively enriches workspaces with validated findings and artifacts, then inherits and combines these workspaces to uncover deeper questions. It constructs tasks from accumulated findings and their dependencies, verifying analytical results both during evolution and after task assembly. Using this framework, we construct DataPraxis-5K, a dataset of 5,000 verified trajectories spanning diverse domains and output formats, to train DataPraxis-27B and DataPraxis-35B-A3B. DataPraxis-27B achieves 55.11 on DABStep and 51.9 on JobBench, matching or surpassing frontier models such as GLM-5.2. DataPraxis-5K is available at huggingface.co/datasets/stargazerhust/DataPraxis-5k

0.28