CS Paper Radar

Bolo: Verified Model Hub for Next-Generation AI Databases

Yunqi Li, Ila Petrovic, Yongjoo Park

cs.DB ·submitted 2026-08-20 19:38 UTC · v1
position

Abstract

Verified, ready-to-use inference pipelines are a cornerstone of future AI databases. They allow multi-modal databases to incorporate specialized language, vision, and tabular models that can deliver both high accuracy and efficiency. Unfortunately, existing model platforms such as Hugging Face fall short of this goal. While they host millions of model repositories, many contain only raw weights without runnable pipelines. Even well-documented models often fail due to missing dependencies, unsupported model classes, or incorrect task assignments. Moreover, different models fail for different reasons, with no uniform solution. Constructing a large-scale, verified model hub is nearly impossible with human effort alone. We argue that AI agents can achieve this at scale. We present \system, a model platform that hosts verified, ready-to-use inference pipelines, powered by a multi-stage agentic system for model remediation. For models that fail under standard usage, the agent inspects errors and repairs broken pipelines (Type~I). For models outside the scope of existing interfaces, it synthesizes pipelines from scratch using model metadata and documentation (Type~II \& III). To prevent incorrect pipelines from entering the database, the agent applies multi-stage verification---checking not only program structure but also semantic model behavior, ensuring pipelines produce meaningful outputs rather than merely executing without error. In preliminary experiments, \system achieves 97.27\% and 86.08\% runnable coverage for Type~II and Type~III models, respectively, demonstrating that agentic synthesis with targeted verification can transform large collections of unusable model weights into a verified database of ready-to-use inference pipelines. The preliminary database is open-sourced at \textcolor{blue}{https://bolobao.ai/}.

Comment
5 Pages

Closest papers

Nearest neighbours by meaning across the whole index.

Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses

cs.AI ·2026-09-28 ·Huachi Zhou, Yujing Zhang, Jiahe Du, Jiacheng Cai, Zijin Hong +5 more
code/data similarity 0.71
Abstract

Large language model agents are increasingly deployed for data-intensive work, yet reliable data analysis requires more than general-purpose reasoning and ad hoc tool augmentation. Data Agents, equipped with workflow harnesses, offer a promising paradigm for automating the end-to-end data science lifecycle. This paper examines Data Agents from a harness-centric perspective. First, we introduce a taxonomy of Data Agents and associated data environments, organizing the literature around five functional stages: perception, planning, execution, verification, and repair. Second, we analyze the key technical routes within each stage, identifying 15 distinct approaches ranging from data structure probing to data state reconstruction. Third, we identify four open reliability problems: inactive semantic calibration, missing clarification, missing experience transfer, and the missing verification-repair repository. These problems explain why silent failures can persist even when individual components function correctly, highlighting the need for rigorous workflow harnesses and shared reliability resources. Finally, we summarize the horizontal task families of Data Agents, examine their vertical application settings, and benchmarks for evaluation, while maintaining a companion repository at https://github.com/DEEP-PolyU/Awesome-Data-Agents.

0.25

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

cs.OS ·2026-10-31 ·Jiarui Guo, Zhi Zhang, Boyu Tan, Zongwei Lv, Haobo Sun +50 more
code/data similarity 0.69
Abstract

Large language models and inference techniques evolve rapidly, making model enablement a recurring production task rather than a one-time integration effort. At Alibaba, a shared inference infrastructure must keep pace with new models and serving capabilities while remaining reliable across heterogeneous production environments. To reconcile rapid evolution with production reliability, we present RTP-LLM, an open-source production inference engine developed and deployed at Alibaba. Its architecture separates model-specific execution from shared request processing, state management, and resource coordination. Explicit interfaces localize compatible model changes while allowing models and deployment configurations to reuse common serving mechanisms. RTP-LLM has evolved from an exploratory system into a shared production engine that supports $\sim$100 models and processes hundreds of trillions of tokens per day. We describe how its architecture and production practices accommodate model evolution and contain the impact of failures, and share lessons for building evolvable and reliable inference engines at enterprise scale.

0.28

Validation-Gated ML Pipeline Synthesis with Expert-Grounded Self-Evolving Skills

cs.AI cs.SE ·2026-11-03 ·Zixiao Zhao, Jing Sun, Zhe Hou, Mengze Li, Qian Liu +5 more
similarity 0.68
Abstract · 49th International Conference on Software Engineering

Recent advances in large language models have enabled agentic systems that automate machine learning pipeline construction from minimal user input. However, existing agentic AutoML methods either depend on rigid templates that limit adaptability, target narrow task domains, or rely on free-form code generation that lacks reliability guarantees, leaving intermediate errors to propagate silently and degrade the final model. We present EvoML, an agentic framework for end-to-end ML pipeline synthesis built on expert-grounded skills that evolve across runs, executed through a validation-gated harness. The framework decomposes pipeline construction into four phases coordinated by three agents: an Orchestrator that enforces validation gates, advancing each phase only when its outputs pass structural, execution, and semantic checks, which precludes error propagation and keeps every artifact auditable; an Executor that runs expert-grounded skills, grounded in the nnU-Net workflow and pairing reusable templates with online adaptation; and a Curator that performs post-run evolution, consolidating the scripts and guidance recovered during a run into the skill library so it improves across datasets. We evaluate EvoML on eight datasets spanning six sophisticated, real-world ML task types, against agentic AutoML systems and command-line coding tools. EvoML achieves full automation and the best final model quality on all eight, spanning segmentation (up to 0.94 Dice), classification (0.99 accuracy), reconstruction (0.95 SSIM), and denoising (0.79 normalized PSNR), while transferring to broader ML tasks.

0.07

Training LLM Agents to Automate LLM Post-Training with RSI-Dojo

cs.AI ·2026-10-29 ·Shuzheng Gao, Zhaojian Yu, Penghao Yin, Yuling Shi, Kai Cai +2 more
similarity 0.68
Abstract

Recursive Self-Improvement (RSI) aims to enable AI systems to autonomously build and improve AI systems themselves, including models, harnesses, and underlying infrastructure. Within this broader agenda, a central capability is enabling agents to train and improve base models. However, existing studies on automated model training primarily focused on workflow- or harness-level optimizations, rather than internalizing the ability to conduct post-training into the model itself. Developing this capability requires executable model training environments that can produce high-quality trajectories demonstrating how to select data, train models, diagnose failures, and iteratively improve results. In this paper, we introduce RSI-Dojo, a collection of executable model training environments designed to teach LLM agents to conduct post-training. To construct RSI-Dojo, we develop RSI-Craft, an automatic pipeline that converts human research artifacts into executable model training environments. RSI-Craft automatically collects research papers from top venues, instantiates datasets and metrics into a unified interface, and reproduces evaluation dependencies. It then filters tasks by data size, execution reliability, and discriminability, yielding RSI-Dojo with 281 seed model training tasks. Building on RSI-Dojo, we employ a trajectory collection method with an improved harness to collect high-quality model-training trajectories from frontier models, then retain examples through rubric-based and outcome-based filtering. We fine-tune Qwen3.5-9B and Qwen3.6-35B-A3B on these trajectories. The resulting RSI-Dojo-9B and RSI-Dojo-35B-A3B improve over their base models by 8.74 and 16.87 PostTrainBench points, respectively. RSI-Dojo-35B-A3B reaches 29.95 on PostTrainBench and 53.29 on PostTrainBench-Lite, matching the performance of frontier models such as GLM-5.2 and GPT 5.5.

0.30

MLOps-Bench: Benchmarking AI Agents on Production ML Engineering in Real Repositories

cs.SE ·2026-10-16 ·Quoc Co Tran, Hitesh Laxmichand Patel, Diwakar Mahajan, Kshitij Bakliwal, Avi Sil +1 more
top venuebenchmark similarity 0.68
Abstract

AI agents are increasingly expected to modify and maintain machine-learning systems in production, yet existing benchmarks primarily evaluate either competition-style ML experimentation or generic repository issue resolution. We introduce MLOps-Bench, a repository-grounded benchmark for production ML engineering. Unlike prior ML-agent benchmarks, MLOps-Bench evaluates executable lifecycle changes inside real ML repositories, spanning data pipelines, feature engineering, model training, model serving, monitoring, CI/CD and governance, and end-to-end integration. Each task is grounded in repository evidence and includes a natural-language requirement, oracle patch, plausible buggy baseline, fail-to-pass tests, pass-to-pass tests, and provenance metadata. The benchmark contains 212 validated tasks from 31 public ML repositories, with 636 fail-to-pass tests and 856 pass-to-pass tests. We evaluate six agents and find that the best system solves 31.2% of tasks while passing 68.7% of tests, showing that current agents often make partial progress but still fail production-relevant behavioral requirements. MLOps-Bench shifts agent evaluation beyond model building and issue fixing toward repository-native MLOps work where correctness depends on lifecycle reasoning, behavioral robustness, and preservation of existing system behavior.

0.36

Raven: The Harness of Harnesses for Composable Agentic Intelligence

cs.AI cs.CLcs.GTcs.MA+1 ·2026-09-27 ·EverMind AI
code/data similarity 0.68
Abstract

As large language models advance, AI agents are moving beyond isolated, domain-specific tasks toward long-horizon, cross-domain workflows. This transition exposes two challenges: increasing harness complexity makes manual design difficult to scale, while tighter coupling to specific domains limits the generality of a single harness. The central question thus shifts from how to engineer a stronger harness for one domain to how to autonomously construct specialized harnesses, improve them through experience, and orchestrate them across domains. We introduce Raven, \emph{The Harness of Harnesses}, an open-source multi-agent ecosystem that automatically constructs and evolves modular harnesses for specific models and domains, treating each executable model--harness pair as a composable unit of intelligence. To support an \emph{All-Domain Collaboration Network}, its Host Agent decomposes goals, matches subtasks to specialized agents, coordinates execution dependencies, and integrates results, while a host archive and EverOS preserve experience across tasks and Skill Forge makes that experience available as reusable procedures. Our theory establishes sufficient conditions for such composition to expand reliable task coverage beyond that of the available individual agents under a shared resource budget. On complex and long-horizon tasks, Raven significantly outperforms the state-of-the-art agent systems, pushing the frontier of composable agentic intelligence.

0.45

ML Engineering Agents - A Data Management Perspective

cs.DB cs.LGcs.SE ·2026-10-12 ·Arnab Phani, Elias Strauss, Tim Schuster, Sebastian Schelter
similarity 0.67
Abstract · 6 pages

Machine learning engineering (MLE) agents are emerging as a new paradigm for automating end-to-end ML pipeline development. Using large language models for code generation and predictive performance as a machine-gradable feedback signal, these agents iteratively synthesize, execute, and refine pipeline code. Yet current systems are largely evaluated on pre-built, single-table benchmarks that abstract away key data management challenges. This tutorial introduces the foundations of MLE agents and highlights research opportunities for the database community. We cover how agents combine program synthesis with iterative and evolutionary optimization, and discuss open challenges in efficient pipeline execution, data discovery and augmentation over heterogeneous sources, reliability and debugging, as well as fairness and compliance. We connect these challenges to existing data management techniques and conclude with a hands-on session using contemporary MLE agents.

0.23

ToolSpace: Scaling Tool-Centric Environment Synthesis for Long-Horizon Agents

cs.CL ·2026-11-03 ·Guanting Dong, Zhichao Sun, Xinyu Yang, Rui Wang, Xin Deng +20 more
community pick similarity 0.67
Abstract · Working in progress

Large language models are deployed as long-horizon agents that compose diverse tools across MCP services into executable workflows. Existing MCP environment-scaling methods are data-centric: they first synthesize a database and scenario, then derive tools and tasks from them. As a result, tool interfaces may not match real ones, tool compositions are not grounded in execution, and tasks rarely span more than one environment. We propose ToolSpace, a tool-centric framework that reverses this order. ToolSpace mines data, state, and retry dependencies from adjacent calls in execution traces of real MCP services and open agent corpora, using deterministic rules rather than LLM judgment, and clusters the resulting weighted tool-composition graph into environments. Databases are synthesized only for state-mutating environments, while query and stateless tools replay recorded returns. ToolSpace then performs state-aware random walks with rollback and decompiles each chain into a natural-language task with assertion-based checks, keeping only samples that replay successfully from the initial state. Applied to AutomationBench and tau3-Bench Banking, the pipeline produces harder Pro versions on which 7 frontier models drop by up to 38.2 points. Across seven long-horizon tool-use benchmarks, ToolSpace raises Qwen3.6-35B-A3B from 23.0 to 41.0 average score (43.2 with RL) and GLM-5.2 from 44.6 to 56.6, on par with Claude Opus 5. Ablations show that execution-witnessed edges clearly outperform LLM-judged ones, and performance keeps improving with data and environment scale.

0.20