CS Paper Radar

Open at the Edge, Captured at the Center: llama.cpp and the Political Economy of Local AI Inference

Woohyeuk Lee, Hanlin Li, David Gray Widder

cs.CY ·submitted 2026-08-19 15:02 UTC · v2

Abstract

Critical scholarship on open AI has focused on model releases and cloud ecosystems, leaving the local inference infrastructure that makes open-weight models runnable on user-owned devices largely unexamined. We address this gap through a mixed-methods analysis of llama$.$cpp, combining 7,681 merged pull requests from March 2023 through March 2026 with repository discussions, corporate statements, and contributor blogs. We show that local inference broadens participation at execution while relocating capture into the infrastructure that makes execution possible. Through hardware backends, model integration labor, and Hugging Face's February 2026 absorption of the project, we document how control shifts to hardware vendors, model distributors, and core maintainers while model owners and individual contributors bear the cost of making models runnable. These dynamics suggest that preserving openness outside the cloud requires attention to the infrastructure that makes models runnable, not just to the models themselves. This calls for policy mechanisms--analysis of format dependencies and vendor influence, model compatibility requirements, and sustained public funding for inference tooling--that extend beyond model release conditions to the infrastructure layer.

Comment
18 pages, 7 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Thomson: Continual Learning of Frontier Models for SovereignAI

cs.AI ·2026-08-27 ·Shengzhuang Chen, Jerrod Parker, Yejin Bang, Andrew M. Bean, Nabeel Seedat +21 more
positioncode/data similarity 0.68
Abstract · Open-weight model: https://huggingface.co/thomsonreuters/Thomson-1.0-Small

The development of frontier models is commonly perceived to be the exclusive remit of a small number of heavily funded players, creating an information, economic and power asymmetry between developers and the diverse user base of modern AI. Recent public discourse acknowledges this concern, calling for SovereignAI (an organisation's capability to independently build, deploy and govern AI use), but offers little concrete advice on how this can be achieved in the short term under a diversity of funding settings. We argue that frontier performance is achievable by a wide range of institutions through Continual Learning on readily available open-weight models. Unlike limited approaches such as small-scale fine-tuning, prompt engineering, or tool-augmentation of a frozen model, our approach exploits a modern mid- & post-training stack while introducing safeguards that preserve both plasticity and stability at each stage, making the minimal number of high-impact interventions on the parameters. This yields gains comparable to those typically seen across multiple successive model generations, at compute and personnel budgets substantially lower than commonly thought, making ownership of large parts of the SovereignAI stack (model, tool infrastructure, values & data privacy) viable for far more actors. We demonstrate this with Thomson, a general-purpose frontier model trained with an enhanced focus on high-stakes professional work. Thomson performs competitively with recent frontier models across agentic tasks, safety, legal, tax & multilingualism, and large-scale Deep Research. Evaluations show a distinctive $π$-shaped pattern: distinct improvements across a wide range of capabilities, including those not explicitly targeted, while almost completely eliminating the forgetting problem common to narrow domain adaptation.

0.47

RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale

cs.OS ·2026-10-31 ·Jiarui Guo, Zhi Zhang, Boyu Tan, Zongwei Lv, Haobo Sun +50 more
code/data similarity 0.65
Abstract

Large language models and inference techniques evolve rapidly, making model enablement a recurring production task rather than a one-time integration effort. At Alibaba, a shared inference infrastructure must keep pace with new models and serving capabilities while remaining reliable across heterogeneous production environments. To reconcile rapid evolution with production reliability, we present RTP-LLM, an open-source production inference engine developed and deployed at Alibaba. Its architecture separates model-specific execution from shared request processing, state management, and resource coordination. Explicit interfaces localize compatible model changes while allowing models and deployment configurations to reuse common serving mechanisms. RTP-LLM has evolved from an exploratory system into a shared production engine that supports $\sim$100 models and processes hundreds of trillions of tokens per day. We describe how its architecture and production practices accommodate model evolution and contain the impact of failures, and share lessons for building evolvable and reliable inference engines at enterprise scale.

0.28

How Developers Configure LLM Inference: A Large-Scale Study of Real-World Practices

cs.SE ·2026-10-20 ·Nada Zine, Matias Martinez, Romain Rouvoy, Clément Quinton
code/data similarity 0.64
Abstract

LLMs are increasingly integrated into software systems, making deployment and inference configuration a recurring software engineering task. Inference backends expose many options that affect model loading, generation, serving, and request processing. Yet little is known about which of these options developers explicitly configure in practice. In this paper, we present a large-scale empirical study of LLM inference configuration practices in open-source software using Hugging Face Transformers, vLLM, and the OpenAI API. We analyze 5,398 GitHub repositories using at least one of these backends and identify 345,117 explicitly configured option occurrences. Our results show that developers use only a small part of the available configuration space regularly. Although 98.8% of the documented backend-specific options appear at least once, 78.9% are configured in at most 10% of repositories using the corresponding backend. Configurations are also small, with a median of 2 options for Transformers and 4 for vLLM and OpenAI, but the combinations of options vary widely: 56.8-61.6% of multi-option signatures appear in only one repository. We also observe recurring option values as well as substantial variation in how values are chosen. Finally, configuration practices differ across application contexts, with option-adoption differences reaching 30.7 percentage points. These findings characterize the configuration surface developers make explicit rather than the importance or optimality of individual options. They provide empirical evidence for prioritizing future studies of inference configuration and for improving backend documentation and configuration support.

0.29

Regulating for an AI Commons: Adopting, sustaining, and defending people-first innovation trajectories

cs.CY ·2026-10-16 ·Abhineet Nayyar, Mallory Knodel, Kimberlee Weatherall, Gunn Jiravuttipong
position similarity 0.63
Abstract

Technological principles of openness, interoperability, and decentralization are integral to building an AI economy that is accountable to the public and truly responsive to local needs and constraints. However, Big Tech's firm control over essential elements of the stack (such as chips, cloud, models, data, and talent) limits valuable alternatives from reaching the scale necessary for sustained growth. Worse still, hyperscalers continue to enclose and extract value from publicly funded and volunteer-driven technologies without adequately distributing these returns. We see consolidated power within a few entities and "enshittification" of once-transformative innovations. Embedding ethical micro-level tools (like open data repositories or small-language models) into workflows is promising, but without a consistent innovation strategy, they also often end up perpetuating and being appropriated by the same exploitative trajectories. Instead, this whitepaper outlines a three-pillared holistic and intentional policy approach that i) actively pushes for the adoption of cooperative technologies in existing and emerging systems, ii) sets up collective institutional structures to sustain and maintain them in the long-term, and iii) introduces regulatory proposals that limit private entities from disempowering them through uncontrolled enclosure and extraction. Through desk research and purpose-specific case studies, we argue for the importance of such an approach in the Global South, where structural constraints around infrastructure, financing, and policy governance make it difficult for AI commons and cooperative AI tools to emerge and survive. The paper concludes with key open questions for campaigners, policy advocates, researchers, and grassroots organizers to consider and explore while working towards peoples' solidarity within and between Global South countries.

0.15

The Illusion of Local Privacy: Confidentiality Boundary Failures in Consumer LLM Serving Systems

cs.CR ·2026-09-16 ·Youssef Hamdi Zafan Ibrahim, Muhammad Ikram, Mohammed Khalaf Salama
positioncode/data similarity 0.63
Abstract · 21 pages, 2 figure, 13 tables. Code and experimental artifacts: https://github.com/0xzodiac/Local-LLMs-Forensics, Preprint for IEEE Transact…

Running large language models (LLMs) locally is often considered more private than cloud-hosted inference because user prompts remain on the device. We ask whether keeping inference local is, by itself, sufficient to keep those prompts confidential. Our results show that it is not: prompt confidentiality also depends on how the surrounding serving software handles prompt data before, during, and after inference. We examine four boundaries at which prompt confidentiality can fail in consumer local-LLM serving systems: model loading, runtime memory, wrapper-level persistence, and the serving interface. To study these boundaries, we develop LLAnalyzer, a measurement framework that tests each boundary separately and traces observed failures to the responsible software component. Applying LLAnalyzer to four open-weight model families and two consumer deployment platforms, we find markedly different behaviour across boundaries. In a 24-hour AFL++ campaign with more than 12 million executions, we observe no parser crashes or successful malformed GGUF loads within the explored state space. Runtime memory tells a different story: we recover prompts after inference because multiple plaintext representations survive in allocator-managed memory, and sanitisation reduces this residue without eliminating it. We also find that consumer wrappers can extend prompt lifetime through plaintext persistence. At the serving boundary, we uncover a previously undocumented authorization flaw in llama.cpp that allows one authenticated client to restore another tenant's saved conversation state; the attack succeeds in 200/200 controlled trials. Separately, shared prompt-prefix caching exposes a remote timing oracle that remains distinguishable under WAN conditions. We argue that local LLM systems need explicit guarantees for prompt lifetime, persistent storage, and tenant isolation.

0.36

Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

cs.AI cs.SE ·2026-08-25 ·Leonardo Liparulo, Francesco Pierri
top venuebenchmarkcode/data similarity 0.63
Abstract

We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. In these environments, engineers issue repetitive, dependency-ordered operations---such as creating components, adding ports, and wiring connections---through specialised tools. Confidentiality constraints on component specifications and naming conventions often preclude hosted proprietary APIs, motivating the use of locally deployed models. To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development and construct a benchmark covering single-operation edits, multi-step dependency chains, invalid requests, misspelled prompts, and multi-server tool contexts. We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures. Results show that strong models can achieve near-complete expected-call coverage on the benchmarked workflows, but reliability depends strongly on both task structure and agent configuration. Comprehensive tool descriptions consistently reduce failures, few-shot prompting can cause severe inaction for some models, cumulative context harms constrained models, and multi-agent decomposition helps weak workers or long sessions at the cost of additional calls. These findings provide practical guidance for deploying local LLM agents in stateful hardware design environments.

0.47

Bolo: Verified Model Hub for Next-Generation AI Databases

cs.DB ·2026-08-20 ·Yunqi Li, Ila Petrovic, Yongjoo Park
position similarity 0.62
Abstract · 5 Pages

Verified, ready-to-use inference pipelines are a cornerstone of future AI databases. They allow multi-modal databases to incorporate specialized language, vision, and tabular models that can deliver both high accuracy and efficiency. Unfortunately, existing model platforms such as Hugging Face fall short of this goal. While they host millions of model repositories, many contain only raw weights without runnable pipelines. Even well-documented models often fail due to missing dependencies, unsupported model classes, or incorrect task assignments. Moreover, different models fail for different reasons, with no uniform solution. Constructing a large-scale, verified model hub is nearly impossible with human effort alone. We argue that AI agents can achieve this at scale. We present \system, a model platform that hosts verified, ready-to-use inference pipelines, powered by a multi-stage agentic system for model remediation. For models that fail under standard usage, the agent inspects errors and repairs broken pipelines (Type~I). For models outside the scope of existing interfaces, it synthesizes pipelines from scratch using model metadata and documentation (Type~II \& III). To prevent incorrect pipelines from entering the database, the agent applies multi-stage verification---checking not only program structure but also semantic model behavior, ensuring pipelines produce meaningful outputs rather than merely executing without error. In preliminary experiments, \system achieves 97.27\% and 86.08\% runnable coverage for Type~II and Type~III models, respectively, demonstrating that agentic synthesis with targeted verification can transform large collections of unusable model weights into a verified database of ready-to-use inference pipelines. The preliminary database is open-sourced at \textcolor{blue}{https://bolobao.ai/}.

0.33

Semantics, Workflows, and Infrastructure: Understanding Agent Serving at Production Scale

cs.DC ·2026-09-28 ·Yihao Zheng, Jingzhe Jiang, Dejiang Zhu, Zhiyuan Tan, Yang Tian +2 more
similarity 0.62
Abstract

Large language model (LLM) agents execute applications through a workflow of inference requests with tool calls and user interactions. Serving these applications at production scale requires understanding how application behavior shapes inference demand and for guiding efficient execution. Recent characterization studies provide request-level workload measurements and agent execution analysis. However, an end-to-end view connecting task initiation, workflow execution, and inference infrastructure remains unexplored. In this paper, we analyze a two-week trace of 11.7 million requests from a large-scale production platform for general-purpose agents, backed by inference infrastructure comprising over 10k GPUs. We characterize the platform at three connected levels: task-level initiation semantics, workflow-level execution patterns, and infrastructure level serving demands. Our measurements reveal workload patterns such as highly skewed request volumes across sessions, rare execution overlap among logical sibling requests, and context reuse across task boundaries. Building on these observations, we analyze deployment implications and identify open problems to guide future research on agent serving systems.

0.42