CS Paper Radar

An Experimental Study on the Functional Correctness of Network Device Configurations Produced by Large Language Models

Anastasios C. Politis

cs.NI ·submitted 2026-10-14 10:40 UTC · v1
top venue

Abstract

This paper presents a preliminary study on the efficacy of network device configuration produced by Large Language Models (LLMs). Currently, the study focuses on routing protocol configurations in a predefined small internetwork with Cisco-based routing devices. The capability of three freely available LLMs with standard capabilities (i.e., with no enhanced features) to produce effective setups is investigated. Prompt engineering methods are excluded in order to assess the "first-response" configurations of the models under consideration. The assessment method is based on the contents of the resulting routing table, aiming at measuring the device functional correctness. For that purpose, the Jaccard index is used to indicate the percentage of correct network prefixes contained in the routing table constructed by the model-based configuration commands. The work provides insights on the maturity and limitations of modern LLMs on aiding network administrators with network device configurations in real deployments.

Comment
5 pages, 4 figures, 2026 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT)
DOI
10.1109/IAICT71158.2026.11621016

Closest papers

Nearest neighbours by meaning across the whole index.

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

cs.NI cs.AI ·2026-08-24 ·Chang Liu, Xiaohui Xie, Xinyi Chen, Yong Cui
benchmark similarity 0.65
Abstract

Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poorly understood. An essential prerequisite is to assess such agents in a realistic but risk-free environment. Existing benchmarks, however, fall short: they often treat configuration as static command generation or rely on overly simplified settings. Such evaluations understate the core challenges of network configuration, where correctness requires reasoning about protocol complexity and topology dependence. We present NetConfArena, an executable benchmark for evaluating LLM agents in closed-loop network configuration. NetConfArena places agents in emulated multi-device networks, provides a standardized and compact action interface for task execution, and evaluates the resulting network behavior with hidden task-specific executable test cases. The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates. We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, yielding 3840 execution trajectories, and show that failures are not limited to command errors. The failures also reveal gaps in task-specification adherence and robust planning and execution. These findings suggest two future directions: using validated trajectories as supervision signals to improve foundation models, and designing harness mechanisms that make agent execution more reliable and accountable.

0.38

Self-evolving network verifiers

cs.NI cs.AI ·2026-08-11 ·Ioannis Protogeros, Tibor Schneider, Laurent Vanbever
position similarity 0.60
Abstract · 8 pages, 5 figures

Symbolic network verifiers can reason about correctness across vast spaces of routing inputs and failures, but only for the protocols and features an expert has encoded by hand. Creating and maintaining a faithful model of the control plane is both difficult and never-ending, since no written source specifies perfectly what a network does: vendor implementations deviate from the RFCs, and behaviour shifts with releases. The burden of constant upkeep ultimately keeps verification out of many networks that need it. We argue that the model should instead evolve automatically to faithfully capture the actual network behaviour. To achieve that, we leverage the only source that specifies it unambiguously: the router software itself. In a counterexample-guided loop, a coding agent proposes extensions to the verifier's symbolic encoding, while a trusted oracle (e.g., emulated routers) supplies the ground-truth routing state. The agent iteratively refines the network model using each disagreement with the oracle. As early evidence, a prototype of this system taught a 3,000-line SMT-based verifier three features it did not support: OSPF areas, BGP route reflection, and L3VPN over EVPN, converging autonomously on models that match the oracle, even noticing vendor-specific behaviour. Automating model growth shifts the hard problem from writing verification systems to systematically testing them; we propose a research agenda for trusting and harnessing automatically evolved verifiers.

0.10

RFCLLM: Evaluating LLMs' Reasoning Ability of Network Protocol State Machines

cs.CL ·2026-09-11 ·Anqi Chen, Dan Goldwasser, Cristina Nita-Rotaru
top venue similarity 0.60
Abstract · Accepted at EMNLP 2026 (Findings)

Mapping textual specifications into formal representations is essential for ensuring the correctness of protocol designs and implementations. LLM-generated mappings, used for networking security or testing, are assumed to capture a perfect understanding of the specification, which may not hold in practice. The goal of this paper is to assess the extent to which LLMs can interpret the specification correctly. We examine the degree to which an LLM's implicit representation of a finite-state transition system-defined via natural language descriptions-aligns with a manually generated ground-truth model. We designed 4 tasks and 1482 task queries for 16 protocols. We evaluated different judge biases, observed the inherent difficulty gaps between tasks, looked into the effect of 4 context types, and the influence of protocol characteristics. Our work contributes to a step toward verifying whether LLMs can really be trusted in FSM (Finite State Machine) reasoning of protocol specifications.

0.37

PEAM: On-device Communication Agent for Personalized, Efficient and Autonomous Mobile Network Configuration

cs.AI ·2026-10-10 ·Haowen Wan, Shangzhuo Xie, Syue-Ju Syue, Xianyue Wu, Runhua Li +1 more
similarity 0.59
Abstract

The growing deployment of compact large language models on smartphones is enabling autonomous on-device decision-making, such as emerging mobile GUI agents. However, many mobile decisions, such as network configuration when users switch apps or connectivity conditions change, must satisfy stringent latency constraints to preserve user experience. In this paper, we propose PEAM, an on-device LLM-based communication agent that autonomously generates personalized network-configuration rules executable within milliseconds. Specifically, PEAM first fine-tune the on-device LLM with a dataset constructed from manually annotated real-user data to integrate the communication knowledge into the local model. PEAM then operates in two phases. In the rule-generation phase, which is performed when the device is idle, such as overnight, PEAM analyzes historical network-configuration records in batches and reflects across batches to consolidate consistent evidence, reduce local bias, and derive robust configuration rules. To enable more accurate and stable rule generation over time, PEAM further constructs a user profile from high-confidence rules to capture the user's network preferences and incorporates this profile into each rule-generation cycle. In the rule-execution phase, PEAM is invoked when the communication context changes, such as when users switch apps or connectivity conditions change, and searches the rule library to identify and execute the appropriate rule. Experiments demonstrate that PEAM, based on a fine-tuned 4B model, achieves 99.13% correct network configuration decisions, while 92.50% of the rules generated by PEAM are valid, outperforming the traditional expert-rule based method by 4.41% and 84.50%, respectively.

0.31

From Intents to Algorithms: Verified Algorithm Discovery for Transport Networks

cs.NI cs.MA ·2026-09-23 ·Behnam Ojaghi, Ricard Vilalta, Raul Muñoz
theory similarity 0.59
Abstract

Intent-based networking decouples desired outcomes from device-level configuration, but most systems still map intents to parameters of an algorithm selected in advance. Large language models (LLMs) create an opportunity to automate algorithm design, yet unrestricted generated code is unsuitable for transport-network control because feasibility, reproducibility, and robustness must be enforced independently of the model. We present VERA-TN, a verification-guided framework that compiles a network intent into a bounded algorithm-design specification. The target architecture uses an LLM as a semantic variation operator over typed request-ordering and path-ranking programs; generated logic remains separated from a trusted allocator that enforces path validity, latency, capacity, and single-path constraints. We prove feasibility preservation under explicit assumptions and establish a sufficient bound for the lexicographic latency tie-break in the exact reference model. The released proof-of-concept instantiates the same interface with a bounded ten-parameter numerical candidate and deterministic replay, rather than a completed live-LLM/AST study. Across 150 certified held-out cases on a 28-node TEFNET24-derived hierarchy, evolutionary search reaches a mean priority-utility ratio of 0.958, compared with 0.952 for equal-budget random search and 0.940 for priority-greedy routing. The gain over random search is small but statistically detectable (Holm- adjusted p = 0.0083). The candidate does not improve congestion relative to MILP-C, and the effect of failure-aware training is inconclusive at the 0.05 level (p = 0.051). Eight discovery runs on the official national topology and replay on 12 unseen metro-regional topologies show no stable intent-specific specialization. These results support the trust-boundary and numerical-evolution claims but do not establish a benefit from LLM generation.

0.48

NetInspector: Measuring and Improving LLM Capabilities for Reliable Intent-Based Networking Policy Generation

cs.CR cs.NI ·2026-09-17 ·Yuxuan Zhang, Hongxin Hu, Guofei Gu
benchmark similarity 0.58
Abstract

Modern networks are large in scale and heterogeneous in configuration, making manual policy management increasingly impractical. Intent-Based Networking (IBN) addresses this by automating the translation of high-level operator goals into low-level network configurations. Yet existing IBN systems rely on static heuristics and fixed-feature classifiers that generalize poorly to distribution shifts such as new service definitions or evolving phrasing in operator requests. Large Language Models (LLMs), with strong reasoning and translation capabilities demonstrated across many domains, are a natural candidate for IBN policy generation. However, it is unclear whether LLMs can be reliably applied to this task, nor whether their use mitigates or worsens the underlying security risk. In this work, we show that while fine-tuned LLMs excel at intent translation, they exhibit false negative rates when checking whether a proposed intent violates an existing security policy. The root cause is not a lack of logical reasoning capability, but LLMs lack of persistent grounding in network topology and group hierarchy. Motivated by this finding, we introduce NetInspector, a three-layer agentic framework that enforces a verify-then-act protocol, decoupling information retrieval from reasoning so that the LLM focuses on symbolic reasoning while every policy decision is grounded in verifiable network facts retrieved from a live Environment Layer before approval. On NetInspector-Bench, a 2,224-sample synthetic benchmark spanning campus, enterprise, and WAN topologies, NetInspector reduces FNR by over 30\% relative to ungrounded baselines and remains robust under linguistic distribution shifts.

0.45

Axiom: a logic programming language to express and enforce network designs

cs.NI ·2026-10-27 ·Tibor Schneider, Laurent Vanbever
similarity 0.58
Abstract

Human errors remain the leading cause of network outages. To prevent them researchers propose general verifiers that analyze the network behavior. Yet, their (symbolic) models struggle to accurately reflect network devices, they fail to scale to large networks, and report misbehavior, not what configuration line caused it. We attribute all three challenges to specifications that describe the intended behavior but not how operators intend to achieve it---the network's design: which protocols to run and how to combine them. We present Axiom, a language for expressing network designs, and a checker that finds every violation and the responsible commands. Operators state their design decision as rules over configuration commands. Axiom then compiles them into a single Datalog program that checks the entire network directly on its configurations, not on the state they produce. Our prototype checks a 754-router network within seconds. Ultimately, a design enforced on every keystroke could give network operators the integrated development environment that programmers have long enjoyed.

0.20

Can LLMs help find Ambiguities in Protocol Specifications?

cs.NI ·2026-09-30 ·Ziyue Dang, Sixu Tan, Atharva Nevasekar, Zhaowei Tan, George Varghese +1 more
similarity 0.56
Abstract

Internet protocol specifications written in RFCs are subject to ambiguities and multiple interpretations that can cause interoperability failure. While these have presumably cleared up after years of experience, such ambiguities can bedevil the adoption of newer protocols like 5G. The 5G specifications pair a formal message syntax (ASN.1) with message-handling procedures written in natural language. This creates semantic underspecification: a syntactically valid message can reach a state whose procedures never say how to handle it, so standard-compliant implementations diverge. We frame this as a gap or a fork in a partially specified communicating state machine, and present SpecLens, which puts that view in front of a language model as a scaffold. Stronger models do not remove the need for it: they broaden the search without disciplining it, and fewer than half their findings survive inspection. Across 36 procedures from six 3GPP and O-RAN protocols, experts accept 185 of 197 SpecLens findings, and 60 drive observable divergence between the OpenAirInterface and srsRAN implementations under differential test. While we use 5G as a canonical example of a newer protocol, we also show ambiguity results for the more mature DNS protocol.

0.17