CS Paper Radar

LoopHIL: Looping Hardware into LLM-assisted Agents for Wireless Embedded Systems

Dhairya Shah, Spanddhana Sara, Savitha Viswanadh Kandala, Ambuj Varshney

cs.NI cs.AIcs.ARcs.SE ·submitted 2026-10-16 20:54 UTC · v1

Abstract

LLM agents are widely used for software development and debugging, but their adoption in embedded systems, especially for wireless tasks, remains limited. This is because embedded development requires hardware-in-the-loop testing. Agents cannot directly observe the spectrum, probe pins, or measure power, so they cannot verify wireless link functionality; currently, human engineers perform this verification. We systematically evaluate whether LLM agents can establish wireless links when provided with hardware access. We introduce LoopHIL, a framework that lets agents control a multi-device RF bench, monitor it with a spectrum analyzer, logic analyzer, and energy meter, and validate links against a verification ladder, all within code-enforced safety limits. Across four increasingly complex tasks, from generating a carrier to backscatter decoding by a commercial receiver, and using six models from two vendors, hardware-in-the-loop increases task completion rates from 54% to 79% and from 17% to 83% on the most challenging task. It also halves task time and output tokens. With debugging tools, agents decrease tag power consumption by up to 96%. However, small models often report unverified claims, so independent hardware checks remain necessary.

Comment
8 pages, 8 figures, 4 tables. Under review

Closest papers

Nearest neighbours by meaning across the whole index.

EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware

cs.SE cs.AI ·2026-11-16 ·Minghui Zhao, Yiming Li, Sophie Zhu, Maxwell Khor, Lauren Ouyang +3 more
similarity 0.62
Abstract

Modern coding agents perform well on general software benchmarks because they can run code and unit tests in a closed loop. Embedded firmware development, however, breaks this loop: a program that compiles can still fail on real hardware due to pin polarity, bus protocol, timing, or networking bugs that an agent at a text terminal cannot observe. Existing embedded coding agents either skip hardware execution, drive GPIO-only simulators with multi-second delays, or test on physical boards in a static environment. In this paper, we present EmbedGym, an interactive virtual hardware environment that lets coding agents develop and debug embedded firmware without physical hardware. EmbedGym uses instruction-level microcontroller unit (MCU) emulators to execute the same binary that would be flashed onto a physical board, connects them to 41 peripheral models, an analog circuit simulator, and virtual WiFi/Bluetooth Low Energy (BLE) endpoints, and exposes an interface for agents to change sensor inputs, press buttons, and inspect hardware states. On top of it, we build EmbedGym-Bench, which consists of 598 tasks across 6 MCU-framework pairs graded by hidden timed test vectors, using a five-stage multi-agent workflow that synthesizes peripheral models and tasks from datasheets and driver libraries. We expose EmbedGym to a coding agent at three levels and evaluate eight models over more than 14K trajectories. Even when compilation succeeds 99.7% of the time, 19.9% of compiled firmwares fail at runtime (up to 73% on individual targets). In contrast, interactive hardware testing in EmbedGym increases pass rates across models by +4.9% to +14.7% (up to +31% on individual targets). Our results also show that an agent with EmbedGym outperforms prior embedded agent frameworks, and lightweight models with EmbedGym match frontier models without it at 30-70x lower cost.

0.26

5uGUE: Agentic Hardware-in-the-Loop Fuzzing for 5G Devices and beyond

cs.NI ·2026-10-21 ·Guoqiang Hua, Sakshi Udeshi, Matheus Eduardo Garbelini, Daisuke Mashima, Sudipta Chattopadhyay
similarity 0.61
Abstract

In this paper, we present 5uGUE, a large language model (LLM)-driven hardware-in-the-loop agent, which enables the wireless fuzzing of 5G-enabled devices (such as mobile phones). 5uGUE combines multi-step reasoning with tool-assisted interaction and integrates into the fuzzing pipeline by leveraging specialized tools for internet search, 5G protocol analysis, memory management, and test-case execution. The agent iteratively generates test cases, executes them on targets, collects runtime feedback, and adaptively refines the fuzzing strategy. We evaluate 5uGUE on five commercial off-the-shelf (COTS) 5G devices across six LLMs. The best performing runs for 5uGUE trigger 167 crashes in total (141 novel and 31 unique) within 60 interaction turns (49 to 133 minutes total runtime). In contrast, the best runs for the state-of-the-art baseline (5GHOUL) discovers only 13 crashes in total (10 novel and 6 unique) within 150 minutes. Overall, by integrating LLM-driven agents, 5uGUE demonstrates significantly better crash discovery performance (12.8x total, 5.2x unique improvement). Notably, our experiments identified 46 unique crashes, 44 of which were confirmed by the vendors but had been previously reported. More importantly, MediaTek confirmed two previously unknown, high-severity vulnerabilities, one of which was assigned CVE-2026-20525. Finally, to demonstrate that 5uGUE is agnostic to wireless protocols, we also extend it to Bluetooth Classic with only minimal code changes. In our evaluations, 5uGUE finds an average of 6.4 novel crashes and 1.6 unique crashes. In a nutshell, 5uGUE highlights that LLM agents are capable to operate effectively beyond software-in-the-loop settings, extending autonomous vulnerability discovery to the proprietary, hardware-bound systems that affect billions of end-user devices across multiple wireless protocols.

0.28

Agentic RF Intelligence: Multi-Timescale 6G Sensing and Reasoning with On-Device Foundation Models

eess.SP ·2026-10-02 ·Jaron Fontaine, Jelle De Moerloose, Xander Vanparys, Anton Lambrecht, Eli De Poorter +1 more
published similarity 0.61
Abstract · Accepted at the NeurIPS 2026 Workshop on On-Device Intelligence: Foundation Models under Real-World Constraints

Future 6G systems share a central challenge with Physical AI: combining sensing, reasoning, and action on network-edge infrastructure in rapidly changing physical environments under strict latency, compute, and energy constraints. Addressing this challenge requires multi-timescale intelligence, combining fast perception that tracks wireless phenomena within milliseconds with slower reasoning that directs sensing and adapts network policies and resources over seconds. In this paper, we present a vision for Agentic RF Intelligence based on a dual-loop architecture that decouples fast, locally autonomous wireless sensing (e.g., detecting spectrum dynamics, interference, and signal sources) from slower agentic reasoning and orchestration. We provide an on-device prototype in which the full stack runs on a single NVIDIA Jetson Thor connected to a B200 Mini USRP. The fast loop processes IQ samples using signal-processing tools and wireless physical-layer foundation models (WPFM), while a local Large Language Model (LLM) asynchronously interprets RF events, invokes tools and steers sensing. Our experiments confirm the separation in timescales, as WPFMs operate at millisecond latency, while LLM interactions take seconds to minutes. The fast loop remains operational during agentic reasoning, providing initial evidence for the feasibility of this decoupled architecture. We conclude by outlining extensions toward memory-driven self-improvement, world models, network control, and multi-agent operation.

0.30

Closed-loop evaluation of LLM agents for embedded software development

cs.SE cs.AIcs.ARcs.LG ·2026-10-08 ·Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté, Juan Trujillo
publishedbenchmarkcode/data similarity 0.60
Abstract · Published in Journal of Systems Architecture 179 (2026) 103937. 23 pages, 4 figures, 5 tables. Code and artifacts: https://github.com/jgcarr…

Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair software iteratively. Embedded firmware is a demanding target because correctness depends on closed-loop behavior under sensing, timing, and safety constraints, not only on static source quality. Yet embedded-agent evaluation remains limited and often emphasizes one-shot synthesis or offline correctness. We present a benchmark for closed-loop evaluation of embedded coding agents. Each task provides a plain-text engineering description, constrained workspace, and visible build-and-runtime surface. The agent must translate requirements into implementation and self-verification steps, then iterate until the required device behavior is achieved. The suite contains five embedded-control tasks and four feedback scenarios: one-shot generation, realistic self-verification, CI-style red/green feedback, and oracle-style detailed feedback. The implementation targets simulated ESP32 firmware for reproducibility. We evaluate seven GPT-family and Qwen-family configurations across five tasks and four scenarios, with three repetitions per condition for 420 runs. gpt-5.4 has the highest pass rate among evaluated configurations but does not saturate the benchmark; qwen3.5-27B is the strongest observed local model; and smaller local models degrade sharply in pass rate and search efficiency. These results suggest that capable local embedded coding agents are emerging.

0.69

DeepRadio: A Human-Agent Workspace for Physical Radio Development

cs.HC ·2026-11-06 ·Jiazhen Lei, Yuxin Sha, Sihan Wang, Wenhui Li, Jinyang Sun +2 more
similarity 0.59
Abstract

Agentic programming can automate software-defined radio (SDR) development, but decisions can propagate from software to hardware without sufficient user visibility or control. We present DeepRadio, a human-agent workspace embedded in GNU Radio Companion (GRC) for jointly constructing, verifying, and operating physical radio systems. DeepRadio uses a dynamic and state-driven workflow to coordinate a traceable radio specification, claim-oriented cross-layer verification, and mixed-initiative control. These mechanisms expose decisions, tie conclusions to available evidence, and return control to users before consequential hardware actions. Across seven atomic SDR tasks, DeepRadio completed nearly all runs while satisfying all prescribed software and radio-frequency constraints. In a 20-participant study, more participants achieved a verified physical deployment with DeepRadio than with Codex-assisted GRC. Participants reported stronger support for shared intent and understanding while retaining initiative and control. These results suggest that DeepRadio can make agent-assisted physical radio development more reliable, understandable, and controllable.

0.29

Independent Data-Integrity Verification for LLM-Driven Functional Coverage Closure in Hardware Design

cs.AR cs.AIcs.SE ·2026-10-15 ·Venkata Krishna Kumar Vedantam
similarity 0.59
Abstract

LLM agents can write directed tests that close functional coverage, but coverage only shows that a state was reached, not that the data on the way there was correct. This is a long-known point in verification and we do not claim it as new. We describe a small coverage-closure agent that applies it to an LLM loop: coverage is read from the report that the simulation itself exports (cocotb-coverage YAML) and never taken from the model's account, and data integrity is checked by scoreboards that are separate from the model. On three architecturally distinct RTL designs (an asynchronous FIFO, a UART transmitter and an AHB2APB bridge) the agent raised coverage from 91.7%, 88.9% and 67.4% to 100% within its 8-iteration budget in every logged run, using one design-agnostic agent. A reactive scoreboard monitor attached to the FIFO's base testbench also found a clock-domain-crossing bug in the author's own stimulus code. The monitor is not yet attached to the LLM-generated tests, so their data integrity is not independently checked; one manual check on a single FIFO test is reported as preliminary. We list the weaknesses of this evidence: three designs, no baseline, no ablation, and logged runs that come from several agent checkpoints.

0.15

EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities

cs.OS cs.AIcs.CLcs.SE ·2026-11-10 ·Zhengxiang Huang, Yujie Sun, Shengheng Chen, Zhaode Wang, Chaoyue Niu +4 more
similarity 0.58
Abstract

Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host--device execution paths. We present EngineAgent, an agentic system for automating on-device LLM model enablement. Its key insight is to separate reusable engine and hardware knowledge from model-specific implementation. Before enablement, EngineAgent constructs a hierarchy-aware engine index that captures engine abstraction layers and cross-layer dependencies, and pre-validates backend-specific build, execution, and debugging workflows using already supported models. For each new model, a builder uses this knowledge to coordinate cross-layer changes, while an auditor checks implementation completeness and drives iterative debugging through device-side validation. Across 12 tasks spanning three mobile LLMs and four engine--backend pairs on CPUs, GPUs, and NPUs, EngineAgent achieves 75% end-to-end success with either of two underlying LLMs, versus 33.3% for the strongest coding-agent baseline. Success requires compilation, on-device execution, and output-logit agreement with reference implementations. EngineAgent also reduces average per-task time and model-API cost relative to baselines using the same underlying LLMs.

0.21

AgentDV: Closed-Loop Agentic AI for Hardware Design Verification

cs.SE ·2026-08-27 ·Navya Goli, Junzhe Liu, Zhenge Jia, Umamaheswara Rao Tida
similarity 0.57
Abstract · The authors are withdrawing this preliminary manuscript due to errors in the methodology and experimental setup in Section 4 and 5. The work…

Register-transfer level (RTL) verification consumes a major part of modern system-on-chip (SoC) development effort. Yet, recent LLM-based verification-code generation often fails to produce runnable, design-consistent, and coverage-producing testbenches. We present AgentDV, a closed-loop agentic AI framework for automated RTL verification environment generation. AgentDV transforms single-shot LLM testbench generation into a tool-grounded verification pipeline by combining LLM-guided analysis, testbench construction, simulation, coverage measurement, and iterative refinement. The framework introduces three key ideas: 1) runnability filtering to reject invalid generated environments, 2) CSR-grounded checking to reduce hallucinated signals and incorrect expected behavior, and 3) coverage-guided iteration to regenerate tests based on measured verification gaps. We evaluate AgentDV using three LLMs on challenge DUTs and public OpenTitan peripheral and security IP blocks. From our analysis, we observed that direct single-shot prompting fails to produce a valid coverage-producing environment on benchmarks. AgentDV achieves 100% pass rate on four DUTs and an average of 80.9% pass rate on all DUTs using Claude Sonnet 4.6. Similarly, an average of 58.7% and 60.6% pass rate is achieved for Llama and Qwen models, respectively. In addition, an average of 74.5%, 69.1%, and 64.9% of line coverage and 88.4%, 82.3%, and 76.7% of branch coverage for the benchmarks under consideration for Claude Sonnet 4.6, Llama, and Qwen models, respectively.

0.08