CS Paper Radar

When Talk Isn't Code: Comparing LLM Agents That Simulate and Develop Software

Xiang Li, Aichun Huang, Mingming Zhang, Zheli Liu

cs.SE cs.CR ·submitted 2026-10-03 16:29 UTC · v1

Abstract

LLMs are used both to simulate network protocol implementations, as honeypots do, and to write them. Prior work evaluates the two uses separately, from a model's answers or conversations in one case and from its generated code in the other. We observe that a knowledge probe (asking the model which security checks an implementation needs) and a conversation can credit security checks that the generated program lacks, but no study has compared them with the code the same model writes. To fill this gap, we present the first such comparison between simulation mode (S-mode), where the model plays the implementation in a conversation, and development mode (D-mode), where the model writes the program and the program is attacked, with a knowledge probe as a baseline. We design and implement a harness that judges all three with one attack suite and one oracle, and evaluate 15 LLMs on four protocol implementations. We find that with the full security specification, the median model's attack success is at most 4\% on reassembly, HTTP, and firewall implementations in both modes, but 13\% in S-mode and 15\% in D-mode on DNS. In D-mode, adding the missing DNS requirement to the specification cuts that attack from 100\% to 34\% and changes only that check, whereas deleting a stated rule weakens the program on other checks too. Asking the model beforehand does not predict which checks a program has, since 13 of the 14 models that write a DNS resolver name the check, yet all 42 programs lack it. S-mode is a useful first filter, flagging 82\% of the real DNS vulnerabilities and leaving 96\% of the safe cases unflagged. It errs in both directions, over-reporting security checks that D-mode does not implement and under-reporting checks that D-mode does. S-mode can screen but not replace D-mode testing, and security requirements should be written down even when a model can recite them.

Comment
21 pages, 6 figures

Closest papers

Nearest neighbours by meaning across the whole index.

ReconPotBench: Evaluating LLM Honeypots in Multi-Round Linux Reconnaissance

cs.CR cs.CL ·2026-11-16 ·Qianli Wang, Junchao Wu, Binkai Ou, Lidia S. Chao, Derek F. Wong
publishedbenchmarkcode/data similarity 0.73
Abstract · EACL 2027 Finding

Large language models (LLMs) are increasingly used to build interactive honeypots, yet existing evaluations mainly assess isolated command outputs and cannot determine whether a system remains believable when later commands depend on earlier results. We present ReconPotBench, a multi-round benchmark for Linux reconnaissance honeypots. It contains 1{,}032 three-round reconnaissance interactions across eight reconnaissance categories, with reference outputs collected from a real Linux environment. Under a unified testing protocol, we compare Cowrie, open-source LLMs, and closed-source LLMs along three dimensions: round-level authenticity, cross-round contextual consistency, and hacker-oriented reconnaissance value. Results show that multi-round evaluation exposes failures hidden by single-round testing, as authenticity declines with interaction depth and both open-source and closed-source LLMs struggle to maintain coherent state across rounds. LLM-based honeypots nevertheless outperform Cowrie in authenticity and reconnaissance value, while Cowrie remains more consistent because of its rigid templated behavior. Category-level analysis further shows that tasks requiring systems to preserve network relationships across rounds are harder than tasks centered on explicit entities such as ports. ReconPotBench enables reproducible evaluation of LLM honeypots across authenticity, entity consistency, and hacker-perspective utility in dependent multi-round interactions. The dataset is available at: https://github.com/qianli-wang/ReconPotBench

0.39

ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?

cs.CR cs.AI ·2026-10-13 ·Fasheng Miao, Zhaoyu Hu, Xiang Li, Changqing An, Jiutian Zeng +3 more
benchmark similarity 0.66
Abstract · 18 pages, 5 figures

Large language model (LLM) agents are increasingly used for vulnerability analysis, yet existing benchmarks provide limited coverage of exploitation that depends on interactions among network protocol components. Such vulnerabilities arise from inconsistencies in message interpretation, timing, state, or trust assumptions across participants. We introduce ProtoArena, a black-box, multi-component, dynamic benchmark with 80 cross-component vulnerability-exploitation environments spanning 11 protocol and application domains and 7 attack categories. Tasks are primarily informed by published protocol-attack mechanisms. Agents interact with running systems through scoped interfaces to achieve security-effect objectives without target source code. Three information levels assess guided reproduction, exploitation with mechanism and topology context, and effect-directed black-box exploration. A hybrid verifier combines rule-based checks with LLM review of independently collected traffic and logs. We evaluate five LLMs. Claude Opus 4.8 achieves the highest observed success rates: 48.8% under L1 guidance and 12.5% under effect-directed black-box L3. In a separate DNSBomb optimization experiment, it achieves a peak downstream-response to average query-input byte-rate ratio exceeding 171,000 times. The results show that agents can achieve verifiable exploitation effects in some cross-component environments, but success with limited black-box information remains low and varies across domains and attack categories. All experiments use controlled, explicitly authorized targets; they do not test production systems or disclose previously unknown vulnerabilities.

0.20

InferD: A Honeypot Framework for AI Inference Infrastructure

cs.CR cs.AI ·2026-10-29 ·Hicham Hammouchi, Gabriele Lenzini
published similarity 0.66
Abstract · AISec Workshop '26, ACM CCS 2026, The Hague, Netherlands

Large language model serving stacks, commercial inference APIs, vector databases, and agent orchestration gateways are now ex- posed on the public internet and indexed by commodity scanners within hours of deployment. The threat characteristics of this sur- face are only beginning to be measured, and the honeypot frame- works that could measure it were designed for conventional net- work services rather than for AI infrastructure. In fact, emulating an AI service is a distinct deception problem: the observable behavior of the service, including model inventories, token formats, tool-call protocols, and authorization flows, must be reproduced faithfully, while the underlying computation, model inference and tool ex- ecution, must never occur. We present InferD, a low-interaction honeypot framework organized around a single principle we call deception without inference: every request an adversary can observe is answered from static response pools, and nothing an adversary submits is ever executed. InferD decomposes the emulation into four deception planes, transport, service protocol, vulnerability surface, and agentic session, adds a provenance plane that links a planted credential to evidence of downstream model-mediated processing.

0.30

Adaptive LLM-Driven Honeypots vs. LLM-Driven Attackers: Does Narrative Coherence Improve Deception?

cs.CR cs.AI ·2026-11-17 ·Pedro Beltrán-López, Raul Gonzalez Jimenez, Dario Maddaloni, Manuel Gil Pérez, Pantaleone Nespoli
similarity 0.65
Abstract

Large Language Models (LLMs) enable honeypots to generate responses and deceptive artefacts on demand, yet existing approaches evaluate generated content one reply at a time and leave open whether artefacts that jointly describe a coherent event retain an attacker better than unrelated ones. This question becomes pressing as autonomous LLM-driven offensive agents emerge as realistic adversaries. This paper proposes narrative deception, a reactive strategy in which the defender maps each observed attacker command to a MITRE ATT&CK technique, selects a compatible scenario from a curated catalogue based on MITRE Engage, and deploys mutually referential artefacts (files, processes, services, and logs) on an emulated Ubuntu host while the intrusion is under way. An LLM-powered defender and an autonomous LLM attacker interact through a real SSH session, and the attacker's termination policy distinguishes abandonment due to suspected deception from abandonment due to loss of interest. A campaign of 1,150 sessions compares four conditions of increasing sophistication (Static, Same-TTP, Multi-TTP, and Chain) across four open-weight attacker models, four defender models, and three attacker skill levels. Engagement increases significantly at every step. With the same artefact budget per reaction, coherent scenarios raise the median session length of advanced attackers from 40 to 165 commands and reduce their abandonment from 86% to 52%. The effect grows with attacker skill, holds across all sixteen model pairs, and comes at comparable inference cost. Nearly all sessions ended through loss of interest rather than detection, which suggests that deception should sustain meaningful findings, not only emulator fidelity.

0.24

LLM-Based Penetration Testing in the Presence of Honeypots

cs.CR ·2026-09-08 ·Xinhong Xie, Piyush Nagasubramaniam, Neeraj Karamchandani, Sencun Zhu
published similarity 0.65
Abstract · 13 pages, 2 figures, 8 tables

Large language model (LLM) agents are increasingly employed for offensive cybersecurity tasks such as automated vulnerability discovery, reconnaissance, and penetration testing. This new capability also threatens one of the defender's most valuable tools: deception. Traditional honeypots rely on realism and obscurity to lure human or script-driven attackers into revealing tactics, techniques, and procedures (TTPs), but LLM-driven attackers can reason about heterogeneous artifacts and use the honeypot suspicion to guide target-selection decisions. We present a systematic study of honeypot-aware budget allocation for LLM attack agents. We formalize the attacker's problem as a budgeted decision process: an agent interacts with potential targets, consuming LLM execution budget during reconnaissance and exploitation, and must decide whether to (continue exploitation) or (skip) when honeypot suspicion arises. Our findings show that with the proposed detector-guided policy, LLM agent attackers can effectively allocate budget to compromise hosts in a host pool, highlighting the importance of dynamically allocating budget in a controlled mixed-host testbed. While defenses are beyond our present scope, we discuss implications for future adversarially resilient and adaptive honeypot design.

0.57

HoneyRoute: Honeypot-Model Routing for Adversarial LLM Serving

cs.CR cs.CLcs.LG ·2026-09-08 ·Han Jin
similarity 0.65
Abstract · Preprint. 14 pages, 4 figures, 1 table, 25 references

We introduce HoneyRoute, an inference-serving layer that detects whether an incoming request is malicious and, if so, routes it to a dedicated honeypot model, shielding production while the adversary's interaction is continuously harvested for intelligence. Existing defenses embed traps inside model memory or rebuild deception at the protocol layer, leaving the serving tier unprotected and feeding nothing back into detection. HoneyRoute couples (i) a streaming router (a frozen 0.8B-embedding backbone with per-domain MLP heads), (ii) a dual-implementation honeypot (a rule/prompt-engineered code honeypot or a dedicated same-family replica), and (iii) an analysis loop that converts trapped interactions into attacker fingerprints for router retraining. On a production trace plus a seven-domain attack corpus, the router reaches F1=.911 at 38 ms median added latency, matching 96% of a two-tier guard-LLM cascade's F1 at 1/385 of its latency with 0% evasion under 13 adversarial transformations; diverting the malicious share cuts production-model token consumption under concurrent flooding with real GCG-suffix payloads by 97.8%; the trained replica agrees with the production model on 92.9% of benign holdout requests, while naive unconditional bait injection collapses to 7.6% and selective camouflaged injection recovers to 88.9%, mapping the recoverable fidelity-traceability frontier; and a loop-trained correction head cuts misrouting of legitimate security research 9x while raising detection F1 to .933.

0.21

COPEX: Benchmarking LLM Robustness to Adversarial Context Across Model Context Protocol (MCP) Layers

cs.CR cs.AI ·2026-10-03 ·Nahom Birhan, Mehrdad Rostamzadeh, Sidhant Narula, Mahmoud Nazzal, Mohammad Ghasemigol +1 more
publishedbenchmarkcode/data similarity 0.65
Abstract · Accepted as a poster at the NeurIPS 2026 Workshop on Agents in the Wild

Large language models increasingly mediate tool use in Model Context Protocol (MCP) systems, where adversarial influence may enter through user instructions, tool schemas, tool outputs, or protocol messages. Existing benchmarks often evaluate deployed agents, conflating model susceptibility with guardrails, orchestration, and general task capability. We introduce COPEX (COntext Provider EXploitation), a controlled benchmark that isolates the model as an MCP client by fixing the surrounding agent stack and varying only the tool-selecting model. COPEX covers 25 attack types instantiated as 125 scenarios across four entry surfaces: model/agent, client, server/tool, and transport. Across nine models and 3,375 trials, the mean attack success rate is 64.4%, with surface-level means ranging from 58.3% to 71.4%. Some client- and transport-level attacks succeed partly outside the model's observation or control, separating system exposure from model susceptibility. Combined input and context scanning reduces mean attack success by 49.6% on an eight-attack defense subset relative to the undefended setting. The benchmark is available at https://github.com/inspire-center/copex.

0.45

Log-To-Leak: Prompt Injection Attacks on Tool-Using LLM Agents via the Model Context Protocol

cs.CR cs.AI ·2026-10-26 ·Yuepeng Hu, Chongyu Fan, Sirat Samyoun, Jian Du
publishedhighly cited similarity 0.64
Abstract

LLM agents increasingly access external tools through the Model Context Protocol (MCP), but the natural-language metadata used to describe those tools creates a prompt-injection surface. Within this established tool-poisoning surface, we study Log-To-Leak, an additive post-hoc logging attack that induces a separate attacker-controlled call after the final legitimate call. Its payload targets the user query, MCP server name, server response, and agent response. We operationalize the injected directive through Trigger, Tool Binding, Justification, and Pressure, and separately measure call success, payload fidelity, and preservation of legitimate behavior. We evaluate five MCP servers and four LLM agents using five metrics spanning effectiveness, utility, and efficiency. Across 18 model-server pairs with valid tool execution, Log-To-Leak achieves an attack success rate (ASR) of 62.64-100% and conditional logging accuracy (LA) of 76.55-95.64%; 16 pairs have absolute target-task completion changes of at most 6.63 percentage points. Three evaluated defenses leave residual ASR above 55% on GitHub MCP with GPT-4o, while an instruction detector flags 85.53% of benign descriptions. These results show that successful task completion can coexist with unauthorized post-hoc disclosure and motivate host-controlled authorization for secondary tool calls.

0.51