CS Paper Radar

On the Effectiveness of Kernel-Level Evidence for Agent Security

Spencer King, Zhilu Zhang, Mikhail Kuznetsov, Kay Liu, Baris Coskun, Wei Ding

cs.CR cs.AI ·submitted 2026-09-24 01:46 UTC · v1

Abstract

LLM agents are deployed into infrastructure that grants them broad host authority, yet existing agent-security benchmarks and defenses operate almost exclusively at the application telemetry layer: the served tool manifest, the user prompt, and the model's messages. Some threats, however, smuggle malicious instructions and actions past the application boundary, leaving them invisible to that layer. In this work, we bridge that gap by pairing application-level agent telemetry with kernel-level syscall traces to present the first paired-evidence characterization of kernel-level versus application-layer signal for agent security. To quantify the value of the enhanced telemetry, we introduce Agent Cross-Layer Evidence (ACE), a paired-session corpus of 4,047 sessions and 17 threat models spanning six delivery-vector families and 14 of the 25 OWASP LLM and agentic threat categories, organized into 12 attack mechanics with per-mechanic characterization of where the most discriminative evidence lies. Across four distinct detector families, we find that kernel evidence is discriminative on its own and that composing it with application-layer evidence generally outperforms either single-layer view, revealing complementary signals that single-layer analyses can miss. We further demonstrate generalization to unseen attack families and transfer to an alternate agent runtime. Together, these findings establish the value of cross-layer evidence for agent security.

Comment
53 pages, 2 figures, 28 tables, including appendices

Closest papers

Nearest neighbours by meaning across the whole index.

A Graph-Based Approach for Mapping Kernel-Level Telemetry to MITRE ATT&CK

cs.CR cs.AI ·2026-09-11 ·Matteo Lupinacci, Luigi Arena, Francesco Blefari, Angelo Furfaro
similarity 0.71
Abstract

Mapping observed system behavior to standardized frameworks such as MITRE ATT&CK is essential for threat-informed defense. Existing automated methods mostly depend on Cyber Threat Intelligence reports, which offer only retrospective accounts of attacks. Kernel-level telemetry, i.e., system calls, instead provides first-hand evidence of adversary behavior, but its volume and complexity have limited its use for automated mapping. Its content also makes it unsuitable for cloud-based analysis: command lines carry credentials, events carry user identifiers and network endpoints, and during an incident the telemetry reveals which defenses failed. In this work, we present Trace2ATT&CK, a framework that maps kernel-level telemetry to ATT&CK techniques entirely within the defender's infrastructure. Trace2ATT&CK collects events via eBPF, reconstructs the attacker's activity as a process-centric provenance graph, and condenses it into a compact execution graph. This graph is classified by locally deployed open-weight LLMs, using either pure prompting or retrieval-augmented generation (RAG) grounded in the ATT&CK knowledge base, producing ranked technique candidates with supporting rationales. We evaluate Trace2ATT&CK on 347 Linux Atomic Red Team tests with seven open-weight LLMs. The best local configuration reaches an HR@5 of 66.57\% at the technique level. RAG improves over base prompting in every model and setting, by up to $+15.44$ percentage points, and the execution graph substantially outperforms raw telemetry while never exceeding the LLM context window. Finally, a frontier cloud model used as an upper-bound reference quantifies the cost of local inference at 24--28 percentage points. These results show that ATT&CK mapping from kernel-level telemetry is feasible without disclosing sensitive evidence to third-party services.

0.23

Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges

cs.CR cs.AI ·2026-09-20 ·Heewon Baek, Alsharif Abuadbba, Kristen Moore, Hyoungshick Kim, Surya Nepal
similarity 0.70
Abstract · 22 pages

Agentic AI extends LLM security beyond generated content to persistent state, autonomous actions, tool use, and interactions with humans and other agents. Existing threat classifications often emphasize individual dimensions, obscuring connections among entry points, affected components, and security consequences. The known threat landscape also differs from the coverage demonstrated by empirical research. Through a structured review of 66 studies published from 2022 to 2026, we introduce T={S, B, P, A}, a cross-dimensional representation linking affected functional or system surfaces {S}, interaction or trust boundaries {B}, violated security properties {P}, and empirically examined architectures {A}. We analyze 22 artifact-backed red-teaming studies and 11 representative security benchmarks to characterize empirical coverage and evaluation maturity. Within the selected studies, evidence concentrates on prompt/reasoning, memory, and tool-mediated attacks, predominantly in single-agent settings. Persistent, Human--Agent, complex multi-agent, systemic, and long-horizon threats receive less coverage. These findings describe the selected corpus rather than establish gaps across all empirical research. Heterogeneous metrics, limited adaptive defense evaluation, architectural imbalance, and incomplete execution-state capture further constrain comparison and reproducibility. We derive 13 open research questions to guide more systematic, architecture-aware, and reproducible security evaluation of agentic AI.

0.19

Say Shibboleth: Fingerprinting the LLMs and Harnesses Behind Agentic Attacks

cs.CR cs.AI ·2026-11-09 ·Dario Pasquini, Reworr, Giuseppe Ateniese, Manish Bhatt, Sergej Epp
similarity 0.70
Abstract · v0.1

We study the fingerprinting of agentic cyberattacks. Our objective is to reliably infer information about the attacker's configuration, such as LLM version and harness, from telemetry. To achieve this, we introduce a technique we call shibboleth traps: modular honeypot services designed to elicit discriminative behaviors from agentic attackers. The traps make informative interactions a condition for obtaining resources the agent seeks, eliciting model declarations and characteristic writing without requiring a successful prompt injection. To achieve harness fingerprinting, we introduce general mechanisms to indirectly probe peculiar execution behaviors of harnesses. These probes reveal how a harness masks secrets in tool output, enforces command time limits, and schedules tool calls, using only the effects visible in subsequent requests. We show that combined declarative and behavioral inference distinguishes among 15 candidate models with an average 90% top-1 accuracy, with behavioral classifiers evaluated on held-out harnesses or task families. In a separate experiment across four harnesses, fixed probe rules identify the correct harness with 92% accuracy. When instructed to conceal their identity, agents still leave useful behavioral evidence even as their declarations become uninformative.

0.15

Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection

cs.CR cs.LG ·2026-09-26 ·Animesh Shaw
benchmarkposition similarity 0.68
Abstract · 7 pages, 4 figures, 3 tables

LLM agents that invoke privileged tools are vulnerable to indirect prompt injection (IPI), in which adversarial instructions embedded in retrieved data hijack the agent's actions. A growing body of work evaluates defenses against IPI, but the validity of that evaluation is rarely examined. We audit an IPI benchmark and its harness and identify four defect classes -- silent payload non-delivery, attack success scored by tool identity rather than arguments, false-rejection rate conflated with model incapacity, and the absence of an audit trail -- each of which yields a plausible, publishable, and incorrect number. We quantify the distortion by re-scoring identical execution traces under the defective and corrected definitions: on real agent behaviour, the tool-identity scorer reports a 21.7% attack-success rate where the true argument-level rate is 1.2%. In the sharpest case, an open model previously reported at 62.8% registers 0% under the corrected harness -- the prior figure largely an artifact of undelivered payloads and identity-level scoring. We release a harness whose construction makes each defect unrepresentable -- machine-checkable payload placement, argument-level attacker predicates, per-scenario environments, and mandatory trace persistence -- and use it to report three quantities the field does not: whether a compromised agent discloses the attack, the full security/utility operating curve of an LLM-judge defense, and tool-calling capability disentangled from defensive over-blocking. A corrected harness further overturns a reported "capability barrier": a model deemed incapable of tool use is in fact fully capable, its earlier result an artifact of environment mismatch. We argue that evaluation validity is a prerequisite for, not a footnote to, defense claims in agentic security, and provide an instrument that enforces it.

0.43

Measuring LLM Agent Defenses by Layer: GDA Measurement and an Envelope-Layer Attack-Defense Model

cs.CR ·2026-09-30 ·Yuelin Han
similarity 0.67
Abstract

Agent interaction protocols such as ACP and A2A let a remote peer deliver a task the receiving agent treats as legitimate, opening a channel for indirect prompt injection. Evaluating and defending such agents is difficult because the attack success rate (ASR) cannot distinguish an LLM recognition failure from an agent-layer block, and so cannot indicate which layer to strengthen. We contribute three artifacts. (1) GDA Measurement, a red-team method that captures the raw agent-LLM context at the gateway, preserves conversation and raw-capture data, and judges outcomes autonomously with an agent selected by an inter-annotator-agreement (IAA) study over six candidates; its four classes, A/B/C/D, replace ASR with the semantic refusal, semantic breach, interception and penetration rates - a layer-resolved view of defense. (2) A2A-TIBA, an implant-command-exfiltration principle on A2A that generates this method's validation corpus, designed to spread outcomes across the four classes rather than to maximize the success rate. (3) ELA-ITL, a three-layer isomorphic attack-defense model induced from that corpus, which identifies the envelope layer - the channel through which content enters an agent - as a defense surface; ablating the A2A, tool and memory channels shows that annotating untrusted content markedly improves recognition of malicious content.

0.43

LEGION: Benchmarking Language Agents on Sparse-Signal Investigation Under Adversarial Noise

cs.AI ·2026-11-09 ·Samuel Joseph Amouyal, Ofek Glick, Yaniv Menasherov, Eyal Fisher, Ohav Barbi
benchmark similarity 0.67
Abstract

Autonomous language-model agents are increasingly deployed on discovery-driven tasks where signal is sparse and most retrieved information is irrelevant. Current benchmarks fail to evaluate this regime due to dense signals, cooperative environments, and ground truth typically being only a few tool calls away. We introduce Legion, a benchmark framework coupling a stochastic enterprise-telemetry simulator (Legion-Sim) with a tiered library of 74 expertly-crafted adversarial scenarios (Legion-Bench) and an evaluation protocol that scores investigation efficiency alongside detection accuracy. Scenarios are injected into freshly simulated companies, allowing independent control over confounder density and event volume. We test five agent architectures on two simulated companies and find that while recall is high (0.82-0.88) precision lags behind (0.61--0.72). They fail to find all of the indicators of compromise of the attack or mistake genuine indicators with attack related ones, which is critical in the context of cybersecurity investigation. Failures are consistent across reasoning tiers and persist across confounder density and event volume. We release Legion, the scenario library, performance on common agent architectures, and pipeline for reproducibility and extension.

0.09

BlueSTAR: Tiered Agentic Architecture for Autonomous Cyber Defense

cs.CR ·2026-09-10 ·Simona Boboila, Xavier Cadet, Edward Koh, Daniel Balasubramanian, Dirk Van Bruggen +2 more
similarity 0.67
Abstract

Cyber attacks are increasingly automated, narrowing the time available for human analysts to detect, reason about, and respond to intrusions. Large language models (LLMs) offer a promising foundation for autonomous cyber defense because they can correlate heterogeneous evidence and reason about previously unseen threats. However, directly applying LLMs to operational security telemetry is impractical: raw logs arrive faster than current models can process them, individual events are often ambiguous, and unconstrained LLM actions can introduce significant operational risk. We present BlueSTAR, a tiered agentic architecture for autonomous cyber defense in enterprise IT/OT networks. BlueSTAR first transforms high-volume security telemetry into compact indicators of compromise. We further introduce a resilience metric that jointly captures attacker reach, impact on mission-critical assets, and disruption caused by defensive actions. We evaluate BlueSTAR on two live enterprise IT/OT cyber ranges using seven attack chains based on real-world intrusion techniques. Across attack chains, BlueSTAR retains the fast containment of deterministic response for known threats while successfully defending against attacks requiring contextual and cross-cycle reasoning, including credential theft, repeated compromise, concurrent attackers, and attacks against physical processes.

0.23

From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage

cs.CR cs.AIcs.MA ·2026-10-07 ·Saimon Amanuel Tsegai, Alex Kantchelian, Danfeng, Yao, Peng Gao
benchmarkhighly cited similarity 0.66
Abstract · Preprint

Security operations centers (SOCs) must triage large volumes of alerts, most of which are benign, while missed attacks can remain uninvestigated. Tool-using large language model (LLM) agents can retrieve evidence during triage, but it remains unclear how reasoning strategies determine what to gather and when an investigation is sufficient to close an alert. We study five representative approaches spanning single-pass tool use, iterative retrieval, sampled investigations, self-review, and explicit verification. To support this study, we build ALERT-BENCH, an interactive benchmark that replays enterprise telemetry through a live SIEM and requires each system to retrieve evidence. Across 1,247 alerts from a multi-stage attack scenario, every approach missed at least 40.4% of attack-related alerts. Trace analysis shows that attack alerts are more likely to be dismissed when searches return no records, same-context review has negative net correction, and dismissal receives no consistently stronger investigation than escalation. Based on these findings, we further design AIDA (Adversarial Investigation and Dialectical Analysis), a multi-agent framework that requires an explicit proposed decision before independent challenge and stronger evidentiary requirements before dismissal. AIDA preserves investigation history in an append-only Investigation Ledger and keeps the challenge in a separate reasoning context. A separate Judge adjudicates the proposed decision and challenge against evidence, resolving the alert or requesting another round when evidence is missing. On the same alerts, AIDA achieves an F1 score of 0.958, compared with 0.371-0.744 for the studied approaches, and reduces the false-negative rate from 40.4% to 3.1% while escalating 18.4% of alerts to analysts. These results show that structuring evidence retrieval and decision review can substantially improve agentic SOC triage.

0.45