CS Paper Radar

REFINE: A Resilient Evolution Framework for Intelligent Enterprise Alert Triage in Security Operations Centers

Huimin Chen, Quan Long, Yanhao Wang

cs.CR cs.AIcs.LG ·submitted 2026-09-26 12:01 UTC · v1

Abstract

Security Operations Centers (SOCs) process large volumes of alerts daily. Alert triage prioritizes high-risk threats while reducing manual review of benign alerts. LLM agents can reason over logs and threat intelligence, but struggle to keep aligned with organization-specific, rapidly evolving SOC operational standards. We introduce REFINE, an LLM-agent framework for enterprise alert triage. REFINE encodes analyst expertise as structured skills and continuously adapts using analyst disposition feedback. It enforces recall = 1.0 as a hard constraint during evolution to maximize auto-closure of false positives, and identifies judgment blind spots by combining alert distributions with model error boundaries. Evaluated on four real industrial SOC scenarios across four MITRE ATT&CK phases with temporal split: REFINE achieves recall=1.0 on all evolution sets. On future test windows, it retains recall=1.0 in three scenarios; the degraded case reaches 0.807 recall, still outperforming self-evolution baselines (0.49-0.58).

Closest papers

Nearest neighbours by meaning across the whole index.

RAVEN: A Self-RAG Based Agent for Threat Classification and Response

cs.CR ·2026-11-19 ·Pironti Francesco Aurelio, Arena Luigi, Blefari Francesco, Lupinacci Matteo, Romeo Francesco +1 more
similarity 0.74
Abstract

Security event analysis in large enterprises can be overwhelming, as each alert typically requires manual in- spection by human operators. This process is time-consuming and prone to inconsistencies, especially in high-volume Secu- rity Operations Center (SOC) environments. In this work- in-progress paper, we present RAVEN: Retrieval-Augmented Vulnerability & Event aNalysis, an agentic framework designed to support SOC analysts by automating the initial stages of alert investigation. The proposed system leverages a Self- Reflective Retrieval-Augmented Generation-based approach to provide an initial classification of security events and enables interactive refinement through analyst-in-the-loop queries. Un- like traditional approaches that rely on generic mitigation strategies, RAVEN generates context-aware remediation sug- gestions by incorporating knowledge of the target enterprise infrastructure. We implement an early-stage prototype and report preliminary results on three balanced test sets of 400 curated web payloads each (SQLi, XSS, SSTI) and a qualitative case study of mitigation generation; an evaluation on IDS- generated alerts and a user study are ongoing and outlined as remaining work.

0.12

From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage

cs.CR cs.AIcs.MA ·2026-10-07 ·Saimon Amanuel Tsegai, Alex Kantchelian, Danfeng, Yao, Peng Gao
benchmarkhighly cited similarity 0.74
Abstract · Preprint

Security operations centers (SOCs) must triage large volumes of alerts, most of which are benign, while missed attacks can remain uninvestigated. Tool-using large language model (LLM) agents can retrieve evidence during triage, but it remains unclear how reasoning strategies determine what to gather and when an investigation is sufficient to close an alert. We study five representative approaches spanning single-pass tool use, iterative retrieval, sampled investigations, self-review, and explicit verification. To support this study, we build ALERT-BENCH, an interactive benchmark that replays enterprise telemetry through a live SIEM and requires each system to retrieve evidence. Across 1,247 alerts from a multi-stage attack scenario, every approach missed at least 40.4% of attack-related alerts. Trace analysis shows that attack alerts are more likely to be dismissed when searches return no records, same-context review has negative net correction, and dismissal receives no consistently stronger investigation than escalation. Based on these findings, we further design AIDA (Adversarial Investigation and Dialectical Analysis), a multi-agent framework that requires an explicit proposed decision before independent challenge and stronger evidentiary requirements before dismissal. AIDA preserves investigation history in an append-only Investigation Ledger and keeps the challenge in a separate reasoning context. A separate Judge adjudicates the proposed decision and challenge against evidence, resolving the alert or requesting another round when evidence is missing. On the same alerts, AIDA achieves an F1 score of 0.958, compared with 0.371-0.744 for the studied approaches, and reduces the false-negative rate from 40.4% to 3.1% while escalating 18.4% of alerts to analysts. These results show that structuring evidence retrieval and decision review can substantially improve agentic SOC triage.

0.45

Can Risk-Based Alerting Mitigate Cybersecurity Alert Fatigue?

cs.CR ·2026-09-02 ·Rafael Uetz, Philipp Bönninghausen, Louis Hackländer-Jansen, Martin Henze
similarity 0.74
Abstract · Submitted to USENIX Security '27

Security operations centers (SOCs) face large numbers of false alerts, making detection of cyberattacks difficult under typical resource constraints. Risk-based alerting (RBA) has been proposed as a means to reduce false alerts and has reportedly succeeded in doing so in various enterprise deployments. However, RBA has not been comprehensively evaluated until now, leaving implementation mostly guesswork based on anecdotal evidence. In this paper, we present the first systematic evaluation of RBA. To this end, we reformulate it as a continuous alert prioritization problem rather than a binary decision problem (i.e., whether an alerting threshold is exceeded), allowing us to evaluate performance across all possible thresholds and thus model SOCs of varying sizes and alert volumes. We distill five fundamental risk hypotheses, formalize them as independently parametrizable modules, and implement them in our novel experimentation suite CATS. We thoroughly assess the hypotheses across eight diverse alert datasets, six of which we created or extended to make such an evaluation possible. Our results show that certain combinations of hypotheses achieve a remarkable alert prioritization performance (AUROC $μ=0.92$, $σ=0.09$ across the eight datasets), outperforming a straightforward prioritization by alert severity level (AUROC $μ=0.72$, $σ=0.21$). We conclude that RBA can substantially reduce the number of false alerts that analysts have to review and thus has the potential to mitigate cybersecurity alert fatigue. In addition, it serves as a strong baseline for more complex, resource-intensive alert triage approaches (e.g., based on large language models).

0.17

Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

cs.AI cs.CLcs.LGcs.SE ·2026-11-10 ·Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary +1 more
top venue similarity 0.67
Abstract · Accepted to KDD 2026, Jeju, Korea. 9 pages, 3 figures

Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.

0.35

Constraint-Aware Evaluation of Reinforcement Learning and Unsupervised Anomaly Detection for Security Operations Alert Triage

cs.CR ·2026-11-03 ·Radityo Ghifiardi, Rifny Rachman
similarity 0.67
Abstract

Security Operations Centers (SOCs) must identify real threats from large alert streams, but limited analyst capacity means that unnecessary escalations can delay more serious investigations. Existing studies often frame SOC alert triage as classification or anomaly detection, but both views are incomplete for production SOCs, where the central problem is selecting alerts worth escalating under limited capacity, asymmetric error costs, and workload constraints. This paper addresses that gap by formalizing alert triage as a constrained decision problem and evaluating three approaches: an offline reinforcement learning policy trained on alert embeddings, an Isolation Forest model built on operational features, and an autoencoder trained on historical benign alerts. These methods represent different triage assumptions: learning actions from reward, isolating unusual operational patterns, and detecting deviations from benign behavior. We evaluate all methods on 74,909 analyst-labeled production alerts under a strict temporal split and benign-escalation constraint. Because the test set contains 503 unique feature patterns, we primarily rank clusters of similar alerts. At the main operating point, all three methods satisfy the constraint, with the reinforcement learning policy achieving the lowest benign-escalation fraction at 0.000, compared with 0.007 for the autoencoder and 0.041 for Isolation Forest. The methods also prioritize largely different alert clusters, showing complementary rather than redundant coverage. These results suggest that production SOC triage can benefit from constraint-aware evaluation and combined or adaptive deployment of different model families.

0.39

Integrating Perception, Comprehension, and Projection for Alert Validation: A Unified Cyber Situational Awareness Framework

cs.CR ·2026-10-29 ·Navodika Karunasingha, Mohan Baruwal Chhetri, Surya Nepal, Cecile Paris, Salil S. Kanhere
similarity 0.64
Abstract · 17 pages

Security Operation Centres (SOCs) must validate large volumes of alerts with limited analyst resources, yet alerts often lack sufficient context to determine whether suspicious activity is attack related. While provenance-based intrusion detection research has progressed from perception-level signals to comprehension- and projection-level cyber situational awareness (CSA), it remains unclear whether richer CSA consistently improves alert validation. This paper presents the first systematic evaluation of perception-, comprehension-,and projection-level CSA within a unified alert validation framework. Using Captain and Kairos, we generate multi-level CSA and formulate alert validation as a positive-unlabelled multi-instance learning (PU-MIL) task over CSA-derived behavioural hypotheses. Experimental evaluation on the DARPA TC dataset shows that comprehension-level CSA substantially improves validation by incorporating causal attack context. Relative to the Queue and Max. CSG baselines, PU-MIL achieves up to 28.6 and 75 percentage-point gains in Recall@5, and 24.6% and 86.8% reductions in mean positive rank on Cadets and Theia, respectively. Projection-level CSA can yield additional benefits, reducing the mean positive rank on Theia from 22.0 to 8.5 at a 40-minute projection horizon, although its effectiveness varies across projection horizons. Overall, richer CSA does not necessarily lead to better alert validation. Rather, its value depends on when and how it is introduced, motivating adaptive CSA progression that selectively enriches alerts according to validation requirements.

0.25

EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents

cs.AI ·2026-10-14 ·Qianhong Guo, Wei Xie, Xiaofang Cai, Shuoyoucheng Ma, Chengxin Hu +1 more
benchmark similarity 0.64
Abstract

Static benchmarks for large language model (LLM) agents face saturation and data contamination risks, motivating continual updates to evaluation tasks. However, aggregate scores and success rates offer limited insight into specific problem-solving bottlenecks and provide limited guidance for subsequent task design. We propose EvoProbe, an evaluation-driven benchmark evolution framework that integrates task generation, diagnostic evaluation, and feedback-driven regeneration into a closed loop. It combines verifiable subtask outcome scoring with evidence-supported multi-judge trajectory evaluation and distills phase-level bottlenecks, success patterns, and failure patterns into diagnostic feedback for generating new executable tasks. Across two rounds of cybersecurity experiments, evaluations of eight models on 66 tasks reveal phase-level performance differences and common failure patterns beyond aggregate scores. In a cross-round comparison with the Examiner model set and solver configuration held fixed, all eight models achieve lower Outcome scores on the updated tasks, with relative decreases of 11.8\% in mean Outcome and 40.0\% in mean Pass\textasciicircum{}3. These results support using fine-grained diagnostics to guide iterative task updates and probe agents' current capabilities and limitations.

0.14

HESP: Separating What to Probe from When to Stop in Local LLM Alert-Triage Agents

cs.CR cs.AI ·2026-09-27 ·Zhuowen Liu, Zhixuan Wang
code/data similarity 0.63
Abstract · 15 pages, 5 figures. Code and data: https://github.com/lzwhehe/HESP

Security operations centers receive far more alerts than analysts can investigate, and organizations that cannot send their telemetry to hosted models must automate triage with small open-weight LLMs on their own hardware. Current LLM agents leave the investigation procedure to the model, and small local models fail at it: they probe without converging, never commit to a verdict, or dismiss real attacks. In this paper, we present HESP, a controller that holds the investigation procedure outside the model. HESP keeps a ledger of competing explanations, selects read-only probes by expected information gain per cost, accepts only verdicts backed by current evidence, can end an investigation itself, and journals every prediction before its observation. We evaluated HESP in four pre-registered studies with five open-weight models from two families (7B to 72B), totalling 7,272 audited episodes in a controlled triage environment. With likelihood tables counted from LLM-free runs, HESP lifts Qwen2.5-7B from 0.125 to 1.000 verified completion, matching oracle tables. The information-gain ranking adds +0.26 to +0.35 on every model that concludes, and a controller-side stop lifts Llama-3.1-8B, which never concludes on its own, from 0 to 0.917. What to probe and when to stop are therefore separate failures, and different small models exhibit different ones. Because HESP and its planner run entirely on local hardware, it suits environments where telemetry cannot leave the premises. We release all code, protocols, and episode journals at https://github.com/lzwhehe/HESP.

0.71