CS Paper Radar

From Noise to Signal: Improving Security Log Anomaly Detection Using LLMs with Endpoint-Specific Logs

Christopher Henshaw, Gour Karmakar

cs.CR cs.LG ·submitted 2026-08-20 11:54 UTC · v1
published

Abstract

Existing approaches to anomalous behaviour log detection, such as Wazuh rely primarily on predefined detection rules, while statistical anomaly detection approaches such as OpenSearch identify deviations from previously observed behavioural patterns. Recent research has investigated LLMs for log anomaly detection because of their ability to interpret semantic and contextual information. However, LLM-based approaches can be affected by prompt construction, noisy log data, and reliance on generic datasets that may lack endpoint-specific authentication behaviours. To address these limitations, this study develops a standardised instruction-based LLM classification framework for detecting anomalous authentication behaviours, including borderline cases. A controlled cybersecurity testbed was developed to generate endpoint-specific authentication data, producing a curated dataset comprising normal, borderline, and anomalous behavioural scenarios. Three instruction-tuned LLMs, Meta Llama 3.1 8B Instruct, Qwen 2.5 7B Instruct, and GPT-OSS 20B, were evaluated against Wazuh rule-based detection and OpenSearch Anomaly Detection using a common ground-truth severity framework. Meta Llama 3.1 8B Instruct achieved the strongest overall end-to-end detection performance, with an accuracy of 89.3%, recall of 88.2%, F1-score of 91.8%, and false negative rate of 11.8%. In comparison, Wazuh achieved an accuracy of 52.0% and false negative rate of 68.6%, while OpenSearch achieved an accuracy of 49.3% and false negative rate of 74.5%. Meta Llama also detected 80% of the borderline anomalous scenarios, compared with 20% for Wazuh and 15% for OpenSearch. Qwen achieved lower overall detection performance than Meta Llama but recorded the lowest average inference latency and 100% structured-response validity. GPT-OSS demonstrated strong classification performance when valid responses were produced.

Comment
The paper contains 35 pages and 3 figures. The paper has not been submitted or published in any conference or journal. The authors have an aim to publish it in a journal

Closest papers

Nearest neighbours by meaning across the whole index.

HSMLog: Small Language Model-Assisted Hardware Security Module Log Anomaly Detection with Behavioral Analysis

cs.CR ·2026-08-30 ·Chia-Hsuan Wu, Dar-Hsin Dustin Wu, Rui Fang, Yi-Ting Lee, Chia-Chih Lin +1 more
published similarity 0.72
Abstract · Accepted to the ISSRE 2026 Industry Track; 6 pages, 2 figures, 3 tables

Hardware Security Module (HSM) logs capture security-critical behavior, but anomalies emerge from relationships across event sequences, keys, object states, sessions, and temporal patterns rather than isolated events. Existing methods separate detection from HSM-specific evidence validation and reporting. In this paper, we present HSMLog, a two-stage framework for HSM log anomaly detection with retrieval-grounded behavioral analysis. In Stage 1, a small language model (SLM) identifies candidate alerts from sliding windows of structured HSM events and performs policy-guided assessment using HSM-specific operational rules. In Stage 2, retrieved policies and historical suspicious-key records strictly predating the alert window, together with candidate-related log context, support conservative candidate review and incident analysis. Evaluated on real industrial HSM background logs augmented with anomaly scenarios co-defined with industrial partners, HSMLog achieves 98.97% precision, 96.00% recall, 98.66% anomalous-event coverage, and a 97.46% F1 score, demonstrating effective anomaly alerting and incident triage in the studied setting.

0.36

Towards Understanding LLM-Based Log Anomaly Detection: An Empirical Study of Performance, Efficiency, and Robustness

cs.LG cs.CR ·2026-09-25 ·Bin Li, Dongdong Wang, Siyang Lu
similarity 0.69
Abstract · 6 pages, 2 figures, 3 tables. Submitted to IEEE ICASSP 2027

Large language models (LLMs) have demonstrated promising performance in log anomaly detection, yet how their adaptation strategies, architectures, and deployment configurations affect detection effectiveness remains insufficiently understood. To investigate these factors, we conduct a systematic empirical analysis across three public log datasets, examining different adaptation strategies, model architectures, parameter scales, and quantization settings. Our results reveal substantial performance differences across adaptation strategies, while model scaling yields varying detection gains across datasets. We further observe that models with comparable detection accuracy can exhibit markedly different computational costs, and that low-bit quantization largely preserves detection performance in the evaluated configurations. Finally, we examine detection robustness under structural, semantic, and label noise at different perturbation levels. These findings provide empirical insights into the performance, efficiency, and robustness of LLM-based log anomaly detection, highlighting practical considerations beyond conventional accuracy-oriented evaluation.

0.10

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

cs.CR cs.AI ·2026-10-22 ·Yifan Wang, Dongliang Xu, Xiaoyong Li, Jing Li, Peter S. Pan +1 more
benchmarkcode/data similarity 0.67
Abstract · 17 pages, 9 figures. Submitted to IEEE Transactions on Dependable and Secure Computing. Code and data: https://github.com/SecLog-Open/SecLog

Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.

0.35

X-WAD: eXplainable Web Anomaly Detection

cs.CR ·2026-08-27 ·Matteo Bitussi, Roberto Doriguzzi-Corin
similarity 0.65
Abstract

The rapid growth of web-based services, particularly API-driven architectures, reflects an increasing reliance on distributed systems, exposing sensitive data to security risks and making the adoption of automated defensive mechanisms essential. In this context, where benign traffic predominates in real-world settings, modern defenses increasingly model normal behavior, relying on semi-supervised approaches trained on only normal data. However, ensuring the complete absence of anomalous instances in such training data is inherently difficult in practice, and mislabeled or contaminated attack samples can introduce backdoors into the learned defense, causing the model to silently misclassify certain attack patterns as normal behavior. This paper investigates the effectiveness of Transformer-based Language Models (TLMs) in the detection of anomalies in HTTP requests, focussing on providing detailed explanations for the detected anomalies. The study employs token-level logit-based surprisal mapping to provide both an anomaly score and a direct, detailed explanation via heatmap-like highlighting. The effectiveness of the proposed explainability approach is demonstrated by the discovery of labelling inconsistencies in a popular public dataset, revealing how anomalous contamination in the training data had induced backdoor-like failures in the detection models.

0.41

Fast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical Study

cs.SE ·2026-11-05 ·Jesse Nyyssölä, Mika Mäntylä, Yuqing Wang
hot direction similarity 0.64
Abstract

Anomaly detection on software logs ensures system reliability through failure identification, root cause analysis and fault localization. While supervised models show excellent performance, in a practical setting there rarely are labeled training data available. This calls for the use of unsupervised models. We study effects of data preprocessing, anomaly detection method, and the dataset. For preprocessing, we examine three options: The use of anomaly-free training data, using randomized as opposed to chronologically selected training data, and the use of a custom loader to extract log message from the log line. Regarding anomaly detection we use four models (KMeans, IsolationForest, RarityModel, Out-Of-Vocabulary Detector) and three representations (Words, trigrams, and parsed events). We evaluate these approaches on seven datasets: BGL, HDFS, Hadoop, Spirit, Thunderbird, Liberty, and LO2Token. We run each combination of the selected factors and measure performance with AUCROC and F1-score metrics. Then we utilize a generalized linear model to establish relationships between the factors. Furthermore, we address execution times. Our evidence reveals large variability across all datasets, preventing us from making recommendations that apply universally. For instance, log parsing enhances performance on some datasets but reduces it on others. The evidence suggests that using anomaly-free data for training unsupervised models can improve anomaly detection performance, though this requires extra effort to identify such data.Previous work has argued that shuffling data leads to unrealistic model performance (data leakage). Yet, our results show this does not hold for the unsupervised models we tested, while it remains true for supervised models in the same setup. For fast execution of the pipeline, use word representation and avoid IsolationForest model.

0.25

LLMs for Zero-Shot Threat Detection via Structured Risk Indicators

cs.CR cs.LGcs.NI ·2026-08-17 ·Abdullah Alghamdi, Siamak Layeghy, Marius Portmann
benchmark similarity 0.63
Abstract

We propose a two-stage large language model (LLM) framework for zero-shot detection of insider threats and advanced persistent threats (APTs) from heterogeneous security logs. The framework models user activity as chronological timelines and incorporates retrieval-augmented generation (RAG) to provide personalised behavioural context from each user's historical activity. Rather than performing end-to-end classification directly from raw logs, it first generates structured, interpretable sets of threat-specific risk indicators, which are then classified jointly across temporal sequences to capture attack patterns spanning multiple windows.The framework is evaluated on two benchmark datasets, CERT r5.2 for insider threat detection and PicoDomain for APT detection, using four combinations of two open-weight LLMs under both retrieval and non-retrieval settings. All configurations outperform the previous state-of-the-art LLM-based framework (GABM), with the best configuration improving the F1-score by 11.40 percentage points on CERT r5.2 and 31.50 percentage points on PicoDomain. Results further show that retrieval mainly benefits weaker LLMs by generating more discriminative risk indicators, whereas stronger models achieve comparable performance without retrieved context. The most effective assignment of LLMs to the two stages depends on the dataset. These findings show that the quality of the generated risk indicators is the main driver of zero-shot cyber threat detection performance.

0.29

The Surprising Effectiveness of LLMs in BGP Security: Mining An Unprecedented Amount of Incidents and Boosting Anomaly Detection

cs.NI ·2026-08-24 ·Libin Liu, Wenzhou Yang, Li Chen, Dan Li, Xiuting Xu
publishedbenchmarkcode/data similarity 0.61
Abstract · Accepted by IEEE ICNP 2026, 10 pages in main body, 20 pages in total

Border Gateway Protocol (BGP) security is critical to Internet infrastructure, yet progress in routing anomaly detection has been limited by the scarcity of publicly available incident datasets, which contain only 18 recorded cases. We observe that public operator mailing lists, e.g., NANOG and AusNOG, contain abundant yet largely untapped reports of real-world routing anomalies. To leverage this source, we develop an LLM-assisted extraction pipeline that identifies 244 candidate incidents from historical discussion threads. After expert validation, we curate a verified benchmark containing 232 confirmed routing anomaly events, making it 11.89X larger than existing dataset. Using this benchmark, we show that existing routing anomaly detection systems generalize poorly to diverse real-world incidents. At the same time, we find that some general-purpose LLMs without routing-specific adaptation can identify a subset of routing anomalies, but their performance varies across models and remains insufficient for reliable routing anomaly detection. Motivated by this observation, we design ROUTELLM, an LLM-based routing anomaly detector that integrates BGP-semantic-aware tokenization, routing-domain adaptation, and time-aware routing evidence retrieval. Experimental results show that ROUTELLM achieves 87.13% event-level accuracy and 94.65% message-level accuracy, outperforming the strongest baselines by 55.30% and 68.50%, respectively. We open-source the verified routing anomaly benchmark, fine-tuned model, and implementation code to support future research on BGP security.

0.48

LLM as Detector: An In-context Learning Approach for Tabular Anomaly Detection

cs.LG ·2026-08-19 ·Tu Anh Hoang Nguyen, Dang Nguyen, Thuc Duy Le, Trung Le, Sunil Gupta
published similarity 0.61
Abstract · Accepted at International Conference on Neural Information Processing (ICONIP) 2026

Anomaly detection in tabular data is challenging because abnormal samples often arise as violations of cross-feature dependencies rather than simple marginal deviations. Existing detectors rely on geometric or reconstruction signals, while prior LLM-based approaches mainly fine-tune LLMs with normal samples or generate synthetic anomalies. We propose LLM-Detector, a framework that utilizes the in-context learning capacity of LLMs for structured, prompt-conditioned scoring synthesis, enabling LLMs to derive anomaly detection logic from structured normal-state knowledge. Specifically, normal training data are converted into statistical summaries, causal dependencies, and distilled prototypes that are organized into a prompt for code generation. The resulting scoring engine evaluates statistical deviation, structural inconsistency, and density-based abnormality then computes an anomaly score for each test sample. We evaluate LLM-Detector on 24 tabular datasets, comparing against 15 SOTA baselines. Results show consistent improvements across both mixed-type and continuous-only settings. Moreover, this design eliminates the need for LLM fine-tuning or neural network training, reducing computational cost and enabling practical anomaly detection in real-world tabular systems.

0.20