CS Paper Radar

RankShift: In-Database Detection and Explanation of Categorical Shifts

Omair Shafi Ahmed

cs.LG ·submitted 2026-08-28 22:43 UTC · v1
unusual

Abstract

A login service can receive its usual number of failed sign-ins while one source grows from 2% to 30% of them. The same pattern appears in system logs when a rare event template becomes common while the message rate stays stable. These events change which categories are active without changing how many events occur. RankShift detects such changes inside the analytical database that stores the data. It compares each window's category shares with a benign reference using a Pearson score whose terms identify the categories responsible for the change. The same query returns the score, calibrated alert, and largest increasing contributions. We evaluate RankShift on HDFS, BGL, and Thunderbird. It matches the count-vector autoencoder within 0.001 AUROC on HDFS (0.999 versus 1.000) and leads on Thunderbird (0.983 versus 0.949). In a controlled fixed-volume experiment, RankShift detects rare-category shifts that are invisible to event-count monitoring, reaching 0.787 AUROC compared with 0.771 for the autoencoder. Across all three corpora, observed false-alarm rates track the requested operating levels. RankShift requires no model training or inference service, and the autoencoders deployed state is 137x larger.

Closest papers

Nearest neighbours by meaning across the whole index.

Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL

cs.DB cs.AIcs.CEcs.IR+1 ·2026-10-05 ·Hang Xiao, Janet Sung, Zhaoyi Li, Gangzhen Qian, Chuhong Xu
published similarity 0.57
Abstract · Accepted at DASC 2026. 8 pages, 1 figure, 8 tables. Reproduction support artifact: https://doi.org/10.5281/zenodo.23151002

Protocol choices can change the conclusions drawn from log anomaly detection benchmarks even when detector settings are fixed. We present a joint empirical study of split construction, representation visibility, and component costs using six fixed count, sequence, and semantic configurations on Hadoop Distributed File System (HDFS) and Blue Gene/L (BGL) logs. Random splits place several configurations near the average-precision ceiling, whereas group-disjoint HDFS and chronological BGL evaluation produce lower scores and different observed orderings. At a fixed BGL cutoff, parser choice spans 0.124 in semantic XGBoost mean average precision while preserving its lead over count XGBoost; the earliest rolling period reverses that ordering. A two-factor cross-system ablation contrasts source-only representations with offline transductive access to unlabeled target templates through the representation corpus and inverse document frequency: HDFS-to-BGL mean average precision moves from 0.191 with source-only access to 0.325 with union-corpus, target-IDF access, and the intermediate conditions reveal direction-dependent interactions in average precision and retrieval at fixed review budgets. Component-level profiling separates parsing and representation costs from classifier training, prediction, and storage. Together, these findings connect detector comparisons to the test population, preprocessing state, visible information, and measured pipeline stages, and identify the protocol fields needed alongside a score to support interpretable comparisons of log anomaly detection accuracy and resource use.

0.64

Fast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical Study

cs.SE ·2026-11-05 ·Jesse Nyyssölä, Mika Mäntylä, Yuqing Wang
hot direction similarity 0.56
Abstract

Anomaly detection on software logs ensures system reliability through failure identification, root cause analysis and fault localization. While supervised models show excellent performance, in a practical setting there rarely are labeled training data available. This calls for the use of unsupervised models. We study effects of data preprocessing, anomaly detection method, and the dataset. For preprocessing, we examine three options: The use of anomaly-free training data, using randomized as opposed to chronologically selected training data, and the use of a custom loader to extract log message from the log line. Regarding anomaly detection we use four models (KMeans, IsolationForest, RarityModel, Out-Of-Vocabulary Detector) and three representations (Words, trigrams, and parsed events). We evaluate these approaches on seven datasets: BGL, HDFS, Hadoop, Spirit, Thunderbird, Liberty, and LO2Token. We run each combination of the selected factors and measure performance with AUCROC and F1-score metrics. Then we utilize a generalized linear model to establish relationships between the factors. Furthermore, we address execution times. Our evidence reveals large variability across all datasets, preventing us from making recommendations that apply universally. For instance, log parsing enhances performance on some datasets but reduces it on others. The evidence suggests that using anomaly-free data for training unsupervised models can improve anomaly detection performance, though this requires extra effort to identify such data.Previous work has argued that shuffling data leads to unrealistic model performance (data leakage). Yet, our results show this does not hold for the unsupervised models we tested, while it remains true for supervised models in the same setup. For fast execution of the pipeline, use word representation and avoid IsolationForest model.

0.25

Harbormaster: Evidence-Gated, Replay-Safe Maritime Anomaly Detection on AWS

cs.CR ·2026-09-30 ·Arun Sharma
similarity 0.54
Abstract

Ships broadcast their positions through the Automatic Identification System (AIS), and those reports can be false or missing. An operator who acts on an anomaly alert needs that alert to be attributable, reviewable, and recoverable after a failure. This paper describes Harbormaster, a production-shaped system on AWS that follows three rules. Physics checks run on every report before any learned model runs. The DynamoDB read store is an idempotent projection of PostgreSQL, and a guard on the log sequence number (LSN) of each change protects every write to it. A candidate model must pass a holdout gate and a shadow comparison before a canary gives it traffic, and a burn-rate check guards each canary step. The paper proves that replaying any prefix or suffix of the change log leaves each projected key at the value of its highest applied LSN. It also notes effects this result does not cover, such as repeated cache invalidations and repeated audit rows. In a bounded AWS window, a one-hour soak returned 35,999 HTTP 200 responses to 36,000 requests, with a 95th-percentile (p95) client latency of 142.751 ms. In a separate bounded AWS run of 900 s, the stream path received a burst of 400 records/s, and its consumer lag later drained to zero. Every number in the paper carries a label that says where it was measured, and the paper lists the parts of the design that were never built.

0.29

TRAIL: Towards Effective Log Parsing for Rare Logs through Template-Preserving Augmentation

cs.SE ·2026-09-21 ·Sepideh Hodaeian, Zhenhao Li, An Ran Chen
benchmark similarity 0.53
Abstract

Log parsing is essential for system log analysis because it supports tasks such as debugging, monitoring, and anomaly detection by transforming unstructured log messages into structured log templates. However, real-world log datasets exhibit highly imbalanced, long-tailed distributions, where a small number of frequent templates dominate while many rare templates appear only a few times. This imbalance causes evaluation results to be overly optimistic by allowing frequent templates to dominate benchmark metrics, while poor performance on rare yet operationally important events remains largely hidden. In this paper, we investigate the prevalence and impact of rare log groups, defined as log groups with fewer than five instances. Our empirical study on the widely used Loghub-2.0 benchmark shows that rare log groups account for nearly 20% of all templates but less than 0.01% of log messages. Because they contain only a handful of instances, all evaluated parsers experience substantial performance degradation on these groups. To address this challenge, we propose TAILOR, a log parsing framework that improves template inference for rare log groups through template-preserving augmentation. TAILOR enriches rare log groups with template- consistent log messages before template inference. The additional structural evidence helps distinguish static tokens from dynamic variables. Our experimental results show that TAILOR improves parsing accuracy on rare log groups by 19% over the strongest baseline while maintaining competitive performance on complete datasets. We further show that the proposed augmentation strategy generalizes across different LLM backbones and consistently improves existing LLM-based parsers without modifying their core architectures.

0.13

Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities

cs.CL cs.CR ·2026-11-02 ·Junwei Yao, Jianan Liu, Yongxi Zhou, Zike Yang
similarity 0.52
Abstract

Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low confidence ranks hijacked answers above clean ones on four of the five systems (AUROC 0.86-0.90). Ranking them does not stop many of them. A threshold set so that at most 5% of clean answers to separate calibration articles alarm falls among the least confident clean answers, and every more confident hijack passes. In our main gate run on the three API-served systems, the monitor misses 51-70% of the hijacks caused by a plain injected note (per-system point estimates). When the note adds one handwritten sentence that calls the target certain and asks for full probability, the monitor misses 99-100% of the hijacks, although low confidence still ranks them at AUROC 0.70-0.79. Confidence monitors should be tested at their operating alarm rate, against attackers who claim certainty.

0.27

Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models

cs.AI cs.CL ·2026-10-31 ·Alejandro Mozo Quesada, Sergio Gálvez Rojas, Francisco López Valverde
similarity 0.52
Abstract

System One decision models replace text generation with typed, probability-scored answers and are entering production. Answers execute without a person in the loop, and no multi-model audit exists on established bias benchmarks. We provide that audit: three benchmarks: BBQ (all eleven categories), CrowS-Pairs, and SilverSpoon, adapted to the typed interface, twelve models plus a no-thinking ablation, and 1,638,000 decisions. Every order-sensitive item runs twice with options swapped; native typed models also swap labels; ten configurations repeat under extra seeds and batch 16. We find widespread stereotype alignment, order-dependent answers, and confidence that tracks instability rather than bias. In BBQ, 97 of 143 model-by-category scores are significant and 94 aligned, while the commercial reference abstains on 99.5% of ambiguous items, hiding aligned probability mass. Option swaps flip 0.0% to 72.6% of choices; typed scoring stays below 10% except Laya (22-33%), and untuned letter prompting reaches 73%. CrowS-Pairs preference spans 0.496 to 0.816, and boolean answers can be decided by position (1.00 versus 0.005). SilverSpoon judgments shift toward lower-SES labels in five of eight decision models, inherited from the 9B backbone or induced by tuning. Audits must permute options and reverse questions by default; order-invariant scoring should become a procurement requirement.

0.28

When Category Means Move: A Paired-Support Audit of Recommendation Logs with Random Insertions

cs.IR ·2026-11-03 ·Yu Zhang, Zhihan Wang, Yuhang Lu, Shuai Li
publishedunusual similarity 0.52
Abstract · 10 pages, 6 figures. Accepted at WSDM 2027 (Main Track)

Content-category dashboards summarize production recommendation logs, although the serving policy selects the users, items, and contexts behind each mean. Using KuaiRand random-insertion impressions, we audit these summaries on the same 12,472 users and 5,486 videos in both logs. Standard means have absolute/centered mean absolute error (MAE) 0.1935/0.0491 and rank correlation 0.7833. User demeaning plus a random grand-mean anchor reduces absolute MAE by 80.0%, but centered MAE by only 21.0%, without reliably improving rank. A complete-support ledger allocates weighted squared error mainly to within-video residual (72.41%) and video mix (23.90%), with 3.65% from random support. In a frozen later test, early category calibration reaches MAE 0.0251 versus 0.0531 for scalar calibration; their direct difference is -0.0280 [-0.0296, -0.0172]. Decision evidence is cutoff-specific: the nominal top-10 regret change for row-weighted demeaning versus naive is -0.0062 [-0.0084, -0.0003], while top-5 and top-20 intervals include zero. The six-contrast family-wise top-10 interval [-0.0136, 0.0013] includes zero. An outcome-free disjoint randomized-reference split also leaves all six nominal contrasts covering zero. A known-truth stress test shows incomplete improvement under a mean-additive positive control and failure under item mix and weak overlap. All decompositions are descriptive and each result is conditional on its overlap cohort. KuaiRand intervals quantify user-cluster variation conditional on the realized catalog and calendar; we claim neither intrinsic category effects nor a universally valid correction.

0.42

Machine learning for journal entry testing: A type-aware evaluation of anomaly detectors under a review budget

cs.LG cs.AI ·2026-10-05 ·Jan Gronewald, Michel Scherer, Nijat Mehdiyev
similarity 0.52
Abstract

Journal entry anomaly detectors are commonly evaluated on the full population with ROC-AUC, precision and recall, ignoring the review budget and which anomaly types are found. We propose a type-aware evaluation combining per-type recall, fair-share type recall (FSR), which caps each type's credit at its budget share, type coverage and first-hit rank. We evaluate nine unsupervised detectors, a supervised reference and feedback-driven Deep Semi-Supervised Anomaly Detection (DeepSAD) on four real client ledgers with injected typed anomalies and a public synthetic ledger. On the largest client ledger, principal component analysis (PCA), an autoencoder (AE) and a variational autoencoder (VAE) each place on average 98 anomalies among the first 100 postings, but at least 95.8 belong to one type. FSR instead favours a nearest-neighbour (kNN) detector and changes the top-ranked detector on three of four client ledgers. Representation also matters: one-hot encoding exposes unseen accounts, whereas frequency encoding leaves unseen contra accounts largely undetected. On the public ledger, the Histogram-Based Outlier Score (HBOS) and Empirical Cumulative Distribution-Based Outlier Detection (ECOD) reach all eight markings within 1,386 entries, whereas kNN, the hit leader at 1,000 entries, first reaches cross-linked clearing at rank 4,641, and the supervised row-level reference misses this marking within 1,000 entries. There, the adaptive DeepSAD review protocol raises mean hits per 100 reviews from 40.0 to 68.3 but type coverage only from 2.7 to 3.0. These findings show that high hit rates can conceal systematic blind spots and suggest that feedback can reinforce existing detection patterns without broadening anomaly coverage.

0.16