CS Paper Radar

Patching the Blind Spots: Reversible LLM Rule Overlay for Frozen Fraud Detectors

Bo Qu, Zhurong Wang, Frank Zahradnik, Ming Ouyang

cs.LG cs.AIcs.CL ·submitted 2026-11-05 21:12 UTC · v2
benchmarkhot direction

Abstract

Gradient-boosted trees dominate industrial tabular fraud classification, but even a well-trained model has blind spots, and retraining a deployed model is not always safe. We bring LLM reasoning to patching such a model without modifying it: LLMs read its tree digest and SHAP attributions, reason about the fraud it misses, and propose rules anchored on features it underweights, screened and calibrated into a reversible log-lift overlay at the production cut-off. Single-LLM benefit is high-variance and inverts between datasets, so we pool three diverse backbones by logit averaging, trading peak gain for a bounded worst case. On an experimental detector trained on sampled, selection-biased real-world e-commerce data, tuned continued training regresses, whereas the overlay gains on average +2.26% recall at equal precision ($Δ$R@P) and +1.60% precision at equal recall ($Δ$P@R). Replayed offline on a legacy production detector, the overlay, calibrated on one month and frozen, stays more precise at equal recall for seven months (+5.64% $Δ$P@R). On seven bias-shift variants of a public benchmark it raises $Δ$P@R on all and keeps $Δ$R@P near neutral. LLM reasoning over a model's diagnostics can improve a deployed tabular classifier through readable predicates naming the fraud they cover.

Comment
12 pages, 1 figure, 8 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Dual-Stream LLM-Guided Reward Design for Reject-Inference-Aware Reinforcement Learning in E-Commerce Payment Fraud Detection

cs.LG cs.AIcs.CL ·2026-11-05 ·Bo Qu, Zhurong Wang, Frank Zahradnik, Ming Ouyang
published similarity 0.68
Abstract · 12 pages, 1 figure, 3 tables. Extends Qu et al., ACL 2025 Industry Track: https://aclanthology.org/2025.acl-industry.9/

Production fraud-detection models learn from a censored slice of their own traffic: transactions a legacy system blocked never receive fraud labels, so models train on a biased, approved-only distribution. We extend LLM-guided reward design, the framework of Qu et al. (2025) in which an LLM evolves the reward of a two-step REINFORCE fraud-blocking policy, to attack this reject-inference problem inside the reward. A single LLM designs a dual-stream reward: stream 1 applies the standard dollar-weighted precision objective to labeled transactions; stream 2 scores legacy-blocked, unlabeled transactions by contrasting the newest gradient-boosted risk signal against the legacy scores that drove the original block. Rather than prescribe which signals matter, we expose a superset of evidence and let the LLM decide for itself which to use, guided by evaluation feedback alone; because it writes the reward as code, that choice is directly auditable. On a held-out four-month out-of-time test, the dual-stream agent beats the prior labeled-only agent at every target recall, by +4.7, +2.4 and +3.0 points of dollar-weighted precision at 80%, 85% and 90% recall, and the production baseline by +6.8 points on average. The resulting policy is architecturally identical to that of Qu et al. (2025) and adds no inference cost.

0.36

Explainable and Adaptive Machine Learning for Fraud Risk Detection in Bangladesh's Digital Banking Ecosystem

cs.LG ·2026-10-29 ·Md Najmus Swaqeeb
publishedbenchmark similarity 0.65
Abstract · 23 pages, 6 figures, 5 tables. Presented at the 11th Annual Banking Conference 2026, Bangladesh Institute of Bank Management (BIBM), Dhaka, …

The rapid expansion of digital banking and mobile financial services creates growing challenges for fraud detection, particularly under severe class imbalance and operational constraints. This study presents an explainable and adaptive machine-learning framework integrating chronological evaluation, leakage-conscious feature engineering, probability calibration, SHAP-based explainability, and adaptive risk prioritization. Experiments use the synthetic PaySim mobile-money benchmark and compare Logistic Regression, Random Forest, XGBoost, and LightGBM. XGBoost achieved the strongest fixed-threshold performance, with PR-AUC of 0.5658, ROC-AUC of 0.9068, and F1-score of 0.5369. Platt calibration reduced the test Brier score from 0.1141 to 0.0245, improving probability reliability. SHAP analysis identified temporal, destination-account, transaction-type, and amount features as important model drivers. An adaptive top-1% policy reduced false positives from 192 to 117, but recall declined from 0.4236 to 0.2122, demonstrating a clear alert-capacity trade-off. The results provide a reproducible methodological framework rather than evidence of deployment performance in Bangladesh, since PaySim is synthetic. Validation using governed real-world financial data is therefore an important direction for future research.

0.29

CRISP: Scalable Importance-Stratified Coresets for Imbalanced Tabular Learning

cs.LG ·2026-09-22 ·Hardhik Mohanty, Indrayana Rustandi, Mohamadreza Sheibani
similarity 0.65
Abstract

Large imbalanced tabular datasets make repeated gradient-boosted tree training expensive. Existing coreset methods often lose accuracy when most majority examples are removed. We present CRISP (Coreset Reduction via Importance-Stratified Pruning), a linear-time method that allocates a negative-class budget across quantile strata of a proxy-model score. Sample weights account for unequal inclusion probabilities. At 95% negative-class reduction on a production fraud dataset, CRISP trains on approximately 1.70M of 25M rows and retains 99.7% of full-data Average Precision. This is a 93.2% reduction in total training rows. On public CriteoPrivateAds, CRISP has the highest mean Average Precision at each tested rate from 90% to 99.4% majority reduction. Sparkov results are mixed at lower rates, but CRISP has the highest mean at 99.2% and 99.4%. Ablations identify budget allocation and inverse-propensity weighting as the main sources of the production-dataset gain.

0.20

When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

cs.AI cs.LG ·2026-08-17 ·Diyorbek Musaev
benchmarkcode/data similarity 0.63
Abstract · 13 pages, 3 figures, 5 tables. Code and per-run metrics released

Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.

0.39

ChequeMark: An Ensemble Machine Learning Framework for After-Hours Business Deposit Fraud Detection

cs.LG ·2026-08-21 ·Ann Youduo Xu, Emily Yu, Justin Leski, William Lam
similarity 0.63
Abstract

Cheque fraud is a material risk in after-hours business deposit operations because funds may be released within one business day, while cheque clearing takes several days. This timing gap creates a fraud exposure window for financial institutions. Prior mitigation relies on static, deposit-level checks and therefore miss historical client behavior and evolving patterns. To address this gap, we propose a multi-view ensemble ML framework that combines: Extreme Gradient Boosting (XGBoost) for known fraud patterns, Isolation Forest for label-free anomaly detection, and Graph Sample and Aggregate (GraphSAGE) for relational patterns associated with transaction activities. We then combine the three outputs into a single client-level risk score. Under stable conditions, performance is comparable to XGBoost; under a targeted distribution shift, our framework performs best (F1: 83.77%, FPR: 0.69%) versus XGBoost (F1: 82.77%, FPR: 0.72%). These results indicate improved robustness to distribution shift while preserving interpretability through plain-language explanations grounded in behavioural, anomaly, and relational evidence.

0.10

SR-Fraud: An Outcome-Supervised Reflective LLM Agent Framework for Non-Stationary Payment Fraud Detection

cs.LG cs.CR ·2026-09-23 ·Xuwei Tan, Yao Ma, Xueru Zhang
benchmark similarity 0.63
Abstract

Real-time payment fraud detection is a non-stationary streaming prediction problem: adversaries adapt before supervised labels mature, and localized burst attacks can cause losses before retraining. Production systems typically rely on tabular classifiers and rules, which can struggle to capture these emerging sequential patterns before periodic retraining occurs. We present SR-Fraud, an outcome-supervised reflective LLM framework that decouples request-time decisions from offline adaptation. A frozen, stateless agent scores each transaction from a Hybrid Episodic Window to track behavioral shifts, while an offline reflection agent proposes boundary hypotheses from matured errors. A deterministic verifier then admits only supported hypotheses into an executable knowledge state. On a production payment-fraud benchmark, SR-Fraud improves all detection metrics over its frozen decision agent, obtains higher point estimates than static and periodically retrained CatBoost, and detects an emerging fraud burst.

0.55

How Much Do Foundation Models Add to Gradient Boosting on Financial Event Sequences?

cs.LG ·2026-11-13 ·Carlos Eduardo Martins Relvas, Gustavo di Giovanni Bernardo
code/data similarity 0.62
Abstract · 49 pages, 5 figures, 24 tables. Code, synthetic data generator and per-seed predictions available at https://github.com/datarisk-io/fm-vs-gb…

Foundation models pretrained on raw transaction sequences are proposed for credit, fraud and churn modeling, with reported gains over gradient boosting, but the data and baselines behind these gains are rarely public. We measure how much the pretraining and adaptation recipes of these models add to a baseline of point-in-time window aggregates and LightGBM, and we propose TBS (Temporal Boosting-Sequence ensemble), which averages the ranks of a boosting model enriched with label-free event-transition features and of a pretrained sequence network fine-tuned jointly with the aggregates. On a synthetic credit-card dataset, the public MBD-mini banking dataset and two proprietary datasets of trade credit and B2B churn, we compare 27 methods at three label budgets with five seeds and paired bootstrap intervals, using sequence models of about 0.4 million parameters pretrained per dataset. Frozen self-supervised embeddings rarely improve the baseline, and a fine-tuned sequence network alone clears it in 9 of 21 target-budget cells. TBS clears it in 19 of 21 cells (18 after FDR adjustment), with a median gain of +1.8 AUC points, and the controls suggest that combining boosting with the pretrained network is the main source of the observed gain. Gains shrink as labels grow and with rich tabular data: with both, the gain on the proprietary datasets is +0.7 and +0.8 points, the latter with an interval including zero. We also re-examine two benchmarks. On IBM TabFormer, the test follows a change in the fraud pattern and the published baseline has no history; with the training period the vendor states, our history-aware LightGBM and our own foundation model stay below the published scores of a commercial model. On IBM AML, a simulation artifact lets a date rule reach F1 0.482, and a point-in-time LightGBM reaches the F1 of graph neural networks whose test graph includes later transactions.

0.33

CARTA: Self-Supervised Reusable Transaction Embeddings via Heterogeneous Tokenization and Cross-Attention

cs.LG cs.CL ·2026-10-17 ·Shreyas K. Chandrahas, Jose Munoz, Satish Babu Bellapu
published similarity 0.62
Abstract · 9 pages, 4 figures. Accepted at ICAIF '26 (7th ACM International Conference on AI in Finance), Milan, Italy, November 14-17, 2026

Acquiring fraud labels for payment transactions is expensive and inconsistent across tasks, and high-cardinality identifiers such as merchant names make one-hot feature pipelines infeasible at scale. We present CARTA, a self-supervised encoder that learns a compact 16-dim embedding from raw transactions without labels; the frozen embedding is then reused for label-efficient fraud probing and exploratory unsupervised risk stratification without retraining the encoder. CARTA uses type-specific projections with learned embedding tables for high-cardinality identifiers, trained with token excision and InfoNCE contrastive loss. On IEEE-CIS, its linear probe at 1% of labels exceeds a raw-feature probe at 100%; on IBM TabFormer (2M rows), its frozen embedding reaches GBDT AUPRC 0.508, above frequency encoding (0.411) and the merchant-dropping baseline (0.022), though fully-supervised CatBoost is stronger. CARTA is a label-free, reusable representation that complements task-specific supervised models.

0.37