CS Paper Radar

Evaluating Near-Perfect Accuracy Claims in Machine-Learning-Based Leak Detection for Two-Phase Pipelines: A Condition-Blocked Audit

Issah N. Suleiman, Sormeh Serpoosh, Nadine Elkholy, Hicham Ferroudji, Mohammad Azizur Rahman, Matthew Hamilton

cs.LG ·submitted 2026-11-15 12:13 UTC · v1

Abstract

Reliable leak detection in gas-liquid two-phase pipelines is important for process safety, yet machine-learning studies frequently report near-perfect performance using evaluation protocols that may not represent deployment to unseen operating conditions. This study audits a pressure-based leak-detection model using condition-blocked leave-one-condition-out validation, in which the classifier is tested on a flow condition excluded from training. Accuracy decreased from 93.3% under within-condition evaluation to 59.1% across fifteen unseen-condition tests. Ablation analysis showed that most of this difference arose from the data-splitting protocol, with a smaller contribution from encoding operating conditions as model inputs. Under condition-blocked evaluation, the false-alarm and missed-detection rates increased to 34.1% and 10.2%, respectively, demonstrating that near-perfect reported accuracy did not translate into reliable operation at unseen flow conditions. The reduction persisted across all evaluated classifiers, indicating that it was caused by the evaluation design rather than a particular model architecture. Grouped evaluation of an independent acoustic pipeline dataset produced a comparable reduction toward chance-level performance. These findings show that validation units must represent the conditions encountered during deployment. Condition-blocked evaluation and explicit reporting of operational error rates are therefore necessary before machine-learning leak detectors can support process-safety decisions.

Comment
8 figures, 8 tables. Analysis code archived at https://doi.org/10.5281/zenodo.24218226

Closest papers

Nearest neighbours by meaning across the whole index.

Verifiable abstention makes AI leak diagnosis accountable in urban water distribution networks

cs.AI ·2026-08-19 ·Tianwei Mu, Yue Wang, Mingzhe Yuan, Manhong Huang, Wenhong Wang +6 more
benchmark similarity 0.57
Abstract · 45 pages, 5 main figures, 1 main table, 5 supplementary figures, 15 supplementary tables. Code and data availability described in the paper

Leak localization is usually evaluated as forced-choice prediction, although sparse hydraulic observations may not justify excavation. Here, we quantify a pressure-information limit and use it to recast localization as selective, evidence-gated decision-making. A physics-grounded executor falsifies competing leak, demand, sensor and valve hypotheses in a hydraulic twin. Deterministic code computes every number and every acceptance predicate; an independent large language model auditor may add a rejection but never overturn a failed check. Forced retrieval placed only 95 of 300 leaks in the correct zone. Across 550 mixed events, the gate acted on 223 (214 correct); on a third-party 33-leak benchmark, all four accepted events were correct. In a replay of 194 audited City D repairs, the pressure tier authorized five excavation recommendations, three matching the repaired district, while the district-inflow tier returned the correct district for 85 events. Observability limits with machine-checkable abstention enable auditable utility intervention.

0.40

Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models

cs.CL ·2026-09-10 ·Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif, Samer Ellaham, Cedric Schmitz
hot direction similarity 0.53
Abstract

Cardiovascular screening models trained on national health surveys routinely report areas under the receiver operating characteristic curve (AUROC) near 0.89. We asked whether that accuracy reflects learning or target leakage, whether tabular foundation models change the answer, and whether the properties deployment requires survive joint examination. We benchmarked ten classifiers spanning linear, tree-ensemble, neural, glass-box, and tabular foundation classes for prevalent myocardial infarction in 442,067 respondents of the 2022 Behavioral Risk Factor Surveillance System across five feature tiers of decreasing leakage risk. Each was audited for discrimination, calibration, fairness at an explicit screening threshold, conformal coverage, explanation faithfulness, and inference cost, then applied -- models and thresholds frozen -- to 430,755 respondents of 2023. Removing two post-diagnostic features cost every model 0.049-0.051 AUROC, collapsing the field into a 0.0045-wide band. The glass-box explainable boosting machine was non-inferior to every alternative within a pre-specified 0.005 margin while scoring the cohort roughly 104 times faster than the strongest foundation model. One threshold detected 75.4% of women's infarctions against 89.0% of men's; editing the model's shape functions reduced the gap to 0.010. Marginal conformal prediction gave 0.86 coverage to men and 0.82 to adults over 60; Mondrian calibration repaired every stratum. Frozen models transported within 0.002 AUROC. Reported headroom in this literature is a property of the feature set, not the learner. Transparency cost nothing measurable and made fairness repair and uncertainty conditioning directly auditable. Evaluation practice, not model capacity, is the binding constraint.

0.37

A Controlled Audit of Architectural Complexity in Uncertainty-Aware Multi-Organ Ultrasound Classification

cs.CV ·2026-08-28 ·Yang Song, Pengbo Sun, Shichang Feng, Ye Zhu, Xin Xu +1 more
similarity 0.53
Abstract · Supplementary material is available as an ancillary file on this arXiv page

Multi-organ ultrasound classifiers increasingly combine attention, mixture-of-experts routing, uncertainty gating, and evidential deep learning (EDL) objectives to address heterogeneous anatomy and acquisition. Yet a plausible design rationale does not by itself establish that an added component improves the trained system. We contribute a controlled complexity-audit framework, applied to the deployment decision between the maximal evidential candidate Full-EDL and simpler alternatives. Six candidates were evaluated on the primary dataset and three in an internal replication, using ten matched seeds, frozen image-level partitions, capacity- and optimisation-aware comparisons, symmetric temperature scaling, paired decision rules, and a separate out-of-distribution (OOD) veto. Retaining Full-EDL did not establish a reliable macro-F1 gain on either dataset, while the simplified alternatives remained inconclusive under the non-inferiority margin. Simple cross-entropy with temperature scaling (Simple-CE+TS) met the calibrated negative log-likelihood criterion on both datasets and showed favourable selective-risk ordering. The raw calibration advantage of evidential training disappeared after temperature scaling and did not recur on the second dataset. The gate had negligible observable influence at the audited checkpoints, and deleting the Full-only chain revealed no stable task or calibrated-loss benefit. Simple-CE nevertheless triggered the OOD veto against the fetal probe but not the lung probe, precluding an unconditional OOD-safety claim. We therefore selected Simple-CE+TS for the evaluated in-distribution objective while retaining Full-EDL as the maximal reference. Components should earn retention through functional and retraining-based evidence, and calibration and distribution-shift reliability should be evaluated separately.

0.19

Robustness of Anomaly Detection Models for Industrial Control Systems under Training-Time Data Contamination

cs.CR cs.LG ·2026-08-24 ·Mustafa Umut Ozbek, Taiwo Ojo, Pooria Madani, Khalil El-Khatib, Li Yang
publishedbenchmarkcode/data similarity 0.52
Abstract · Accepted and to appear in IEEE CASCON 2026. Code is available at: https://github.com/ANTS-OntarioTechU/Robustness-Anomaly-Detection-ICS-Data…

Machine-learning-based anomaly detection is increasingly used in industrial control systems (ICS), yet most studies assume that detector training data is trustworthy. In practice, training data may be corrupted through compromised logs, labeling errors, manipulated historian records, or unsafe retraining processes. This paper evaluates the robustness of offline ICS anomaly-detection pipelines on the Secure Water Treatment (SWaT) benchmark under training-time contamination. We assess 11 heterogeneous anomaly detectors under three contamination strategies: random injection, similarity-targeted injection, and feature-noise injection. The first two insert attack samples into the nominal training pool, while the third adds bounded Gaussian noise to selected normal training samples. These attacks are contamination-based rather than gradient-driven poisoning methods. Contamination budgets from 1% to 10% are evaluated using clean validation and test sets under a unified offline protocol. The results show that robustness is strongly model-dependent and cannot be predicted from clean-data performance alone. Injection-based contamination causes the greatest degradation, particularly for local-density and distance-based detectors, whereas feature-noise contamination has a comparatively limited effect. PCA, SVM, HBOS, and IForest remain relatively stable, while the tuned neural detectors demonstrate intermediate robustness. Overall, the findings highlight the importance of training-data integrity in ML-enabled ICS monitoring, subject to the evaluated dataset, models, and threat assumptions.

0.56

Multiple Leak Localization in Water Distribution Networks through Graph Neural Networks

cs.LG cs.AI ·2026-10-26 ·Arvin Ajoodani, Sara Nazif, Pouria Ramazi
benchmarkunusual similarity 0.52
Abstract

Leak localization in water distribution networks is challenging when multiple leaks occur simultaneously, particularly under sparse monitoring. Existing data-driven methods often assume a single or predefined number of leaks or require extensive enumeration of leak combinations. This study proposes a graph neural network (GNN) model that localizes a variable number of simultaneous leaks without specifying the leak count in advance or exhaustively generating every possible combination of leak locations for training. The model is evaluated on two benchmark networks of different sizes, the 31-node Hanoi network and the larger 92-node NET3 network, under variable demand, sparse pressure-sensor coverage, variable leak magnitudes, and varying numbers of simultaneous leaks. The GNN outperforms multilayer perceptron, support vector machine, K-nearest neighbors, and random forest classifiers by up to 12% and 20% in accuracy respectively in Hanoi and NET3 networks. Consistent performance across both networks supports the scalability of the proposed approach.

0.27

Acoustic Anomaly Detection for Early Identification of Drill Jamming

cs.LG ·2026-11-05 ·Maria Beatriz Silva, Lucas Edson Silva de Araújo, Sergio F. Chevtchenko, Ermeson Andrade, Danilo Araujo +1 more
publishedbenchmarkdataset similarity 0.52
Abstract · Accepted for publication at IEEE INDIN 2026. 8 pages, 6 figures. (c) 2026 IEEE. Personal use of this material is permitted

In the context of Industry 4.0, predictive maintenance is essential to minimize unplanned downtime and ensure operational safety. In machining processes, drill jamming is a critical failure that compromises both workpiece quality and tool integrity. This work proposes an acoustic anomaly detection system using ultrasonic sensors to identify pre-failure patterns in Computer Numerical Control (CNC) machining. We release a public dataset of time- and frequency-domain acoustic features from 17 drills, and benchmark semi-supervised autoencoder-based normality modelling against supervised XGBoost classification using Leave-One-Drill-Out validation. The results demonstrate that the Autoencoder is highly effective for industrial safety, achieving a Recall of 1.00 and a consolidated F1-score of 0.81. Furthermore, the threshold-independent analysis shows that the Autoencoder better separates normal and pre-failure acoustic patterns, achieving a consolidated AUC of 0.8294, compared with 0.7559 for the supervised model under inter-tool covariate shift. In contrast, the supervised XGBoost achieved an average F1-score of 0.55 and failed to detect failures in some of the evaluated drills. The findings highlight that the ultrasonic spectrum provides a reliable window for identifying incipient mechanical instabilities, and the AE model offers a robust safety envelope by anticipating all jamming events across the tested drill bits.

0.34

A Hybrid Two-Stage Machine Learning Pipeline for Fault Detection and Classification in Power Transmission Systems

eess.SY cs.LG ·2026-08-24 ·Sahil Manikshete, Atharva Gujarathi, Thanh Long Vu, Akhtar Hussain, Van-Hai Bui
benchmarktheory similarity 0.51
Abstract · 21 pages

Rapid and accurate fault detection in high-voltage transmission networks is essential for grid reliability and equipment protection. Transmission fault datasets are frequently imbalanced, and certain fault types produce electrical signatures that fall within the normal operating envelope, causing single-model classifiers to fail on safety-critical cases. This paper proposes a hybrid two-stage machine learning pipeline that decouples detection from classification. Stage 1 combines an Isolation Forest anomaly detector with an optional supervised binary detector through an OR-fusion rule; the supervised branch is allocated automatically during training for any fault class the anomaly detector cannot resolve, and is omitted when no such class exists. Stage 2 applies a Random Forest multiclass classifier only to samples flagged by Stage 1. Feature engineering is expressed as a per-measurement-point operator mapping six raw channels to eighteen features, including zero-sequence symmetrical components derived from Fortescue's theorem, yielding 18L features for L measurement points. On the TLFaultDataset, the pipeline raises Line-fault end-to-end accuracy from 31.3% to 95.8%. On an independent single-point dataset, the same framework attains 97.25% end-to-end accuracy across all classes including normal operation, exceeding the TLFed federated benchmark of 94.84% without GPU or federated infrastructure, at 0.05 ms per sample on CPU. Ablation on both datasets shows zero-sequence features resolving the three-phase versus three-phase-to-ground ambiguity, raising the F1-score of that class pair from 0.39 to 0.997. The direction of the zero-sequence signature is found to be system-dependent, motivating a learned decision boundary in place of a fixed relay threshold.

0.28

Leakage-Safe Machine Learning for Hydrogen Embrittlement Detection in 316L Stainless Steel: A Region-Held-Out Evaluation of Texture and Deep Features in SEM Micrographs

cs.LG cond-mat.mtrl-sci ·2026-09-23 ·Muhammad Awais, Muhammad Yaseen, Abdul Shakoor, Niaz Ahmed Niaz, Huria Zia +1 more
similarity 0.51
Abstract

Scanning electron microscopy (SEM) is routinely used to characterize the microstructural changes caused by hydrogen embrittlement (HE) in structural steels. Machine learning can automate this characterization, but models are often evaluated using image-level splits. When several images come from the same specimen region, such splits leak information between the training and test sets. Here, we propose a region-held-out protocol for classifying as-received (AR) and hydrogen-charged (H2) SEM micrographs of 316L stainless steel, based on Leave-One-Region-Out (LORO) cross-validation over 14 spatial regions (8 AR, 6 H2; 31 images). We compared six feature-classifier combinations built on local binary patterns (LBP), grey-level co-occurrence matrices (GLCM), self-supervised convolutional embeddings pretrained on 143 unlabeled SEM images, and a convolutional neural network (CNN). The simplest texture approach, LBP with a support vector machine (LBP+SVM), performed best, achieving a balanced accuracy of 0.79, H2 recall of 0.69, and H2 precision of 0.82, outperforming every deep-learning and combined-feature model. A group-level permutation test (500 permutations sampled from the 3,003 possible region-to-label assignments) yielded p = 0.008, indicating that the result cannot be explained by a chance alignment of the region structure. Grad-CAM maps from a CNN trained on the full dataset tended to concentrate on localized surface and grain-boundary features, where hydrogen-induced morphological changes are known to occur. Under a leakage-safe, statistically validated protocol, texture descriptors recover a hydrogen-charging signature from SEM micrographs even with few samples, and the same protocol can be extended to larger HE detection studies in other alloy systems.

0.26