CS Paper Radar

LLM Ensemble Fault Classification for Automotive HiL Validation

Hamza Ouarrad, Mohammad Abboush, Andreas Rausch

cs.SE ·submitted 2026-08-11 09:31 UTC · v1
top venue

Abstract

Automotive HiL validation generates large multivariate test recordings whose analysis remains challenging due to manual review effort, rule-based limitations, and the need for explainable diagnostic decisions. Recent machine-learning and deep-learning approaches have improved fault diagnosis, but they often require large labelled datasets, generalise poorly across operating conditions, and provide limited insight into their predictions. This paper proposes an explainable multi-LLM ensemble framework for sensor-level fault classification in automotive validation. The framework uses compact evidence representations of fault-injection recordings and combines the outputs of heterogeneous large language models to improve diagnostic robustness, ranking quality, confidence reliability, and interpretability. The approach is evaluated on gasoline-engine and electric-vehicle HiL systems across three driving settings and ten single-fault classes. Among the individual models, Mistral Small~24B provides the strongest overall single-model trade-off, achieving 0.903 Top-1 accuracy, 0.887 MCC, and the lowest Brier score of 0.102. The final Top-3 ensemble combines Mistral Small~24B, Qwen2.5~32B, and Phi-4~14B using confidence-weighted voting, improving the scenario-averaged results to 0.917 Top-1 accuracy, 0.913 macro F1, and 0.902 MCC, while also providing the best calibration among the tested ensemble strategies. A Top-5 ensemble does not improve over the Top-3 configuration, indicating that model complementarity is more important than ensemble size. The results show that coordinated multi-LLM reasoning can support robust, calibrated, and engineer-interpretable fault classification for automotive HiL validation.

Comment
8 pages, 3 figures. Accepted at the 23rd Workshop on Model Driven Engineering, Verification and Validation (MoDeVVa 2026), co-located with MODELS 2026. Original submitted version

Closest papers

Nearest neighbours by meaning across the whole index.

When More Modalities Hurt: Modality Dropout for Heavy-Duty Vehicle Engine Diagnostics

cs.LG ·2026-08-24 ·Adeel Zafar, Slawomir Nowaczyk, Hamid Sarmadi, Saeed Gholami Shahbandi
similarity 0.69
Abstract

Heavy-duty vehicle diagnostics generate three disconnected data modalities: unstructured multi- lingual service complaints, high-dimensional sensor telemetry with over 80% missing values, and Diagnostic Trouble Codes (DTCs). We investigate whether fusing these modalities improves engine component classification on a proprietary dataset from a major truck manufacturer. Through 5-fold cross-validation across multiple model configurations spanning three model families on five engine component classes (885 samples, the full cross-database matched population for this manufacturer), we find that naive fusion provides modest gains over text alone (65.3%). However, modality dropout during training, which randomly disables entire modalities per batch, forces the network to exploit weaker inputs and achieves 68.8% accuracy on text+DTC fusion (weighted F1: 0.67), a 3.5-point improvement over text-only (65.3%, weighted F1: 0.64) and the best result across all methods including logistic regression and gradient-boosted trees. Per-class analysis shows that the dominant modality varies by fault type: text describes symptoms, DTCs encode structured fault signals, and sensors measure physical state. On intake/exhaust faults, sensors alone reach 93% where text achieves 80%. On fuel system faults, fusion with modality dropout nearly triples accuracy from 15% to 38% over text alone. To our knowledge, this is the first application of three-way modality fusion combining text, sensors, and fault codes in industrial vehicle diagnostics.

0.12

Knowledge-Graph-Guided Retrieval-Augmented LLMs for Explainable Root Cause Analysis in Automotive HiL Validation

cs.CR cs.SE ·2026-08-11 ·Hamza Ouarrad, Mohammad Abboush, Andreas Rausch
published similarity 0.68
Abstract · 10 pages, 3 figures, 5 tables. Accepted for publication and oral presentation at the 10th International Conference on System Reliability and…

Hardware-in-the-Loop validation of automotive software systems generates large multivariate time-series recordings whose manual analysis is time-consuming and often limited to anomaly detection and fault classification rather than root-cause analysis. Although deep learning methods have shown strong performance in fault detection and classification, they usually require task-specific training or retraining when new fault locations, systems, or operating conditions are introduced. They also tend to treat localization as a classification task, without explicitly representing the spatial and functional relationships between fault locations, sensors, and downstream subsystem effects. This limits their generalizability and their usefulness for engineering root cause analysis and diagnosis. This paper proposes a knowledge-graph-guided retrieval-augmented large language model framework for RCA (root cause analysis) and fault localization in automotive HiL data. The method converts raw time-series recordings into compact diagnostic evidence, enriches this evidence with sensor-to-location and propagation knowledge, and retrieves similar historical cases to support the final reasoning step. The LLM is then used as a decision and explanation layer rather than as a direct time-series classifier, producing a ranked fault-location prediction together with an interpretable RCA explanation. The framework is evaluated on two automotive HiL case studies: an ASM gasoline engine and an electric vehicle system. The best-performing model achieves Top-1 accuracies of 90\% and 94\%, respectively, while recording-level aggregation reaches perfect file-level fault localization in the evaluated subset. These results demonstrate the potential of KG-guided RAG-LLM reasoning for explainable and generalizable HiL RCA.

0.25

Sensor Diagnosis for Fuel and Emissions Monitoring in Heavy-Duty Diesel Vehicles

cs.LG eess.SY ·2026-10-15 ·Tejas Dahiya, Saurabh Kumar Gupta
published similarity 0.63
Abstract · Accepted at the Tackling Climate Change with Machine Learning workshop at NeurIPS 2026

Faulty sensor readings can distort estimates of fuel use and carbon dioxide emissions from diesel engines. We evaluate two procedures that use relationships learned from healthy measurements to diagnose faults and guide fuel correction. The diagnosis procedure tests whether replacing a suspected faulty reading with an estimated healthy value helps a model predict another sensor's recorded reading more accurately. It then predicts the fault type and decides whether to report the diagnosis. We evaluate five synthetic fault types on 217 steady-state operating points from a heavy-duty diesel engine, excluding all measurements at selected engine speeds from training. Across 31,500 fault-injected groups of 20 operating points, it issues diagnoses for 77.3% of groups, with 92.1% of diagnoses issued on these groups correct. The fuel correction procedure uses a different sensor-selection rule at individual operating points, testing how each replacement changes the largest prediction error across all six sensors. Fuel correction requires a fault alarm and engine speed and torque close to the training conditions. A fuel reading is replaced only when its replacement reduces the largest prediction error sufficiently more than replacements for the other sensors. Assuming faults in 1% of one-hour scenarios, equally likely sensors and fault types, and equal time at each measured condition, selected fuel corrections reduce mean absolute error in calculated combustion CO2 by 83.8%. On this engine map, learned relationships support complete sensor diagnoses and fuel corrections that improve emissions accounting while preserving unaffected readings.

0.34

Neutrosophic Ensemble Classification for Uncertainty-Aware Bearing Fault Detection: Evidence from Laboratory and Variable-Speed Industrial Benchmarks

cs.LG cs.AI ·2026-09-18 ·Maikel Leyva-Vazquez, Dayron Rumbaut Rangel, Lorenzo Cevallos-Torres, Alexis Matheu Perez
code/data similarity 0.62
Abstract · 18 pages, 5 figures

Machine learning classifiers for bearing fault detection produce scalar confidence scores that conflate confident errors with genuinely ambiguous predictions, and the conventional truth/falsity pair (F = 1 - T) is algebraically redundant by construction. We operationalize a refined neutrosophic decomposition of a Random Forest + XGBoost + Logistic Regression ensemble into four indicators -- T-hat (top-class evidence), F-hat (best-competitor evidence), predictive entropy I1-hat, and decision disagreement I2-hat -- evaluated on two bearing benchmarks (CWRU and JNU, 600-1000 rpm) under a leave-one-condition-out protocol. On CWRU, after correcting a file-to-class mapping error, the ensemble reaches 100.00 percent accuracy on three of four held-out loads (92.27 percent on the fourth), leaving too few errors for uncertainty analysis. On JNU, holding out 1000 rpm, accuracy collapses to 40.64 percent, below a majority-class baseline; Logistic Regression (57.91 percent) generalizes far better than the tree ensembles. I1-hat shows a robust association with error beyond T-hat/F-hat, while I2-hat contributes little; standalone Logistic Regression confidence outperforms the full decomposition, a boundary condition we report honestly. Two further results extend this: fusing a time-domain and a frequency-domain model of the same signal and scoring their Jensen-Shannon divergence beats that model own entropy (AURC 0.29 vs. 0.36 on the standard split; 0.54 vs. 0.73 under a harder single-condition reproduction), the only indicator moving correctly under a CWRU-versus-JNU distributional-shift contrast; and, on CWRU alone, literature-verified bearing fault frequencies, correctly demodulated via the envelope spectrum, separate most fault classes almost perfectly (99.57 percent) using three interpretable features. Code, logs, and figures are released for independent verification.

0.38

Not All Synthetic Data Are Equal: Expert-Committee Audit Screening for Imbalanced Crash-Injury-Severity Prediction in Automated Driving Systems

cs.LG ·2026-08-30 ·Zewei Li, Qiaoqiao Ren, Hang Yang, S. C. Wong, Stergios-Aristoteles Mitoulis +1 more
similarity 0.61
Abstract

Automated driving systems (ADSs) are increasingly operating on public roads, raising safety concerns, yet reliable prediction of crash injury severity remains difficult because crash reports are limited, severe outcomes are rare, and injury classes are highly imbalanced. Existing augmentation methods mainly increase minority-class sample size but rarely assess whether generated samples are credible for safety-critical prediction. This study proposes Expert-Committee Audit Screening (ECAS), a credibility-aware sample acceptance framework for ADS crash injury severity prediction under data imbalance. Using 1,477 incident-level ADS crashes from the National Highway Traffic Safety Administration Standing General Order records, ECAS audits generated minority samples through a real-data-only expert committee based on label support, boundary separation, committee agreement, and local plausibility. Within-class percentile normalization and Pareto non-dominated sorting select accepted samples without manually assigned evidence weights. With a fixed backbone combining normalizing flow augmentation and a Tabular Prior-data Fitted Network (TabPFN) classifier, the best ECAS configuration achieved the highest balanced accuracy, macro-F1, and minor-injury recall among all evidence configurations. Local neighborhood analysis showed that ECAS-accepted samples were better supported by nearby real minority crashes than unscreened retained samples. Shapley additive explanations and partial dependence plots further indicated that lower injury severity classes were mainly associated with crash counterpart and pre-crash movement, whereas moderate-plus injuries were more sensitive to posted speed limit and operating context. These findings support a shift from quantity-oriented augmentation to credibility-aware sample acceptance for ADS safety prediction and risk governance.

0.17

Physical-Variable-Informed Deep Learning for Robust Diagnosis in Industrial Fluid Transport Systems

cs.LG cs.AI ·2026-10-20 ·Yu Sha, Junqi Tao, Dixin Zhou, Horst Stoecker, Domagoj Vnucec +2 more
similarity 0.60
Abstract · 29 pages, 5 figures

Industrial fluid transport systems are monitored through heterogeneous sensing streams, but robust fault diagnosis remains difficult because response signals vary with operating conditions and differ from physical variables in sampling rate, observation duration and data granularity. To address these challenges, we propose a physical-variable-informed deep learning framework (PhyNet) for robust fault diagnosis under real-world sampling heterogeneity. PhyNet treats physical variables as global operating constraints and response signals as local diagnostic evidence. Its physical-to-response cross-attention mechanism connects sequence-level physical-variable tokens with a learnable global response token, enabling interaction without strict temporal alignment. Across four benchmarks, including three real-world cavitation datasets and one public two-phase-flow dataset, PhyNet achieves the best performance across all evaluated metrics. Class-wise analyses, together with attention and masking analyses, show improved discrimination of ambiguous states and indicate that physical variables meaningfully constrain local response interpretation. These findings demonstrate a deployment-oriented approach to robust diagnosis in heterogeneous industrial environments.

0.33

SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment

cs.CL ·2026-09-17 ·Chenxi Wu, Zimu Wang, Haiyang Zhang, Wei Wang, Zhijie Xu
top venuebenchmarkcode/data similarity 0.60
Abstract · Accepted at EMNLP 2026 Industry Track

Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows remains underexplored. We introduce SAFARI (Safety-Aware Functional Automotive Risk Inference), the first industrial benchmark for LLM-assisted automotive Hazard Analysis and Risk Assessment (HARA) under ISO 26262. It contains 3,000 de-identified industrial HARA cases and evaluates two coupled tasks: open-ended hazard analysis and standards-grounded risk assessment. To evaluate open-ended HARA artifacts, we propose the first reference-anchored LLM-as-a-judge protocol with high expert correlation. Experiments with nine frontier LLMs show that models often produce plausible hazard narratives but remain weak at ISO 26262 risk classification, with the best ASIL macro-F1 reaching only 0.261. Chain-of-Thought prompting provides limited benefit and often degrades categorical risk assessment. Error analysis further localizes major failures to scenario-critical context omissions during hazard generation and to controllability misjudgments during risk assessment, indicating where expert oversight should be concentrated. The dataset can be obtained from https://github.com/xixi47520-hash/HARA.

0.47

A Deep Learning-Based Stacking Ensemble Framework for Turbofan Engine Remaining Useful Life Prediction

cs.AI ·2026-08-28 ·Limon Bin Hossain, Md. Salehin Seyam, Md Rashedul Islam, Abdur Rahman, Md Sharifuzzaman
benchmark similarity 0.58
Abstract

This study proposes a two-level stacking ensemble framework for Remaining Useful Life (RUL) prediction of turbofan engines, evaluated on the NASA C-MAPSS benchmark using the FD001 and FD003 subsets. The framework integrates four heterogeneous deep learning base learners: Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), CNN-LSTM, and CNN-GRU, whose out-of-fold predictions are combined by an XGBoost meta-learner to capture complex degradation patterns while mitigating individual model biases. Comprehensive experiments demonstrate that the stacking ensemble achieves superior predictive performance, with Root Mean Square Error (RMSE) of 9.989 and 8.613, Mean Absolute Error (MAE) of 7.081 and 5.195, and R-squared values of 0.899 and 0.906 for FD001 and FD003, respectively. Compared to the best-reported baseline (TCAT: RMSE 11.12 and 11.02), the proposed method achieves RMSE reductions of 10.2 percent and 21.8 percent for FD001 and FD003, respectively. Feature correlation analysis, residual diagnostics, and training convergence curves validate the model's robustness. These findings underscore the efficacy of stacking ensemble methods for prognostics and health management in safety-critical aerospace applications.

0.29