CS Paper Radar

Self-Supervised Pretraining for PHM: What Transfers, What Does Not, and What Federating It Costs

José Sanz-Durán, David Solís-Martín

cs.AI cs.CEcs.LGcs.PF ·submitted 2026-10-26 11:50 UTC · v1

Abstract

Self-supervised pretraining has produced capable time-series foundation models, but its value for Prognostics and Health Management (PHM) has not been measured under leakage-controlled conditions. We measure it on a corpus of 47 public PHM datasets, harmonized to a single tensor contract and partitioned so that 11 datasets are withheld from pretraining in full. Transfer turns out to be task-dependent: a compact channel-independent Transformer encoder (801,808 parameters) pretrained with masked patch prediction raises fault-classification macro-F1 from 0.350 to 0.812 on CWRU and from 0.569 to 0.951 on HSG18, yet on CMAPSS remaining-useful-life estimation no pretrained checkpoint reliably improves upon a from-scratch baseline. We pretrain under two regimes---centralized, and simulated cross-silo federated with ten clients grouped by physical domain---holding architecture, objective, source data and optimization budget fixed. Two further findings are negative. At matched budget the centralized encoder leads by 15.5--20.2 macro-F1 points in four of four comparisons, although the comparison is confounded by a learning-rate schedule that favours the centralized arm and rests on a single pretraining run per regime; on RUL the two regimes are indistinguishable. And the intrinsic SSL loss is not a valid selection criterion: the centralized encoder more than halves the federated reconstruction loss without a corresponding gap on RUL, and a server-momentum sweep moves downstream validation macro-F1 by up to 0.29 while the SSL loss varies by 0.021. Transfer also hinges on the backbone learning rate: the same checkpoint yields 0.169 or 0.829 macro-F1 on CWRU depending on a single order of magnitude. We release the corpus pipeline, the protocol and the code.

Closest papers

Nearest neighbours by meaning across the whole index.

Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study

cs.LG ·2026-08-16 ·Fanqi Wang, Weisheng Tang, Hairong Qi
similarity 0.64
Abstract · 42 pages, including supplementary material

Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-horizon predictability as a measure of temporal redundancy: where, when, and under which training conditions recent updates contain information about near-future parameter motion. We combine three complementary probe families, displacement-direction, subspace-residual, and predictor-based probes, with convention-aware, null-calibrated group-level readouts, and apply them to multi-pass vision training on CIFAR and public Pythia pretraining checkpoints. Across both regimes, vector-like tensors such as normalization parameters and biases (auxiliary parameters) exhibit simpler short-horizon dynamics than matrix-like feature-transforming weights (bulk parameters), whose predictable behavior concentrates in localized, time-varying pockets. Agreement within and across probe families, and with independent trajectory diagnostics, indicates that these measurements capture intrinsic trajectory structure, while probe differences distinguish complementary forms of temporal organization. Controlled CIFAR comparisons further show that architecture and training recipe systematically modulate the measured structure. A Pythia-70M case study further exposes a sequence of role-, depth-, and scale-dependent events, including bulk ESA falling below the random sign-agreement level and the emergence and redistribution of predictable qkv pockets across layers. These results position short-horizon predictability as a retrospective, parameter-resolved diagnostic of training dynamics.

0.10

Reliability-Aware Benchmarking of Self-Supervised Learning in Histopathology Under 1-10% Label Regimes

cs.CV cs.LG ·2026-11-04 ·Alicem Koyun, Pranjali Changadeo Pingale, Sachin Mote, Gianluca Pizzuti
publishedbenchmarkcode/data similarity 0.63
Abstract · Oral presentation, CIBB 2026 (21st International Conference on Computational Intelligence Methods for Bioinformatics and Biostatistics, Rome…

Expert annotation in digital pathology is expensive, and low-label regimes remain common in realistic histopathology workflows. Self-supervised learning (SSL) offers a practical way to learn transferable representations from unlabeled pathology images, but performance-only evaluation can hide unreliable confidence behavior under domain shift. We present a reliability-aware benchmark of SimCLR, BYOL, and an MAE-inspired masked-reconstruction baseline on PatchCamelyon (PCam; 96x96) and PANDA (prostate cancer) under 1%, 5%, and 10% labeled regimes with frozen-encoder linear probing. To address common single-run concerns, ResNet-18 results are repeated over 3 splits x 3 seeds (n=9) and ViT-B/16 over 1 split x 3 seeds (n=3) under practical compute constraints, with mean+/-std reporting and per-run logs. Beyond AUROC, accuracy, and F1-score, we report Expected Calibration Error and Brier score, OOD confidence diagnostics for cross-dataset PCam<->PANDA scoring, federated-vs-centralized SimCLR comparisons on PCam, and cross-dataset transfer to colorectal histology tiles. SSL improves label efficiency on the more controlled PCam task, while PANDA's noisier patch-level proxy can remain near coin-flip with higher variance. The study highlights that histopathology SSL should be assessed not only by ranking performance but also by calibration, robustness, and transparent reproducibility diagnostics before clinical interpretation.

0.46

FreqCondNorm: Towards Cross-domain Predictive Maintenance through a Frequency-Conditioned Transformer Foundation Model

cs.AI ·2026-09-17 ·Zaynab Raounak, Camille LHermine, Zhiguo Zeng
similarity 0.63
Abstract

Deep learning predictive maintenance models suffer from poor transferability across machines and operating conditions, especially when labelled data are scarce and signals span five orders of magnitude in sampling frequency (1 Hz to ~100 kHz). We propose FreqCondNorm, a Transformer-based architecture that introduces a FiLM-style frequency-conditioned normalization layer to unify heterogeneous time-series within a single model. The architecture is pretrained on five public predictive maintenance datasets (CWRU, MFPT, UOC18, PRONOSTIA, CMAPSS) using masked auto-encoding and contrastive learning with balanced domain sampling. On fault diagnosis, the model achieves 99.2% accuracy on CWRU (+6.4 pp over CNN) and 82.1% zero-shot accuracy on MFPT, demonstrating strong transfer across sampling frequencies. However, the approach does not improve remaining useful life prediction, suggesting a mismatch between pretraining and RUL objectives that warrants future investigation.

0.22

SLICES: A Benchmark and Evaluation Framework for Self-Supervised Learning on Sparse ICU Time Series

cs.LG ·2026-11-13 ·Hannes Ill, Dimitrios Proios, Leo Anthony Celi, Peter J. Schüffler
top venuebenchmarkcode/datahot direction similarity 0.62
Abstract · Accepted at the Machine Learning for Health (ML4H) Symposium 2026, Proceedings track (PMLR 346). 28 pages, 8 figures, 21 tables. v2: author …

Self-supervised learning (SSL) results for intensive care unit (ICU) time series are hard to compare when data processing, encoders, and adaptation vary together. We present SLICES, a benchmark and evaluation framework for comparing and extending SSL methods under a shared clinical setup. It fixes 24-hour value and observation-mask inputs, a Transformer encoder, patient-level splits, four endpoints, two adaptation protocols, and five matched seeds across MIMIC-IV, eICU-CRD, and their pooled union. Masked autoencoding (MAE), joint-embedding prediction (JEPA), and contrastive learning provide reference implementations. Among these objectives, MAE leads mean AUPRC in all nine frozen classification comparisons, also under converged probes, and in seven of nine fine-tuning comparisons. Fine-tuned MAE leads hospital mortality at full labels and scores higher than a supervised Transformer of the same architecture at every label fraction, and SSL fine-tuning gives the lowest length-of-stay error. Supervised baselines remain strong elsewhere: GRU-D leads 24-hour mortality and XGBoost leads acute kidney injury in all three regimes. Rankings depend on the adaptation protocol, the comparator set, and whether the probe loss matches the metric. A two-seed MIMIC-IV rerun with the maintained code stays within two across-seed SDs of the reference in all 48 downstream runs. The code repository supports new objectives, shared downstream evaluation, and regeneration of the reference results, so new SSL methods can be compared with these references under the same conditions. Generalization to unseen hospitals is not tested here and is the natural next extension.

0.59

Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation

cs.CV cs.AI ·2026-10-13 ·Jiaqi Tang, Yu Wang, Yang Liu, Jing Tao, Fandong Zhang +5 more
benchmark similarity 0.62
Abstract

Self-supervised pretraining has expanded the pool of 3D medical foundation models, but published evaluations use different datasets, endpoints and fine-tuning protocols. Models also differ in architecture, pretraining objective and data, while downstream targets range from voxel-level masks to patient-level diagnoses and outcomes. As a result, published findings provide little directly comparable evidence for choosing a model for a new task. Direct comparison of adaptation performance requires fine-tuning candidates on labelled target data, at substantial computational cost. We benchmark 23 candidates across the same 14 segmentation, classification and regression tasks spanning multiple anatomies and modalities. No candidate ranked among the top three across all segmentation tasks or across all classification and regression tasks; a convolutional encoder could outperform a transformer trained with the same objective on one task but underperform it on another. We benchmark transferability estimators against the fine-tuned rankings and find that their agreement varies across dense segmentation, classification and regression. InsightRank uses task-appropriate frozen-feature statistics and architecture-aware pairwise comparisons to rank candidates before fine-tuning. Across the 23-model pool, mean Kendall tau-b was 0.431 for segmentation and 0.352 for classification and regression. The segmentation Top 3 contained a task winner on four of nine tasks; the image-level Top 3 contained none on the five tasks, despite better full-ranking agreement than the matched conventional estimators. A three-model shortlist reduced recorded extraction-plus-training job time by 85.8% across seven segmentation tasks. Together, the benchmark provides a common reference for model selection and InsightRank offers a lower-cost way to prioritize fine-tuning, with shortlist performance that differs by target type.

0.20

Parameter-Efficient Self-Supervised Adaptation for EEG-FM under Fixed Computational Budgets

cs.LG cs.AI ·2026-08-25 ·Meghal Dani, Stefanie Liebe
top venuecode/data similarity 0.62
Abstract

EEG foundation models pretrained via self-supervised learning promise transferable representations, but their generalization remains limited, especially across diverse clinical datasets. Full fine-tuning is impractical for resource-constrained clinical settings due to high computational requirements. In this work, we investigate whether parameter-efficient self-supervised adaptation, updating only 9% of parameters suffices to align representations to target tasks. We evaluate our method on two state-of-the-art models with different pretraining objectives: BIOT (contrastive) and CBraMod (masked reconstruction), and evaluate on three clinical EEG datasets for abnormality detection (TUAB), event classification (TUEV), and seizure detection (CHB-MIT) under both in-distribution and out-of-distribution conditions. SSL adaptation yields consistent gains over linear probing, up to 20x AUCPR. Under a fixed compute budget, peak performance requires only 20--50% of available unlabeled data. Critically, when total window count is fixed, performance remains invariant to patient count, suggesting that performance is dependent on overall temporal window diversity only. Our findings demonstrate that parameter-efficient adaptation enables effective deployment of EEG Foundation models (EEG-FM) with minimal computational overhead and data collection burden. Code available at: https://github.com/c3n-group/efficient-eeg-adapt

0.54

Tabby: An Open Pretraining Recipe for Time Series Foundation Models

cs.LG ·2026-09-12 ·Shifeng Xie, Bahaeddine Abdessalem, Zehao Xiao, Youssef Attia El Hili, Ambroise Odonnat +7 more
benchmarkcode/data similarity 0.61
Abstract · 43 pages, 3 figures, 32 tables. Technical report

In this report, we release Tabby, a long context probabilistic time series foundation model, together with a complete and open recipe of how it was built. Tabby adopts an encoder-only patch Transformer architecture and concentrates the contributions on the data and the training procedure. The pretraining corpus combines an extended real-world collection, GIFT-Eval-Pretrain+ and BLAST, with synthetic data from KernelSynth and CauKerV2, an online generator that composes temporal dynamics through randomly sampled structural causal models. Training couples a progressive convergence schedule, which yields reusable intermediate checkpoints, with a deep quantile supervision objective for intermediate layers. The resulting 145M parameter backbone supports contexts of up to 8,192 observations and serves forecasting, classification, and anomaly detection, while a prompt-tuning module further improves in-distribution forecasting performance with the pretrained weights frozen. Tabby achieves competitive zero-shot forecasting performance on GIFT-Eval and the out-of-distribution TIME benchmark, while the same pretrained backbone also supports classification on the UCR Archive and zero-shot anomaly detection on TSB-AD-U. We release training pipeline and model as open source at huawei-noah/trustworthyAI.

0.44

Self-supervised Pre-training Helps Retinal Disease Progression Modelling Most When Data Is Scarce

cs.CV ·2026-09-11 ·Ifeoma Veronica Nwabufo, Julius Gervelmeyer, Sarah Müller, Philipp Berens
similarity 0.61
Abstract

Modelling how a disease progresses over time requires longitudinal imaging cohorts, which are scarce and small, whereas cross-sectional data -- one image per participant -- is abundant. Self-supervised pre-training on such data offers a way to bridge this gap, but it is unclear which strategy best supports progression modelling, or how that answer depends on the amount of labelled longitudinal data. We study this for age-related macular degeneration (AMD), pre-training encoders on the large cross-sectional NAKO cohort and predicting time to late AMD on the longitudinal AREDS dataset. We compare in-house self-supervised encoders against a general-purpose (DINOv2) and a domain-specific (RETFound) foundation model, across contrastive, masked-autoencoding, and self-distillation objectives, under frozen and fine-tuned protocols, and across labelled training sets from 100 to 32,250 examples. Which model performs best depends on how the encoder is used. When the encoder is frozen and labels are few -- the regime typical of longitudinal cohorts -- pre-trained representations reach clinically reasonable discrimination from a few hundred labelled samples, while models trained from scratch do not; this advantage fades under fine-tuning. Transfer is governed by the self-supervision objective rather than corpus scale or domain match, so that an encoder pre-trained on a modest cross-sectional cohort matches or exceeds a far larger in-domain foundation model. Together, these results offer a practical recipe for building progression models where longitudinal data is scarce: a frozen self-supervised encoder with a lightweight survival head.

0.15