CS Paper Radar

SensOFormer: Foundation Models for Particle Accelerators

Mahindra Rautela, Alexander Scheinker

cs.LG physics.acc-ph ·submitted 2026-11-01 16:25 UTC · v1
code/data

Abstract

Large-scale scientific facilities, such as particle accelerators, are complex systems composed of thousands of interacting components that must operate collectively to support diagnostics and control, and ensure safe, reliable operation. These facilities rely on extensive sensor networks that generate large volumes of heterogeneous, incomplete, and noisy telemetry, often sampled at different rates across subsystems. In this paper, we adopt the foundation-model paradigm to learn general-purpose sensor representations from historical accelerator data through self-supervised pretraining. We introduce SensOFormer, a denoising masked transformer with a Perceiver-style encoder-decoder architecture designed to handle variable sensor sets, accommodate heterogeneous sampling rates, and learn robust representations from noisy measurements. The model is pretrained on multiple particle-accelerator datasets and transferred to downstream tasks, including missing-data imputation, anomaly detection, cavity identification, and fault identification. Through extensive evaluations and comparisons, we demonstrate that a single self-supervised model can learn reusable representations for diverse operational tasks in large-scale scientific facilities. Project page: https://mahindrautela.github.io/sensoformer.

Closest papers

Nearest neighbours by meaning across the whole index.

How Architecture and Training Affect TPC Representations Across Experiments

cs.LG cs.CVnucl-exphysics.ins-det ·2026-08-22 ·Tyler Wheeler, Michelle P. Kuchera, Raghuram Ramanujan, William Sieland, Ryan Krupp +10 more
similarity 0.64
Abstract

Deep-learning efforts have increasingly shifted toward foundation model approaches. In experimental physics, this allows models and learned representations to be reused beyond the experiments in which they were developed. This work evaluates the reusability of representations across experiments and detector systems using probes on frozen encoders. These probes reveal task-relevant structure before downstream adaptation, complementing fine-tuning. Together with random-weight controls, they distinguish contributions from architecture and encoder training that downstream performance alone cannot resolve. Time projection chamber (TPC) data provide a useful testbed because events from TPC systems can be represented as variable-length sparse tensors, while detector geometries, event topologies, and scientific tasks can differ substantially. We investigate whether fixed-dimensional TPC event representations can be reused across classification tasks, experiments, and detector systems. Sparse ResNet and PointNet-style encoders produce 512-dimensional embeddings for four datasets from the GADGET II TPC and AT-TPC. Randomly initialized encoders isolate the contribution from architecture before supervised training. We then train each encoder on a classification task, freeze its parameters, and train a linear or nonlinear probe for each downstream task. We find that this architecture-induced structure remains useful across experiments and detector systems. The randomly initialized PointNet-style representation is highly informative on several tasks. The two architectures organize their embedding spaces differently, but neither exhibits a large, systematic loss of utility cross-detector. These results show that architecture is a major source of task-relevant structure in TPC embeddings and should be treated explicitly when assessing representation learning and developing reusable detector models.

0.11

Self-Supervised Learning for Robust Resonance Mass Regression in Cascade Decays

cs.LG hep-ex ·2026-09-15 ·Ho Fung Tsoi, Alex Yang, Luis Felipe Gutierrez Zagazeta, Shion Chen, Dylan Rankin
similarity 0.64
Abstract · 9th Workshop on Machine Learning and the Physical Sciences (ML4PS 2026)

Reconstructing the mass of a heavy resonance from its decay products with missing energy is one of the central tasks that directly determine the sensitivity in new physics searches at collider experiments. Supervised learning approaches to this problem often struggle to generalize well due to the presence of various systematic uncertainties and distribution shifts. Exhausting all possible variations in the labeled data can be very compute-intensive, while a failure of the model to generalize can corrupt the reconstructed resonance widths that are critical in peak-hunting analyses. In this work, following the foundation model paradigm, we use a self-supervised approach to pre-train a transformer encoder with VICReg to learn an embedding invariant to various corruptions, then fine-tune it for mass regression on a heavy resonance with masses ranging from 2.5 to 6.5 TeV and a SUSY-like cascade decay into an eleven-body final state. We show that the pre-trained model reconstructs sharper resonance peaks and has a more stable performance under various realistic corruptions, compared to a supervised model of the same architecture trained on the same augmented data from scratch.

0.17

Similarity Pairing with Energy Mover's Distance for Self-Supervised Pre-Training at the LHC

cs.LG hep-ex ·2026-09-15 ·Ho Fung Tsoi, Dylan Rankin
similarity 0.63
Abstract · 9th Workshop on Machine Learning and the Physical Sciences (ML4PS 2026)

Many self-supervised methods for training foundation models at the Large Hadron Collider (LHC) rely on data augmentations to encourage the model to embed events into a representation space invariant to certain physical or detector symmetries. A common challenge arises from the large freedom in choosing a proper set of augmentations on which downstream performance depends. The implementation of augmentations involves either modifying existing events, potentially breaking the event fidelity, or simulating more event variants, which is computationally intensive. In this work, we present a data-driven method of pairing events by their similarity via the energy mover's distance (EMD), which measures how similar two events are in terms of the work required to transform one into the other. With this approach, distinct events are sampled and matched by their similarity to serve as views for learning invariance, keeping the physics content of each event intact without handcrafted distortions. We demonstrate this augmentation-free pairing method by pre-training on QCD jets via self-distillation and show that it can yield semantic jet embeddings with downstream discrimination power comparable to or better than an augmentation-based baseline.

0.26

Panda Diplomacy: Foundation Model Pre-training across Particle Imaging Detectors for High Energy and Nuclear Physics

cs.CV hep-ex ·2026-09-01 ·Samuel Young, César Jesús-Valls, Kazuhiro Terao
similarity 0.62
Abstract · 24 pages, 11 figures, preprint

Foundation models are increasingly being pursued in particle and nuclear physics, but existing approaches remain strongly tied to individual experiments through detector-specific architectures or pre-training objectives, limiting their reuse across sensing modalities. We show that a point cloud self-distillation framework yields a substantially more general sensor-level pre-training recipe. We show that the same refined architecture and objective can be independently pre-trained with minimal changes on three qualitatively different detector modalities: liquid argon time projection chamber (LArTPC), collider TPC, and water Cherenkov. Using 1,000 labeled images for downstream task adaptation, Panda V2 matches or exceeds specialized foundation-model baselines trained with orders of magnitude more supervision, matching state-of-the-art particle-clustering performance with 70x fewer labeled events on sPHENIX while substantially improving particle identification, and on LArTPC data matching Panda (arXiv:2512.01324) particle reconstruction with up to 1,000x fewer labels. Beyond reconstruction, simple linear probes reveal physically meaningful latent structure associated with particle causality and track curvature.

0.47

Leveraging Industrial Foundation Models at the Edge of Particle Physics Detectors via Distillation Learning and Hardware Co-design

cs.LG physics.ins-dethep-ex ·2026-09-20 ·Gia Ancone, Qibin Liu, Liangyu Wu, Julia Gonski
similarity 0.62
Abstract · 7 pages, 1 figure, 1 table

Data acquisition (DAQ) systems at future particle physics experiments stand to benefit from the extremes of AI/ML development: large-scale foundation models can enhance the performance of feature extraction algorithms, and small-scale on-detector deployments can enable real-time intelligent data handling. This work provides the first fine-tuning of an industrial foundation model for particle physics DAQ. Starting from the backbone of Google Research's TimesFM (Time Series Foundation Model), we demonstrate fine-tuning on real-time regression tasks for drift chamber trackers and dual-readout calorimeters. Furthermore, the fine-tuned TimesFM model is distilled into a student and co-designed with FPGA implementation to enable these models to run in real-time at future colliders. The fine-tuned distillations meet or exceed the performance of previously published AI/ML solutions for each task. Further, the pipeline of distillation and model compression from TimesFM is generic and can be easily adapted to a variety of 1D waveform tasks across domains.

0.28

Fast and Precise Learned Charged-Particle Trajectory Regression at the Large Hadron Collider

cs.LG hep-exhep-phphysics.data-an ·2026-09-26 ·Jonathan Renusch, Benjamin Huth, Daniel Murnane, Eleni Xochelli, Doğa Elitez +7 more
similarity 0.62
Abstract

We propose a training recipe that treats charged-particle trajectory parameter regression on high-energy physics detector data as a sequence-modeling task. Kalman filters and linearized least-squares fits have been the classical standard approach for this task: they are optimal estimators for sparsely sampled linear-Gaussian data and are commonly used for trajectory parameter regression (fitting). The classical fitting techniques implemented for this domain reach a final precision of one part in $10^5$ through detailed modeling of detector geometry, material, detection effects and precise numerical integration of the equations of motion through the detector's inhomogeneous magnetic field. With this study, we demonstrate that using a bidirectional gated linear recurrent encoder, one is able to reproduce the full precision of classical track fitting techniques. Using a custom kernel, we also achieve significantly higher throughput during GPU inference, compared to classical fitting software running on similarly priced multi-core CPU servers representing typically employed hardware. Such a speedup would lead to considerable cost savings for the pattern recognition at the Large Hadron Collider. To our knowledge, this is the first end-to-end learned track fit to reach the full precision and, at the same time, offer the opportunity to reduce the computing costs.

0.14

"Transforming" LHCb: self-supervised maps of heavy-flavour decays

cs.LG hep-ex ·2026-09-08 ·Marko Stamenkovic, Greg Landsberg
similarity 0.61
Abstract · 31 pages, 14 figures

Decays of beauty and charm hadrons provide sensitive probes of physics beyond the standard model, including decays with invisible particles, in which part of the final state leaves no reconstructed detector signature. The large heavy-flavour data samples recorded by the LHCb experiment at the CERN LHC, together with its precise tracking, displaced vertex reconstruction, and particle identification, make it particularly well suited to learning a map of reconstructed heavy-hadron decay environments directly from data. We propose to bring recent advances in jet flavour tagging at ATLAS and CMS to significantly improve on the performance of the current LHCb taggers and extend them to the reconstruction of heavy-flavour decays with several invisible particles in the final state. To achieve this, we introduce a self-supervised transformer architecture that learns the decay maps without flavour or exclusive-decay labels by inferring masked particle identification information and completing jets from which constituents have been removed. Across five classification tasks in simulated LHCb Open Data, the self-supervised model outperforms an otherwise identical transformer with random weights, and performs comparably to a fully supervised transformer. We achieve a tagging power of about 10\%. In addition, removing constituents from reconstructed exclusive decays also systematically increases the model anomaly score relative to random removals from the same heavy hadrons. We confirm this behaviour directly in 2017 LHCb proton-proton collision Open Data: the score increases for all eight studied heavy-flavour channels, and the signal region response exceeds that in the adjacent sidebands. These studies provide a proof of principle that mapping heavy-flavour decay environments through jets can transform flavour tagging in LHCb and extend the discovery reach for incomplete or otherwise unusual decays.

0.36

Finding and using interpretable latents in a neutrino foundation model with sparse autoencoders

cs.AI astro-ph.HEcs.LGhep-ex ·2026-08-26 ·Raphaël Bonnet-Guerrini, Johann Ioannou-Nikolaides, Inar Timiryasov, Vincenzo Piuri
similarity 0.61
Abstract · Submitted to SciPost Physics

We present a first application of sparse-autoencoder-based mechanistic interpretability to particle physics. Studying a neutrino foundation model pretrained on IceCube data and fine-tuned for direction reconstruction, we identify a validated atlas of physical concepts in the model representation, using a strict validation protocol consisting of held-out tests, matched nuisance controls, and replication across independent dictionary trainings. Causal interventions show that the direction head barely draws on this atlas. Motivated by this underused information, we train an uncertainty head on the same event-level representation to predict the model's angular reconstruction error. Unlike the direction head, it depends causally on quality and brightness features from the atlas. These results suggest that mechanistic interpretability can reveal learned latent physics encoded within a model's internal representation and help design downstream tasks that exploit it.

0.41