CS Paper Radar

A method for multimodal analysis of TAIGA experiment data using essential features

Alexander Kryukov, Julia Dubenskaya, Elena Fedotova, Elizaveta Gres, Stanislav Polyakov, Eugene Postnikov, Alexander Razumov, Pavel Volchugov, Dmitry Zhurov

cs.LG astro-ph.IMastro-ph.HE ·submitted 2026-10-06 18:44 UTC · v1
unusual

Abstract

The aim of processing and analyzing experimental data from physical experiments is to obtain physically significant information about the phenomenon under study. This goal is achieved by multi-stage processing of experimental data, during which noise associated with measurements is suppressed and the dimensionality of the input data is reduced. In this paper, we propose a new method based on the use of neural networks such as autoencoders to extract essential features. The special value of the proposed approach lies in the possibility of its application to the analysis of multimodal data received simultaneously from several installations. We will apply this approach to a multimodal data (MMD) of the experiment TAIGA. Currently, the analysis of the MMD is carried out independently for each installation separately. Therefore, the development of methods for the joint analysis of MMD from TAIGA-type installations is an urgent task in cosmic ray physics and gamma-ray astronomy. Based on Monte Carlo simulation, it is shown that the proposed method allows for effective MMD analysis. It can also be used for MMD analysis at other experimental complexes.

Closest papers

Nearest neighbours by meaning across the whole index.

Deep learning methods for gamma event selection in TAIGA-IACT image analysis in stereo-mode

cs.LG astro-ph.IMastro-ph.HE ·2026-10-12 ·E. O. Gres, A. P. Kryukov, P. A. Volchigov, A. P. Demichev, J. J. Dubenskaya +4 more
published similarity 0.60
Abstract · 10 pages, 4 figures, Proceedings of The 10th International Conference in Deep Learning in Computational Physics, July 6-8, 2026, Moscow, Rus…

In astrophysical complex TAIGA there are four Imaging Atmospheric Cherenkov telescopes (IACTs), located at a distance of 300-500 m from each other, due to which extensive air showers (EAS) generated by cosmic and gamma radiation are recorded by a single telescope or by multiple telescopes simultaneously (stereo-mode). The analysis of EAS images from two or more Cherenkov telescopes together is of particular interest, since for this approach the reconstruction of physical parameters of the primary gamma ray is much more accurate than in mono-observations. However, the gamma-ray event reconstruction strongly depends on the quality of the hadron background suppression of the classifier, therefore, the gamma event selection stage is one of the important ones in the further analysis of IACT data. We propose and study several neural network methods for gamma event selection in joint TAIGA-IACT image analysis. Neural network methods we suggested involves a single-stage analysis of IACT images from two telescopes, while the second method considers the formation of autoencoder-derived essential features that contain the EAS physical parameters, including the type of primary particle.

0.33

SWXAI: A Unified Multimodal Explainability Library for Space Weather Forecasting and Heliophysics

cs.LG astro-ph.SRphysics.space-ph ·2026-11-07 ·Benedict Antonio Mervyn, Anli Ji, Manolis K. Georgoulis, Berkay Aydin
top venuebenchmarkcode/data similarity 0.49
Abstract · 6 pages, 5 figures. Accepted at CIKM 2026 (Resource Track), Rome, Italy, November 7-11, 2026. Code: https://bitbucket.org/gsudmlab/swxai Doc…

Space weather forecasting and heliophysics increasingly rely on machine learning models that process heterogeneous inputs such as multi-viewpoint multi-wavelength solar images and rasters, multivariate time series, spatial annotations, and tabular event-level data. However, explainability workflows remain fragmented across libraries, model interfaces, and output formats, limiting reuse and reproducibility. We present SWXAI, an open-source library that unifies explainable AI (XAI) workflows for image, time-series, and tabular data in space weather forecasting and heliophysics. SWXAI provides a common, extensible interface across commonly used vision, time-series, and tabular learning models, with validation, adaptation, and persistent explanation objects. It also includes domain-grounded examples, reproducible notebooks, tests, and documentation, enabling users to inspect outputs, swap models or explainers, and extend workflows to new heliophysics datasets. SWXAI reduces modality-specific engineering overhead and supports comparable, reusable explanations across heterogeneous scientific machine learning pipelines. A benchmark against direct backend implementations shows that SWXAI substantially reduces the code users write to produce an explanation, with no practical runtime cost.

0.45

OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora

cs.CL ·2026-08-26 ·Hao Chen, Yumin Lin, Nadila Yushanjiang, Xin Lin, Min Zhang
top venuebenchmarkcode/data similarity 0.47
Abstract · Accepted to Findings of EMNLP 2026

Multimodal Large Language Models (MLLMs) have demonstrated strong abilities in solving diverse visual and textual reasoning tasks. However, their development in the physics domain is significantly hindered by the lack of a comprehensive benchmark. To fill this gap, we introduce OmniPhys, a large-scale benchmark for multimodal physics understanding and reasoning, covering middle school through university-level problems from Chinese Educational Corpora. OmniPhys consists of 15,246 questions and 19,850 images, accompanied by detailed annotations that support fine-grained analysis of reasoning processes and knowledge usage. Beyond conventional evaluation, OmniPhys is a benchmark that systematically evaluates multimodal outputs in the physics domain, including models' ability to generate structured physics diagrams, which constitute a fundamental component of authentic physics problem solving. Extensive evaluations reveal critical gaps in the capabilities of current MLLMs, especially in complex reasoning and visual generation. To address this, we release OmniPhys to serve as a foundational resource for advancing multimodal intelligence in physics and scientific domains. Codes and data are available at https://github.com/ECNU-RAIL/OmniPhys-EMNLP2026.

0.36

Multimodal Model Diffing for Feature Discovery and Control

cs.CV cs.AIcs.CLcs.LG ·2026-08-10 ·Hunar Batra, Lachin Naghashyar, Ashkan Khakzar, Philip Torr, Christian Schroeder de Witt +2 more
top venue similarity 0.46
Abstract · Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026)

Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control. We introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses: (i) feature isolation, by diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training; (ii) task-specific feature detection, via per-token contrastive firing analysis that isolates causal features; and (iii) feature-level control, by causally removing or steering the discovered feature directions. We train multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding, multimodal safety, and OCR. MMDiff discovers sparse, causally specific features whose removal selectively degrades target behaviors by an average of 12% on spatial tasks and 17% on OCR, and reduces attack success rate by 24% on multimodal safety attacks, with no impact on VQA performance. Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average over a standard single-layer steering baseline. These results show that multimodal SAEs can serve not only as interpretability tools, but as mechanisms for auditing, steering, and controlling MLLMs behavior toward safer and more capable generations.

0.31

UniData: Universal Multimodal Instruction Generation Pipeline

cs.AI cs.CL ·2026-10-08 ·Jiaqi Tang, Yi-Feng Wu, Yuting Zhang, Hao Lu, Bowen Fu +10 more
top venue similarity 0.46
Abstract · Accepted by EMNLP 2026 Findings

Multimodal Large Language Models (MLLMs) are increasingly being applied in a wider range of real-world scenarios. However, due to the substantial labor cost, creating high-quality multimodal instruction datasets for MLLMs remains a significant challenge. Although some methods propose to generate instruction data, they often face limitations in modality support and struggle with generating multi-round instructions. To address these problems, we introduce UniData, a universal instruction generation pipeline, to transform simple user requirements into multi-round, multimodal instructions. Specifically, UniData first expands user requirements into multiple diverse events. Using these events, UniData then integrates an any-to-any large model for multimodal instruction generation. Finally, UniData enhances data quality by correcting irrelevant and redundant inference flow, leveraging correlations between instruction rounds. To train this pipeline, we also build UniDataset, a dataset comprising 20,000 entries across nine modalities for improved multimodal generation. Our experiments demonstrate that UniData achieves SOTA performance in data quality and can also enhance the understanding and generation capabilities of other multimodal models.

0.34

LingLan: An Advancing Traditional Chinese Medicine Diagnosis LLM with Multimodal Data

cs.AI ·2026-09-22 ·Zheng Chen, Zhicheng Du, Haoxuan Li, Yingshan Liang, Peiwu Qin
similarity 0.45
Abstract · 6 pages, 5 figures

Though artificial intelligence (AI) increasingly transforms modern medicine, its integration into Traditional Chinese Medicine (TCM) has been relatively slow, primarily due to TCM's reliance on holistic, subjective diagnostic methods---namely Inspection, Auscultation and Olfaction, Inquiry, and Palpation(I-AOI-P)---which are difficult to align with quantitative, standardized medical systems. In this work, we introduce a Unification Framework for Multimodal Data (UFMD), which automatically processes tongue and pulse images into structured, clinically standard descriptions, integrating multi-source diagnostic information into a unified digital record of I-AOI-P process. Building on this structured data, we create LingLan-14B, a TCM-specific large language model fine-tuned via supervised learning to emulate the diagnostic logic and workflow of I-AOI-P process. Experimental results show that our method significantly enhances diagnostic accuracy, achieving a relative improvement of 103.5% over the baseline (62.72% vs. 30.82%) and reaching an F1-score of up to 82%.

0.13

Information fusion and machine learning for sensitivity analysis using physics knowledge and experimental data

cs.CE cs.LGstat.MEstat.ML ·2026-08-18 ·Berkcan Kapusuzoglu, Sankaran Mahadevan
publishedcode/datahighly cited similarity 0.45
Abstract · Reliability Engineering & System Safety

When computational models (either physics-based or data-driven) are used for the sensitivity analysis of engineering systems, the sensitivity estimate is affected by the accuracy and uncertainty of the model. This paper considers global sensitivity analysis (GSA) for situations where both a physics-based model and experimental observations are available, and investigates physics-informed machine learning strategies to effectively combine the two sources of information in order to maximize the accuracy of the sensitivity estimate. Two representative machine learning (ML) techniques are considered, namely, deep neural networks (DNN) and Gaussian process (GP) modeling, and two strategies for incorporating physics knowledge within these techniques are investigated, namely: (i) incorporating loss functions in the ML models to enforce physics constraints, and (ii) pre-training and updating the ML model using simulation and experimental data respectively. Four different models are built for each type (DNN and GP), and the uncertainties in these models are included in the Sobol indices computation. The DNN-based models, with many degrees of freedom in terms of model parameters and training options, are found to result in smaller bounds on the sensitivity estimates when compared to the GP-based models. The proposed methods are illustrated for additive manufacturing and lake temperature modeling examples.

0.62

Track finding and particle identification performance with particle physics foundation models

cs.LG physics.data-anhep-exnucl-ex ·2026-10-22 ·Shuhang Li, Haiwang Yu, Gabor Galgoczi, Yeonju Go, Yi Huang +7 more
similarity 0.44
Abstract

Modern high energy nuclear and particle physics experiments produce large amounts of sparse data that must be reduced significantly in size before performing physics analysis. Recently, there has been an effort to evaluate Foundation Models as a potential tool for performing data analysis. These models have been shown to learn features in high dimensional datasets based on the multi-modal data they are trained on. To show their effectiveness at performing complex pattern recognition tasks, such as reconstructing charged particles in a magnetic field, we evaluated the performance of a Foundation Model based reconstruction on several downstream tasks with a dataset composed of spacepoints in a Time Projection Chamber. We show that the performance of the Foundation Model based data analysis significantly improves upon that of traditional methods, and furthermore can be adapted to learn additional tasks that are not necessarily related to the initial training objective.

0.20