CS Paper Radar

Does Transolver really need a Transformer?

Shizheng Wen, Siddhartha Mishra

cs.LG ·submitted 2026-09-26 12:10 UTC · v1

Abstract

The widely used Transolver family of neural operators is based on physics-attention, which softly assigns the points of an unstructured mesh to a small number of slices, applies self-attention among the resulting tokens, and broadcasts the result back to the points. We provide a comprehensive empirical and theoretical analysis to elucidate the mechanisms which are responsible for model performance. To this end, we perform careful ablations on a challenging suite of nine 3D fluid dynamics benchmarks to find that replacing token attention with a constant linear map does not affect the accuracy. Thus, Transolver does not need a Transformer at all. However, removing the global mixing (slicing/deslicing) or doing it only once leads to performance collapse. We leverage the theory of averaging neural operators to explain and corroborate our findings by showing that just slicing/deslicing, in conjunction with pointwise MLPs, already suffices for universal approximation of continuous operators and attention is redundant in this context. Finally, we provide a novel FlashAttention-style efficient implementation of the key slicing/deslicing module of Transolver. This flashslice kernel streams slice and deslice over the points without materializing the heavy slice-weight tensor, while reproducing the best available implementation to floating point error. At the same time, it leads to very significant memory and compute savings, particularly at large slice counts.

Closest papers

Nearest neighbours by meaning across the whole index.

Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations

cs.AI math-phcs.LGmath.DS ·2026-09-30 ·Ming Zhong, Antonio Colanera, Gianluigi Rozza, Zhenya Yan
similarity 0.66
Abstract · 30 pages, 9 figures

Traditional simulations of parametric partial differential equations (PDEs) rely on repetitive computations for each parameter, which makes high-fidelity design impractical. Neural operators address this issue by learning solution operators, accelerating parameter-space mapping by orders of magnitude. Recent Transformer-based neural operators attempt to capture global dependencies, but often at the cost of quadratic attention complexity. Transolver resolves this problem by projecting physical states into a reduced slice space for attention computation. Although fast, this projection sacrifices fine spatial information. Moreover, by operating in this reduced space with shared weights across attention heads, it may constrain the model's flexibility, thereby limiting its capacity to capture complex phenomena. To address these issues, we propose the Cluster Attention Neural Operator (CANO), which reformulates attention via a novel cross-attention mechanism that dynamically clusters queries while preserving full-resolution keys and values. This avoids slice compression loss and removes weight-sharing limits. At the same time, the model remains fast without losing global interactions. Empirically, CANO achieves state-of-the-art performance across canonical PDE benchmarks, covering fluid and solid dynamics (e.g., Navier-Stokes, Airfoil, Plasticity), irregular unstructured geometries (e.g., Pipe Turbulence, Composites), and long-term temporal rollouts. Across solid deformation and turbulent flow benchmarks, CANO achieves lower errors than baselines and exhibits strong geometric adaptability and temporal consistency.

0.26

Neural Operator Splitting for Fluid Dynamics in Complex Geometry

cs.LG physics.flu-dyn ·2026-10-20 ·Jianxiang Li, Xi Deng, Hong Chul Nam, Anima Anandkumar
similarity 0.63
Abstract

Solving dynamical systems in complex and diverse geometries remains challenging for both numerical solvers and learning-based surrogate models. We propose Neural Operator Splitting, a framework for learning fluid dynamics while preserving the physical structure of fractional-step numerical schemes. Instead of learning the full-step evolution monolithically, we decompose each time step into physically interpretable steps, and learn each operator separately using intermediate states from the underlying numerical solver. Guided by the linearity properties of the fractional steps, we introduce a conditional-linear Fourier neural operator (CL-FNO) whose linear state mappings are modulated by nonlinear conditioning on inputs such as boundary geometry. We further introduce two complementary strategies, symmetry sampling and corrective cycling, to mitigate error accumulation during autoregressive rollout. Experiments on 2D and 3D fluid flows with varying geometries and initial conditions show that our full model reduces mean relative $L^2$ rollout error by 68\% in 2D and 40\% in 3D compared with a single full-step neural operator. At comparable error levels, neural-operator inference is approximately $10^3$ times faster per step than the Monte Carlo solver.

0.24

Disentangling Representation Evolution in Transformers through Directional Decomposition

cs.CL cs.LG ·2026-09-14 ·Shwai He, Haichao Zhang, Shen Yan
top venuecode/data similarity 0.62
Abstract · Findings of EMNLP 2026

Transformer representations evolve through learned additive transformations that either preserve their current direction or redirect it. We study this evolution as a functional geometry, decomposing learned updates into parallel and perpendicular components. Across pretrained models, we find substantial parallel components beyond the residual identity path. We then apply the decomposition in two spaces: to attention and MLP updates relative to the hidden state, and to attention value aggregation relative to the current token's value. Targeted edits reveal a strongly space-dependent asymmetry: exclude-self value-space parallel manipulation is markedly more robust than residual-space and perpendicular counterparts, preserving the direct self message while scaling only the non-self aggregate. The same decomposition gives a component-resolved description of compression-induced update error: perpendicular error separates compression methods more clearly than parallel error. Extensive experiments further demonstrate that full-aggregate parallel suppression during from-scratch pretraining lowers validation-loss trajectories and improves downstream averages, with the value-space variant strongest. Together, these results connect representation geometry to editing robustness, compression diagnosis, and training-time intervention. Code is available in the \href{https://github.com/Shwai-He/Transformer-Geometry}{project repository}.

0.58

SimpFlow: Transferable Pretraining for Fluid Dynamics via Simplified Physics

cs.CE ·2026-11-09 ·Ruijie Li, Tianbai Xiao
benchmark similarity 0.62
Abstract · 29 pages, 20 figures, 19 tables

Neural surrogates enable rapid fluid simulation, but learning across diverse three-dimensional (3D) geometries requires costly computational fluid dynamics (CFD) labels. We introduce SimpFlow, a pretraining approach that generates supervision from simplified physics on unlabeled geometry libraries. A source-panel potential-flow solver generates velocity and pressure fields for approximately 9,000 3D geometries, capturing geometry-induced acceleration, stagnation, and suction while omitting viscosity and separated wakes. The solver imports only a surface mesh and reuses precomputed velocity fields across inflow directions, enabling inexpensive label generation. In our benchmark, amortized labeling takes 0.111 s per example using four CPU threads. We pretrain a neural surrogate on these labels and fine-tune it on independent CFD datasets. Across five flow tasks, SimpFlow improves over training from scratch in all 24 task-budget settings. With fine-tuning fixed at 200 epochs, interpolation indicates that SimpFlow requires 35-68% fewer CFD training cases to match the error of scratch models trained at each task's largest data budget. These results demonstrate that simplified physical solutions can provide transferable supervision, offering a practical way to reduce the high-fidelity data requirements of fluid surrogates.

0.13

ISLA: A Coordinate-Free, Measure-Weighted Transformer for Boundary-Driven PDEs

cs.LG cs.CEmath-phphysics.flu-dyn ·2026-11-12 ·Peter Sharpe
similarity 0.61
Abstract · 49 pages, 14 figures, 27 tables. Companion paper to "AeroZoo: A Benchmark of Well-Posed Flow Problems for Foundation Models of Aerodynamics"…

Symmetry is a foundational principle of physics: the behavior of a PDE-governed system does not depend on the coordinate frame, unit system, or mesh used to write it down, and neither should a surrogate model. Yet today's leading neural surrogates lack these invariances and fail under simple transformations of the same problem: for models trained on PDEs of cars in one pose (i.e., translation and rotation), rotating the full PDE raises the surface-pressure error from 4% to more than 100%, and re-sampling its boundary raises it to about 30%. We introduce ISLA (Invariant Slice Attention), a slice-attention transformer for steady, boundary-driven PDEs. Because the boundary determines the interior solution, ISLA encodes the posed boundary once into learned slices and evaluates arbitrary interior points independently through a tree-accelerated learned boundary kernel. Inputs are nondimensionalized in scales computed from the boundary, sums over boundary cells are area-weighted quadratures that converge under mesh refinement, and geometry enters only as distances and relative angles to equivariant slice anchors, making the model equivariant to rigid motions and reflections up to optional token subsampling and tree clustering. On AeroZoo, 2,816 3D flows spanning diverse geometries, poses, physical scales, and boundary conditions, ISLA's mean reference-scaled velocity and pressure errors are at least 1.9 and 2.2 times lower than any baseline's. On full AirfRANS, ISLA is second only to an explicit boundary-integral network (GLOBE), with errors at least 1.6 times lower than Transolver's and GeoTransolver's in every field.

0.38

CIS-Transolver: A Cross-layer Incremental-State Neural Operator for Fine-Scale Automotive Aerodynamic Flow Prediction

cs.LG cs.AI ·2026-10-19 ·Jianglong Liu, Zhicheng Zhu, Jia Hao, Liangyue Jia, Wenxuan He
similarity 0.61
Abstract

Neural operators offer an efficient data-driven approach to predicting complex automotive aerodynamic flows. However, spectral bias limits their ability to reconstruct fine-scale flow variations around geometric features such as side mirrors and wheel arches. Reliable assessment of local geometric changes therefore requires models that capture fine spatial detail together with global flow interactions. Inspired by iterative refinement and state evolution, we propose CIS-Transolver, a cross-layer incremental-state neural operator that integrates local multiscale information into global flow modeling. The method extracts local feature variations and encodes them as zeroth- to second-order geometric moments. These encodings are combined with information from earlier layers to form a cross-layer memory. Memory-derived corrections enter the value branch of Physics-Attention, allowing local details to inform global physical-state interactions throughout the network. We systematically evaluate CIS-Transolver on three public automotive aerodynamics benchmarks, ShapeNet-Car, AhmedML, and DrivAerNet++, covering surface pressure fields and volumetric pressure and velocity fields. CIS-Transolver achieves the lowest mean relative L2 error among the evaluated methods for all five prediction metrics across the three benchmarks. Relative error reductions average 8.72% compared with the best baseline for each metric. Also achieves the lowest high-frequency spectral errors across all five metrics, with an average relative reduction of 16.06% compared with the best baseline. These results support cross-layer reuse of local information for fine-scale automotive flow prediction.

0.21

Flux-form spatiotemporal neural operators for coarse-grained dynamics of multiscale PDEs

cs.LG nlin.CDmath.APphysics.comp-ph ·2026-08-11 ·Junfeng Chen
similarity 0.60
Abstract

We study data-driven prediction of coarse-grained dynamics in multiscale PDE systems. Adopting a closure-free operator-learning viewpoint, we apply a linear coarse-graining map and learn a surrogate evolution operator for the resolved field directly from filtered high-fidelity trajectories. Motivated by the Mori-Zwanzig formalism, we propose a spatiotemporal neural operator mapping a resolved history slab on $Ω\times[-T_{\mathrm{in}},0]$ to a resolved future slab on $Ω\times[0,T_{\mathrm{out}}]$. Spatial mixing uses Fourier convolution, while temporal mixing uses a causal kernel operator with position-attention weights on time lags. This causal temporal operator encodes finite-memory effects in the resolved dynamics while preserving the directionality of the history-to-future map. To improve rollout robustness and suppress nonconservative artifacts, we embed a flux-form inductive bias by parameterizing the windowed update in explicit divergence form. We also provide a data-driven guideline for selecting the memory length $T_{\mathrm{in}}$ via the decorrelation time of a closure-injection diagnostic computed from filtered trajectories. We validate on the coarse-grained viscous Burgers' equation, the Kuramoto-Sivashinsky equation, and two-dimensional turbulent flows, obtaining stable autoregressive rollouts with improved long-horizon accuracy and statistical fidelity.

0.38

A Variational Optimal Transport Operator on Incompressible Flow

cs.LG cs.GR ·2026-09-12 ·Jinjin He, Shenyifan Lu, Sinan Wang, Zhiqi Li, Duowen Chen +1 more
similarity 0.60
Abstract

We present the Variational Incompressible Optimal Transport (VIOT) operator, a generative neural operator for amortized incompressible density transport. Given a new source-target density pair, VIOT predicts a divergence-free velocity field and generates the full transport trajectory by feed-forward inference, replacing the hour-scale per-pair optimization used by adjoint fluid solvers and differentiable simulation baselines. The system consists of three components: a stream-function or vector-potential representation that enforces incompressibility by construction, a regularized incompressible transport objective that balances endpoint accuracy and flow smoothness, and a Fourier Neural Operator backbone that amortizes the solve across new pairs and grid resolutions. Together, these components make incompressible transport a reusable neural operator that facilitates various transport processes. Further, the generative capability extends beyond the training distribution, with VIOT producing incompressible transports for user-drawn source-target pairs in a real-time interactive system. We demonstrate VIOT on 2D and 3D density-transport benchmarks. Both 2D and 3D rollouts complete in seconds per pair, while per-instance baselines in our 2D comparisons optimize each new pair from scratch and require on the order of an hour, a roughly $10^4\times$ online speedup.

0.22