CS Paper Radar

asdex: Automatic Sparse Differentiation in JAX

Adrian Hill, Guillaume Dalle

cs.MS cs.LGmath.NA ·submitted 2026-10-08 17:13 UTC · v2

Abstract

Many tasks in scientific computing and machine learning require the Jacobian or Hessian matrix of a function. Automatic differentiation (AD) computes these derivatives to machine precision, but materializing a dense $m \times n$ Jacobian requires $n$ forward-mode or $m$ reverse-mode AD passes, one per column or row. For a large class of functions, each output depends on only a few inputs, making the derivative matrix sparse. Automatic sparse differentiation (ASD) exploits this structure in four steps: detection of the input-agnostic sparsity pattern, coloring of a graph to group columns or rows that can share an AD pass, compressed differentiation to compute a compressed derivative matrix with one AD pass per color, and finally decompression into the original sparsity pattern. The number of colors, and hence of AD passes, is often independent of the problem dimension: a banded Jacobian with $b$ contiguous bands, for instance, only ever requires $b$ colors, regardless of its size. asdex offers the first standalone ASD toolkit in the popular JAX ecosystem. With $\texttt{asdex.jacobian}$ and $\texttt{asdex.hessian}$, it provides sparse drop-in replacements for $\texttt{jax.jacobian}$ and $\texttt{jax.hessian}$.

Comment
1 table

Closest papers

Nearest neighbours by meaning across the whole index.

Colour me shocked: Exact Molecular Hessians from local MLIPs in O(N) time using sparse differentiation!

cs.CE ·2026-09-21 ·Luca Thiede, Andreas Burger, Alán Aspuru-Guzik
benchmark similarity 0.58
Abstract

The Hessian of the energy with respect to the nuclear positions is indispensable in atomistic modelling. However, constructing this matrix requires $O(N)$ Hessian vector products, traditionally limiting high-accuracy Hessians to small systems. Machine learning interatomic potentials (MLIPs) have accelerated atomistic modelling by providing highly accurate energies and forces at $O(N)$ cost, yet the resulting $O(N^2)$ cost of Hessians remains a practical bottleneck for large systems. Based on the insight that we can derive the sparsity pattern for an MLIP's Hessians in closed form, we show in this paper how to use techniques from sparse automatic differentiation to reduce the cost of a local MLIP's Hessians to a system-size-independent number of Hessian-vector products, yielding overall $O(N)$ total cost without any approximations. We benchmark our approach on a variety of systems ranging from alkane chains to water clusters to $A\beta40$ conformers. Depending on the MLIP configuration, we achieve the linear scaling regime already on relatively small systems, resulting in large runtime reductions between 2$\times$-15$\times$ for these systems. This opens up the possibility of scaling high-accuracy MLIP Hessians to very large systems, such as proteins that were previously inaccessible.

0.06

Coordinate-Free Automatic Differentiation

cs.LG cs.MSmath.DG ·2026-10-08 ·Alex Guerra, Joshua Aduol, Yaqian Tang, Daniel Williams, Ryan P. Adams +1 more
similarity 0.57
Abstract · 34 pages, 18 figures

Automatic differentiation (AD) is a cornerstone of modern machine learning. Since all AD frameworks implicitly fix a basis of Euclidean space to define gradients and Jacobians, they natively process mathematical objects that are arrays of a fixed set of coordinates. This makes it awkward to express objects such as rotations, volume forms, and curvatures that arise, for example, in applications of AI to science. Bespoke workarounds lack reusability. We show that (i) a single, basis-independent derivative operator can represent many geometric differential operations, and (ii) this operator can be expressed entirely in terms of existing AD primitives. This reusable transformation can be composed with JAX batching, compilation, and hardware acceleration. Geometry enters as algebraic data. We experimentally demonstrate its reuse across different manifold operators, its composition through a spacetime inverse-problem pipeline, and its use in optimization over different rotor subgroups.

0.27

GRADSOLVE: fast exact gradients for ODE ensembles on GPUs

cs.MS cs.DCcs.LGmath.NA ·2026-09-02 ·Alessio Spurio Mancini
code/data similarity 0.56
Abstract · 38 pages, 12 figures. GRADSOLVE available at https://github.com/ECLIPSE-AI4Science/gradsolve

Ordinary differential equations (ODEs) underlie models in science and engineering, and many applications need derivatives of their solutions with respect to parameters. Ensembles of independent trajectories suit graphics processing units (GPUs), but current GPU software forces a trade-off: the fastest ensemble solvers cannot be differentiated in reverse mode at the speed they solve, and the solvers built for differentiation solve more slowly. No single tool has yet offered a reverse-mode gradient at the speed of a fused-kernel solve. We present GRADSOLVE, an open-source JAX library for solving and reverse-mode differentiating low-dimensional ODE ensembles on NVIDIA GPUs. It records the steps an adaptive solver accepts and differentiates a fixed-step replay of them; the returned gradient is the exact discrete adjoint of those steps, the same derivative Diffrax returns by default, obtained more cheaply from a fixed-length chain than from an adaptive loop. It targets ensembles differentiated many times against one recorded mesh, keeps Diffrax as a fallback, and supports explicit and Rosenbrock integrators. Used as a solver, GRADSOLVE's forward-only kernel ran 2.8x faster than DiffEqGPU.jl; used for gradients, once a record exists, it computed them 5.6-14.1x faster than Diffrax's checkpointed adjoint at matched forward-state accuracy across three GPU generations, the advantage narrowing on large ensembles and, on stiff systems, down to parity at tight accuracy. GRADSOLVE is released at https://github.com/ECLIPSE-AI4Science/gradsolve.

0.64

Branch-Extended Automatic Differentiation

cs.MS math.OC ·2026-11-05 ·Jan Hückelheim, Jeffrey Larson, Matt Menickelly
similarity 0.56
Abstract

We present branch-extended automatic differentiation (BEAD), a software package that provides information about inactive branches of piecewise-differentiable functions, intended for use in nonsmooth optimization algorithms. BEAD is implemented in Python, building on the popular JAX framework. For supported nonsmooth operations such as abs and max, BEAD identifies alternative branches within tolerance from the given input, and computes gradients of the resulting branches extended to the given input. Such information has been previously used for robust optimization methods, but has required error-prone and tedious manual differentiation, unlike BEAD, which is fully automated for functions written in NumPy-like JAX code. Our experiments show that our approach is as accurate as manually written derivative code, and offers more flexibility in modifying the function and choosing between branch-discovery rules. While we were originally motivated by the manifold sampling algorithm for nonsmooth composite objectives, we also integrate BEAD branch-extension gradients into a nonsmooth BFGS-SQP solver. On two small nonsmooth test problems, the resulting path-aware search resolves failures or substantially reduces the number of evaluations required by the BFGS-SQP solver.

0.29

Block-Sparse Gauss-Newton Optimization of Finite Basis Physics-Informed Neural Networks: Sparse Automatic Differentiation Approach

cs.LG math.NA ·2026-11-09 ·Denis Korolev, Adrian Hill, Jeeeun Lee
similarity 0.55
Abstract

In this work, we develop an efficient Gauss-Newton optimization approach for finite basis physics-informed neural networks (FBPINNs). The key ingredient is the overlapping domain decomposition architecture of FBPINNs, which localizes the influence of each subnetwork and induces block sparsity in both the residual Jacobian and the corresponding Gauss-Newton matrix. We characterize this structure and exploit it through sparse automatic differentiation, using graph coloring to partition structurally orthogonal Jacobian columns or rows. This reduces differentiation costs and memory requirements while preserving all structurally nonzero derivative information. The resulting compressed Jacobian representations allow us to assemble the Gauss-Newton system directly in block-sparse form, enabling the use of sparse direct solvers and preconditioned iterative methods. In this way, the otherwise expensive second-order optimization problem is reduced to scalable sparse linear algebra. We demonstrate the approach on multiscale PDEs, where accurate approximation requires rich neural network parametrizations and large numbers of collocation points. Numerical experiments show that our block-sparse Gauss-Newton method attains high accuracy and substantially accelerates optimization relative to first-order baselines while remaining computationally tractable for large problem sizes.

0.07

Truncated automatic sparse differentiation for machine learning interatomic potentials

cs.LG physics.chem-phcond-mat.mtrl-sci ·2026-09-17 ·Marcel F. Langer, Adrian Hill, Michele Ceriotti
hot directionhighly cited similarity 0.55
Abstract · 19 pages, 5 figures, 4 tables (7 pages main text). Additional information at https://marcel.science/tasd4mlip

Machine learning interatomic potentials (MLIPs) learn the mapping from atomic positions to potential energy. The forces, the negative gradient of this energy, drive molecular dynamics and are readily obtained using automatic differentiation. Higher-order derivatives, most notably the Hessian, describe collective motion and allow the direct prediction of experimental observables, but are considered computationally inaccessible for large systems. We suggest a solution: in physical systems, interactions decay with distance, and most MLIPs build on this locality through message passing up to a finite receptive field. This implies both sparsity of higher-order derivatives and their decay with distance. This structure can be exploited using automatic sparse differentiation (ASD). We explain how to compute the sparsity pattern for MLIP derivatives and demonstrate that, for multiple foundation MLIPs, ASD computes full Hessians of large porous materials exactly, but with modest speedups at best. The larger gains come from truncated ASD: discarding small, but nonzero, Hessian entries between distant atoms yields order-of-magnitude speedups with negligible impact on predicted observables.

0.65

Making Tensor Decoupling More Robust and Less Parametric

cs.LG eess.SP ·2026-10-12 ·Maxime Rochkoulets, Joppe De Jonghe, Mariya Ishteva
code/data similarity 0.54
Abstract

Tensor decoupling algorithms recover compact representations of multivariate functions by combining linear factor matrices with univariate internal nonlinearities. We identify two limitations of recent methods. First, their performance is not invariant to mismatched output or gradient magnitudes, and second, their spline internals require manual tuning of their degrees of freedom. We propose two lightweight fixes. The first is a preprocessing step that rescales the input Jacobian tensor, removing both the bias towards larger magnitude outputs and the ill-conditioning of certain factor updates caused by such a mismatch. The second replaces B-spline internals by penalized splines coupled with generalized cross-validation, making the method less dependent on the user selecting the right number of control points. Finally, we release an efficient open-source JAX implementation of our algorithm.

0.36

KernelPack-JAX: a meshless differentiable GPU framework for node generation and forward and inverse PDE problems

cs.MS math.NA ·2026-10-18 ·Alberto Cattaneo, M. Keith Ballard, Robert M. Kirby, Grady B. Wright, Varun Shankar
similarity 0.53
Abstract

We present KernelPack-JAX, a meshless, differentiable, GPU framework in JAX for solving forward and inverse PDE problems in domains whose boundaries, including internal obstacles, are given only by samples of their smooth pieces. Our node generator is based on a new stacked representation of piecewise-smooth boundaries, in which a parametric partition of unity of each smooth piece induces the supports of a curl-free implicit partition of unity of the whole boundary. This representation keeps the edges and corners of the boundary and classifies every point of space without a global fit. Boundary and interior nodes are then selected by a parallel greedy independent-set algorithm that reproduces sequential Poisson-disk sampling exactly. We discretize elliptic and parabolic problems with radial basis function-generated finite differences (RBF-FD) on stencils selected by a new visibility heuristic to avoid crossing the boundary, with ghost nodes for the boundary conditions. In addition, our framework exposes a GPU sparse direct solver to JAX as a differentiable linear solve, so that Jacobians of time-dependent problems cost a small multiple of a forward solve. We derive error estimates that separate consistency from stability and bound the parameter error by the forward error. We demonstrate high-order convergence for Darcy flow, reaction-diffusion, and advection-diffusion in 2D and 3D domains, including one with an internal obstacle, and show that the recovered parameters largely follow the forward errors. We also present timings for every stage.

0.20