CS Paper Radar

Branch-Extended Automatic Differentiation

Jan Hückelheim, Jeffrey Larson, Matt Menickelly

cs.MS math.OC ·submitted 2026-11-05 19:35 UTC · v1

Abstract

We present branch-extended automatic differentiation (BEAD), a software package that provides information about inactive branches of piecewise-differentiable functions, intended for use in nonsmooth optimization algorithms. BEAD is implemented in Python, building on the popular JAX framework. For supported nonsmooth operations such as abs and max, BEAD identifies alternative branches within tolerance from the given input, and computes gradients of the resulting branches extended to the given input. Such information has been previously used for robust optimization methods, but has required error-prone and tedious manual differentiation, unlike BEAD, which is fully automated for functions written in NumPy-like JAX code. Our experiments show that our approach is as accurate as manually written derivative code, and offers more flexibility in modifying the function and choosing between branch-discovery rules. While we were originally motivated by the manifold sampling algorithm for nonsmooth composite objectives, we also integrate BEAD branch-extension gradients into a nonsmooth BFGS-SQP solver. On two small nonsmooth test problems, the resulting path-aware search resolves failures or substantially reduces the number of evaluations required by the BFGS-SQP solver.

Closest papers

Nearest neighbours by meaning across the whole index.

Automatic Certified Differentiation in Lean

cs.SC cs.MS ·2026-10-26 ·Ramy Shahin
similarity 0.57
Abstract · DATAMOD preprint

Automatic differentiation (the automatic computation of numerical function derivatives) has many applications, especially in scientific computing and machine learning. Several approaches to automate the calculation of derivatives have been presented. Common challenges facing those approaches include (1) having to define derivative rules for different classes of functions, (2) having to manipulate program symbols to understand the semantics of the function being differentiated, and construct the derivative function, and (3), more importantly, ensuring that the calculated derivative is correct. In this paper, we present an approach for automatic \emph{certified} differentiation using the Lean proof assistant. By certified, we mean a correctness certificate is computed together with the derivative function and a domain on which the derivative is guaranteed to be valid. Our approach leverages existing derivative theorems included in the Mathlib Lean library instead of having to define our own rules. We also utilize Lean metaprogramming to manipulate the symbolic structure of the input function, and to construct the derivative function. The correctness certificate is computed step-wise as we apply Mathlib theorems. To make the search efficient, we index the derivative theorems by the function they differentiate. We evaluate our approach on 500 randomly generated functions: with the index, all of them are differentiated, with a median time of 0.18\,s per function, 8.6 times faster than without the index.

0.32

Coordinate-Free Automatic Differentiation

cs.LG cs.MSmath.DG ·2026-10-08 ·Alex Guerra, Joshua Aduol, Yaqian Tang, Daniel Williams, Ryan P. Adams +1 more
similarity 0.56
Abstract · 34 pages, 18 figures

Automatic differentiation (AD) is a cornerstone of modern machine learning. Since all AD frameworks implicitly fix a basis of Euclidean space to define gradients and Jacobians, they natively process mathematical objects that are arrays of a fixed set of coordinates. This makes it awkward to express objects such as rotations, volume forms, and curvatures that arise, for example, in applications of AI to science. Bespoke workarounds lack reusability. We show that (i) a single, basis-independent derivative operator can represent many geometric differential operations, and (ii) this operator can be expressed entirely in terms of existing AD primitives. This reusable transformation can be composed with JAX batching, compilation, and hardware acceleration. Geometry enters as algebraic data. We experimentally demonstrate its reuse across different manifold operators, its composition through a spacetime inverse-problem pipeline, and its use in optimization over different rotor subgroups.

0.27

asdex: Automatic Sparse Differentiation in JAX

cs.MS cs.LGmath.NA ·2026-10-08 ·Adrian Hill, Guillaume Dalle
highly cited similarity 0.56
Abstract · 1 table

Many tasks in scientific computing and machine learning require the Jacobian or Hessian matrix of a function. Automatic differentiation (AD) computes these derivatives to machine precision, but materializing a dense $m \times n$ Jacobian requires $n$ forward-mode or $m$ reverse-mode AD passes, one per column or row. For a large class of functions, each output depends on only a few inputs, making the derivative matrix sparse. Automatic sparse differentiation (ASD) exploits this structure in four steps: detection of the input-agnostic sparsity pattern, coloring of a graph to group columns or rows that can share an AD pass, compressed differentiation to compute a compressed derivative matrix with one AD pass per color, and finally decompression into the original sparsity pattern. The number of colors, and hence of AD passes, is often independent of the problem dimension: a banded Jacobian with $b$ contiguous bands, for instance, only ever requires $b$ colors, regardless of its size. asdex offers the first standalone ASD toolkit in the popular JAX ecosystem. With $\texttt{asdex.jacobian}$ and $\texttt{asdex.hessian}$, it provides sparse drop-in replacements for $\texttt{jax.jacobian}$ and $\texttt{jax.hessian}$.

0.52

Benchmarking Optimizers to Solve Inverse Problems with Differentiable Physics Simulators

cs.LG math.OCphysics.comp-ph ·2026-09-12 ·Xiang Chen, Huanhuan Xia
similarity 0.55
Abstract

Solving inverse problems with differentiable physics simulators holds the potential to revolutionize scientific discovery and engineering design, as it enjoys both the strict physical correctness from rigorous numerical physics simulators, and the high efficiency and effectiveness from automatic differentiation and gradient-based optimization. However, currently, this paradigm faces performance issues in optimization. In this work, we target benchmarking the performance of different optimizers to solve various inverse problems. We construct 12 differentiable physics simulators spanning physics domains including discrete mechanics, continuous mechanics, atomistic simulations, rendering, and semi-empirical physics models. Based on these simulators, we design corresponding inverse problems that can be categorized into parameter identification, inverse design, and optimal control. Finally, we conduct extensive experiments to compare the performance of different optimizers, including regular first-order methods, approximate second-order methods, as well as global optimizers, on these inverse problems, and analyze the results to provide insights on how to choose and design optimizers for differentiable programming. We hope such benchmarks can inspire the development of more effective optimizers, and further promote the applications of differentiable programming in various scientific and engineering domains.

0.22

Differentiable High-Order Mesh $r$-Adaptivity

cs.MS math.NA ·2026-11-17 ·Ketan Mittal, Julian Andrej, John Camier, Veselin Dobrev, Tzanio Kolev +2 more
benchmark similarity 0.55
Abstract · 14 pages, 10 figures, 2 tables

High-order mesh r-adaptivity typically entails minimizing a nonlinear functional using its gradients and, for Newton-based methods, its Hessian. These derivatives can be expensive to derive, must be updated with any changes to the functional, and errors in the derivatives directly impact the convergence of r-adaptivity. Additionally, derivatives of the r-adaptivity operator are required for consistent sensitivity analysis for inverse problems. This work develops a framework for differentiable r-adaptivity based on the Target-Matrix Optimization Paradigm. Given a pointwise functional defining the local mesh-adaptation objective, the proposed framework uses $\partial$FEM to automatically generate pointwise first- and second-order derivatives. These local derivatives are then assembled into a global gradient vector and a Hessian operator through the finite element operator decomposition. The derivative kernels support portable CPU and GPU execution, with partial assembly and tensor sum factorization enabling matrix-free application of the Hessian operator. The framework also provides consistent sensitivities of the r-adaptivity operator through automatically generated mixed derivatives and implicit differentiation of the mesh stationarity equations. Numerical experiments show GPU solve times comparable to those of hand-derived TMOP implementations on a benchmark problem, improvement in r-adaptivity convergence due to accurate computation of the derivatives, and demonstrate the use of mesh sensitivities for material-interface identification in an inverse heat-conduction problem.

0.42

NextEval: An Agent-Based Solver and Benchmark for Derivative-Free Optimization

cs.LG cs.AImath.OC ·2026-10-27 ·Shuo Chen, Li Cui, Cunxin Huang, Ya-Feng Liu, Minghang Xie +2 more
benchmark similarity 0.55
Abstract · 27 pages

Derivative-free optimization (DFO) addresses problems where derivatives are unavailable or costly to obtain. This setting often arises in simulation-based and experimental applications, where evaluating the black-box objective can itself be costly. Thus, it remains a central challenge in DFO to effectively exploit accumulated observations to select informative evaluation points under a limited budget. In this light, we introduce the NextEval solver, an agent-based DFO solver, which integrates the capabilities of a large language model (LLM) to guide the sequential selection of evaluation points. Specifically, we establish a system that combines an LLM agent with a numerical environment. The environment provides tools for analyzing past evaluations and executing numerical methods, while maintaining a shared evaluation history and method-specific state to support continuation across method switches. Using observations and tool feedback, the LLM agent selects tools, configures their parameters, and allocates the evaluation budget, while numerical methods compute and evaluate candidate points. Building on this system, we also introduce NextEval Bench, a benchmark protocol for comparing agent configurations on shared DFO tasks using the same numerical tools under matched evaluation budgets and information constraints. Numerical results on low-dimensional DFO problems demonstrate that the NextEval solver outperforms the state-of-the-art DFO baselines in function-evaluation efficiency.

0.41

SMILE: Bridging Continuous Optimization and Discrete Symbolic Recovery

cs.LG ·2026-09-04 ·Mansooreh Montazerin, Antonio Ortega, Ajitesh Srivastava
top venue similarity 0.55
Abstract · Accepted to NeurIPS 2026

Symbolic regression (SR) discovers closed-form mathematical expressions from data, offering interpretability beyond black-box models. Existing methods suffer from slow convergence in combinatorial search spaces and lack mechanisms to exploit compositional structure in the data. We introduce SMILE (Sine, Multiplication, Identity, Logarithm, Exponential), a hybrid framework that unifies continuous gradient-based optimization with discrete symbolic recovery through three stages: structural analysis of the data to identify the compositional hierarchy of the target expression, continuous optimization to learn parameters of a network that encodes the target expression using interpretable activations, and symbolic recovery through structured pruning, coefficient optimization, and rounding. This final stage distills the learned network into a compact expression with exact symbolic constants. We evaluate SMILE on SRBench across ground-truth and black-box datasets, with ablation studies validating each component. SMILE achieves the highest symbolic solution rate at the largest noise levels, demonstrating strong robustness where competing methods degrade substantially. It consistently lies on the Pareto front of accuracy versus complexity, recovering significantly simpler expressions in a fraction of the time required by the competing methods.

0.25

ADSPICE: A Method of Automatic Differentiation for Gradient-Based Lumped Circuit Optimization with Native SPICE

eess.SY ·2026-11-17 ·Marcus W. Wolff, Alan Brannon
similarity 0.54
Abstract · 12 pages, 14 figures

This work presents a novel computer-aided design (CAD) tool for gradient-based circuit optimization using SPICE and automatic differentiation, which we term ADSPICE. Automatic differentiation (AD) is a technique for evaluating a derivative by breaking it into its elementary differentiation operations and repeatedly applying the chain rule, in contrast with a finite-difference-based derivative computed from a small perturbation. Native SPICE (realized with ngspice) is equipped with automatic differentiation toward gradient-based optimization, particularly adjoint-mode optimization. This is implemented with custom C code and shared library control, allowing easy integration with existing SPICE infrastructure. We show a computational performance analysis, demonstrating faster optimization vs. an existing finite-difference method, SPICE OPUS, the closest competitor in SPICE optimization tools. The result is a speedup factor of 383x for the largest circuit compared within the state-of-the-art, SPICE OPUS. For circuit sizes above approximately 3,500 nodes, the time taken by ADSPICE optimization decreases with increasing problem size, as a proportion of total computational time. The speed bottleneck is shown to be SPICE solves rather than ADSPICE enhancements. Voltage gradients needed for forward-mode AD are benchmarked against Xyce, another state-of-the-art simulator, using an integrated CMOS operational amplifier test circuit. The results match with a maximum error of 0.203%, further validating the accuracy of our method. ADSPICE is shown to enable SPICE, and SPICE-based simulators, with automatic differentiation for gradient optimization, without requiring creation of a new circuit simulator.

0.39