CS Paper Radar

Adaptive Partitioning Schemes for Optimistic Optimization

Raja Sunkara, Ardhendu Tripathy

cs.LG stat.ML ·submitted 2026-10-02 20:45 UTC · v1
top venue

Abstract

Applications such as engineering design often require us to optimize a black-box function, i.e., a system whose inner processing is not analytically known and whose gradients are not available. Practitioners often have a fixed budget for the number of function evaluations and the performance of an optimization algorithm is measured by its simple regret. In this paper, we study the class of "Optimistic Optimization" algorithms for black-box optimization that use a partitioning scheme for the domain. We develop algorithms that learn a good partitioning scheme and use flexible surrogate models such as neural networks in the optimization procedure. For multi-index functions on an $m$-dimensional subspace within $d$ dimensions, our algorithm attains $\tilde{O}(n^{-β/ d})$ regret, where $β= 1 + \frac{d-m}{2m-1}$, as opposed to $\tilde{O}(n^{-1/d})$ for SequOOL, a state-of-the-art optimistic optimization algorithm. We use our approach to improve the quality of Activation-aware Weight Quantization (AWQ) of the OPT-1.3B model, achieving $\sim10\%$ improvement in performance relative to the best possible unquantized model.

Comment
Accepted at ICML 2025
Journal ref
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:58029-58064, 2025

Closest papers

Nearest neighbours by meaning across the whole index.

Artificial Neural Networks as Surrogate Models in Black Box Optimization

cs.NE cs.AI ·2026-09-16 ·Md Khadimul Islam Zim, Martin Holeňa
similarity 0.62
Abstract

Black-Box Optimization (BBO) is often applied in several engineering fields and can utilize an advancement of numerical measure- ments and simulation technologies. It deals with the optimization func- tions, where an analytical description is unavailable. It relies on meth- ods that require only an input point in the search space, paired with its corresponding objective function value, obtained through non-analytical means, e.g., sensors, experiments, or simulations. Common approaches include evolutionary optimization and other metaheuristics. Since BBO methods rely solely on objective function values, they typically require many evaluations, which becomes problematic when evaluating the ob- jective function is time-consuming or expensive. This leads to using surrogate-based optimization which evaluates selected true objective val- ues and trains a regression model to approximate the objective function across the search space. Surrogate-assisted black-box optimization is a small-data learning problem because the optimizer must approximate an expensive objective function from limited evaluations. Surrogate models act as data-efficient regressors, guiding the search toward promising or informative points under a restricted evaluation budget. In this paper, a new surrogate model using artificial neural networks, called Adaptive- Fidelity Nexus Covariance Matrix Adaptation Evolution Strategy (AFN- CMA-ES), is proposed for the selective evaluation of objective functions. The experimental results show its competitive performance compared to state-of-the-art surrogate-assisted BBO methods.

0.28

The Devil Is in the Reconstruction Loss Scale: Rethinking Optimization in LLM Quantization

cs.CL cs.AI ·2026-10-01 ·Chao Li, Shigeng Wang, Anbang Yao
code/data similarity 0.61
Abstract · Project page: https://github.com/IntelChina-AI/RMSE

Post-training quantization (PTQ) methods typically use sequential quantization that partitions a pre-trained LLM into a series of units (e.g., transformer blocks), with one unit quantized at each stage. State-of-the-art PTQ methods are predominantly learning-based, optimizing auxiliary quantization parameters (e.g., scaling factors, rotation matrices, clipping thresholds, and adapters) via gradient descent to minimize a reconstruction loss. A common practice is to use mean squared error (MSE) as the reconstruction loss function, yet its induced optimization behavior remains largely unexplored. In this work, we take a holistic view of sequential quantization and systematically investigate how optimization evolves from the first quantization stage to the last, aiming for a deep understanding of optimization in learning-based PTQ schemes. Through extensive empirical studies spanning representative learning-based PTQ methods, LLM families, model scales, architectures, quantization settings and various tasks, we consistently uncover Optimization Imbalance: reconstruction loss magnitudes vary dramatically across stages, accompanied by highly uneven gradient magnitudes and parameter updates under MSE. We term the cross-stage range of loss magnitudes the reconstruction loss scale, and reveal that MSE translates the unexpectedly large reconstruction loss scale into highly uneven gradient magnitudes, which in turn lead to uneven optimization strength across quantization stages. This finding suggests a general principle for improving learning-based PTQ: optimization strength across stages should be decoupled from the reconstruction loss scale. Theoretically, we show that root mean squared error (RMSE) variants defined at the sample, channel, token, and element levels naturally realize this principle through implicit gradient normalization, outperforming MSE significantly as a drop-in replacement.

0.34

DCEmbed: Scalable Optimization over Neural Surrogates

cs.LG math.OC ·2026-09-27 ·Akshay Sreekumar, Nicolas Christianson, Priya L. Donti, Ellen Vitercik, Ram Rajagopal
similarity 0.61
Abstract

Neural surrogates can accelerate large-scale optimization by replacing expensive or intractable model components with efficient learned approximations, but solving the resulting embedded problems can remain prohibitively costly. For instance, standard exact encodings of neural networks with ReLU activations allow the problem to be solved by mixed-integer solvers, but add large numbers of binary variables to accommodate the nonlinearity of the activations, which can render the problem computationally prohibitive. To address this, we propose DCEmbed, a heuristic for optimization problems with embedded neural surrogates that leverages the difference-of-convex (DC) representation of the network and avoids adding activation binaries. Exploiting shared structure within the DC representation of a ReLU network, we derive a reduced-size, exact formulation for its convex components that can be embedded in optimization problems using just two linear inequalities and one continuous auxiliary variable per hidden neuron. Using this, the problem is solved via an iterative penalty convex-concave procedure, where only the concave portions of the neural terms are approximated at each stage. The original objective, constraints, and any discrete decisions are retained, allowing standard convex or mixed-integer optimization solvers to optimize the host and surrogate jointly at each iteration. In experiments on quadratic programs, mixed-integer resource allocation, and neural two-stage stochastic programming, our method demonstrates much faster progress toward high-quality feasible solutions than approaches using exact mixed-integer embeddings. In particular, DCEmbed achieves $4\times$ lower normalized primal integral than the best exact baseline on the resource allocation problem, while in two-stage stochastic programming it reaches the global surrogate optimum $\sim 5\times$ faster than Gurobi ML.

0.06

KT-EGO: A Knowledge Transfer Assisted Efficient Global Optimization Algorithm for Solving High-Dimensional Expensive Black-Box Problems

cs.CE cs.LG ·2026-09-28 ·Qineng Wang, Liming Song, Yun Chen, Guangjian Ma, Zhendong Guo +1 more
publishedbenchmarkhighly cited similarity 0.60
Abstract · 25 pages, 12 figures; abridged author manuscript

Many engineering problems involve optimizing a high-dimensional expensive black-box (HEB) design space. To solve such problems efficiently, we propose a knowledge transfer assisted efficient global optimization (EGO) algorithm, labeled as KT-EGO, which extends the EGO algorithm for solving problems over higher dimensions (i.e., $d>20$). Specifically, the original design space is divided into several low-dimensional subset design spaces. More importantly, in order to extract information from the subset design spaces to accelerate the progress of full optimization, we propose a surrogate-based data fusion strategy in KT-EGO. And further, a searching strategy with an adaptive variable range is devised to enhance the exploitation of promising areas. To show the effectiveness of our proposed algorithm, it is compared against the state-of-the-art algorithms over 12 benchmark functions and a 28-dimensional engineering optimization for the design of compressor blade, which fully validates the effectiveness of the KT-EGO for solving HEB problems.

0.49

Blog: Survey of Optimizers

cs.LG cs.AI ·2026-08-28 ·Ruoran Xu
survey similarity 0.60
Abstract

Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.

0.10

Input convex neural networks as surrogates in mathematical optimisation

cs.LG math.OC ·2026-08-10 ·Yu Liu, Jan Kronqvist, Fabricio Oliveira
similarity 0.60
Abstract

Embedding trained neural networks as surrogates within optimisation problems is an established practice in operations research. The prevailing approach uses feedforward neural networks (FNNs) with ReLU activations, whose piecewise-linear structure admits an exact but computationally intensive mixed-integer programming (MIP) reformulation as the networks grow. We advocate input convex neural networks (ICNNs) as structurally superior surrogates when the underlying response is approximately convex or concave. The convex architecture offers two computational advantages. First, the ICNN-MIP formulation tends to yield a tighter linear programming (LP) relaxation than its FNN-MIP counterpart, with no integrality gap in favourable instances. Second, ICNNs uniquely admit an LP-based reformulation via epigraph representations of ReLU activations, though this embedding is not always exact. When it is not, we exploit the properties of ICNNs to construct the strongest continuous relaxation over box domains, namely, the convex hull of the ICNN's graph, bounded below by the epigraph and above by the concave envelope; this construction is tractable under input convexity but hard for general ReLU networks. On this basis, we develop a branch-and-bound algorithm that builds this relaxation at each node, branches directly on input variables rather than intermediate variables as in MIP reformulations, and terminates at the root node whenever the epigraph embedding is valid. Case studies on humanitarian food aid, oil well routing, and wine blending show that ICNN surrogates match FNN accuracy and deliver gains in solve time and scalability, supporting ICNN as the default surrogate when the underlying function is convex, concave, or well-approximated as such.

0.40

Rethinking Learnability in Offline Data-driven Optimization

cs.LG cs.AIcs.NE ·2026-09-01 ·Chao Qian, Chen-Guang Wang, Rong-Xi Tan, Ke Xue
theory similarity 0.59
Abstract

Black-Box Optimization (BBO) has broad applications, while traditional algorithms such as evolutionary algorithms and Bayesian optimization face efficiency challenges as real-world BBO problems grow increasingly complex. Data-driven optimization has been the most popular paradigm to improve the efficiency of BBO, by learning from data. Offline data-driven optimization seeks high-quality solutions using only a fixed set of previous evaluations, attracting substantial attention because it requires no additional online evaluations. Many offline optimization methods have been proposed, but a fundamental question remains unanswered: what learnability is sufficient for offline optimization? Prior theoretical studies show that Probably Approximately Correct (PAC) learnability is insufficient, as the optimal region may remain poorly learned even when most regions are well learned. In this paper, we propose algorithm-dependent learnability, which requires accuracy only on the optimizer's trajectory. We prove that its value-query form is sufficient for representative discrete settings, including greedy and local search for submodular maximization, while its first-order analogue is sufficient for projected gradient descent on convex minimization. Motivated by this notion, we formalize a trajectory-learning framework comprising trajectory construction, trajectory modeling, and candidate generation, and analyze existing trajectory-based methods under it. We further propose Uncertainty-aware Gradient-guided Trajectory Learning (UGTL), which constructs locally coherent improvement trajectories reflecting plausible search paths, models them with conditional diffusion, and selects a diverse candidate set. Our experiments show that UGTL achieves the best average rank, 3.1/25, among 25 methods on Design-Bench tasks, and confirm that our trajectory construction plays a significant role in the improvement.

0.27

Learning-Augmented Optimization for Strategic Two-Echelon Spare Parts Network Design

cs.LG math.OC ·2026-09-11 ·Donato Maragno, Marco Caserta, Alberto Sinigaglia, Komlanvi Ametana, David Corredor Montenegro +1 more
similarity 0.59
Abstract

We study the strategic design of a two-echelon spare-parts inventory network where evaluating each candidate topology requires an expensive inventory optimization model. The design partitions hundreds of sites into feasible clusters and selects a central replenishment site for each cluster to reduce costs while maintaining service levels. Because the optimizer favors candidates with high predicted savings, it can exploit optimistic surrogate errors. We develop a conservative framework combining a graph neural network ensemble, variable neighborhood search, and set-partitioning recombination. The surrogate is trained on exact cluster evaluations, while a lower quantile of ensemble-predicted savings guides the search to limit optimism. Clusters found during the search are recombined through set partitioning using surrogate-based objective coefficients. The resulting network is evaluated with the exact inventory model, and only this evaluation is used to report performance. In a case study of 246 fulfillment centers in Amazon's North American network, the framework improves combined savings by 30.5% over an optimization baseline based entirely on exact cluster evaluations, while maintaining approximately 99.8% service across six independent replications. Under equal computational budgets, graph-surrogate-guided search achieves higher mean exact savings than a tabular alternative under both scoring schemes. Conservative scoring improves mean savings for both surrogate classes and reduces the share of final-network clusters overestimated by the graph surrogate from 68% to 28%. Predictive and ranking accuracy deteriorate among search-generated candidates with high surrogate scores, indicating that random holdout performance can incompletely characterize surrogate quality during optimization.

0.25