CS Paper Radar

ExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale

Kevin Dick, Katie Lonsway

cs.LG ·submitted 2026-11-02 19:38 UTC · v1
benchmarkcode/data

Abstract

Hyperparameter optimization (HPO) returns a single optimal configuration, systematically discarding the explored performance landscape that encodes how results depend on configuration choices. Fitness landscape analysis comprises a robust collection of methods from evolutionary computation offering ruggedness indices, autocorrelation functions, and fitness-distance correlation that have never been applied to machine learning hyperparameter spaces at scale, given that fine-grained ground-truth enumeration data has not existed. We present ExaTune, an open-source HPC system to produce a new artifact class: the hyperparameter landscape map. ExaTune exhaustively evaluates every point on a discretized hyperparameter grid via $K$-fold cross-validation, aggregates results into reproducible parquet benchmarks, and applies a suite of five formalized landscape metrics: spatial autocorrelation (Moran's $I$), ruggedness, local optima density, fitness-distance correlation (FDC), and variance-based parameter importance. We release a benchmark collection of 8 complete landscape maps (totalling 285,080 exhaustively evaluated configurations) spanning five model families (SVMs, random forests, XGBoost, ridge regression, and $k$-nearest neighbors) across six classification and two regression benchmarks, hosted on HuggingFace. Empirical analysis demonstrates that Bayesian optimization, Optuna, Hyperband, and Random Search at standard trial budgets systematically miss instability regions, secondary optima, and parameter interaction effects that are visible in complete landscapes; information that is structurally unavailable to any optimizer by design.

Comment
11 pages, 3 figures, includes supplementary materials

Closest papers

Nearest neighbours by meaning across the whole index.

A Flexible and Generic Approach for Explainable Landscape Analysis and the pyXla Toolbox

cs.NE cs.MS ·2026-10-05 ·Tony Ombaso, Anna S. Bosman, Arnaud Liefooghe, Katherine M. Malan, Sébastien Verel
similarity 0.59
Abstract · 40 pages, 32 figures, submitted to Evolutionary Computation Journal

Landscape analysis has been successfully applied to understand complex optimisation problems, gain insights into algorithm behaviour, and automate algorithm selection and configuration. Although many landscape analysis techniques have been developed over the last decades, it remains difficult for researchers and practitioners to decide which approaches are appropriate and to implement them in practice. Some tools are available, but these are either restricted to particular problem domains (e.g., unconstrained black-box continuous optimisation), or are limited in what they model and measure. In addition, output from landscape analysis is often not easily interpretable, especially when computed landscape features do not correspond with aspects of problems that practitioners are familiar with. In this paper, we introduce a principled approach for explainable landscape analysis (XLA) with an associated Python package called pyXla. The approach is generic in that it applies to problems with different representations (continuous or combinatorial), with single or multiple objectives, with or without constraints. The extent of analysis provided by the XLA framework depends on the data available, with richer analysis offered as additional information is provided by the user. We demonstrate the explainable output produced by pyXla on a selection of hand-crafted problems with diverse landscape characteristics.

0.30

Learning to Optimize UAV Path Planning for Data Sensing in Wireless Sensor Networks

cs.RO cs.NE ·2026-09-15 ·Sijie Ma, Zeyuan Ma, Weijia Cao, Yue-Jiao Gong, Lingling Ma +2 more
similarity 0.57
Abstract

UAVs have emerged as highly flexible platforms for data sensing in Wireless Sensor Networks (WSNs). Path planning for UAVs in such tasks plays a key role to assure remote sensing effectiveness and friendly energy consumption. However, existing approaches show two key limitations: i) they are primarily hand-crafted with certain design biases that harm adaptation on unseen tasks. ii) they predominantly assume idealized spatial complexities of actual environments through simplified simulation, causing them to underperform during real-world deployment. In this paper, we propose a novel learning-assisted planning framework, termed Landscape-Aware Meta Differential Evolution (LAMDE), to tackle the mentioned limitations. The major contributions come from the following aspects. We first re-formulate such UAV path planning problem to embrace challenging constraints. To efficiently navigate this highly constrained space, we propose a bi-level learning to optimize approach, where the meta-level is a trainable algorithm configuration policy that meta-learns an adaptable planning strategy for low-level planning algorithm. To address the potential training data scarcity and distribution shift in real-world environments, we introduce a landscape-aware automatic augmentation scheme that enriches training data. At the low-level, a Differential Evolution algorithm is deployed for solving the path planning tasks. To enhance the solving flexibility, we further design a variable-length encoding strategy that dynamically prunes redundant hover points and optimizes continuous flight parameters concurrently within a unified search space. Based on all proposed designs, we meta-train LAMDE and compare it with representative baselines. Comprehensive experiments demonstrate that LAMDE achieves state-of-the-art performance on the tested complex UAV path planning tasks in WSN data collection scenarios.

0.42

A Controlled Evaluation of Curvature Inspired Scalar Feature Scaling for Domain Generalization

cs.LG cs.AIcs.CV ·2026-10-13 ·Sunny Gupta, Amit Sethi
published similarity 0.57
Abstract · Accepted at TMLR, 9 pages, 1 figures

Domain-specific curvature is an appealing idea for domain generalization (DG), but curvature learning can denote substantially different computations. This paper evaluates one deliberately narrow implementation: the code calls it HyperDG, but it rescales Euclidean encoder features during training by learned positive scalars and uses an ordinary linear classifier on unscaled Euclidean features at inference. It does not implement a Lorentz or Poincaré representation, geodesic prediction, or a Sharpness-Aware Minimization inner loop. We audit the executed objective and evaluate six conditions on PACS, VLCS, Office Home, and TerraIncognita. The completed design contains 384 runs: six conditions, four held-out targets, and four run seeds per dataset. Checkpoints are selected using only the three source-validation environments. Inference first averages paired seed differences within each held-out target; targets, rather than 16 seed-by-target observations, are the per-dataset clusters. The full implementation's equal-dataset macro point estimate is 1.61 percentage points below native DomainBed ERM (95% clustered CI [-3.17, -0.04]). All four dataset point estimates favor ERM, but an exact four-dataset sign-flip test gives p = .125 and Holm-adjusted p = .250; we therefore claim no adjusted evidence of a difference. The cleaner domain-specificity contrast, separate learned source scales without their variance penalty versus one shared learned source scale, is -0.13 pp (95% CI [-0.53, 0.27]). Its 90% CI lies within a revision-stage $\pm 1$ pp smallest effect of practical interest (TOST p = .003), supporting equivalence only for the source-selected suite macro, not every dataset, target, or final-step endpoint. The study thus supplies a bounded negative result for this scalar training package; it does not test or refute genuine hyperbolic representation learning.

0.49

Amortizing Scaling Law Construction Costs

cs.LG cs.AI ·2026-09-04 ·Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik, Swagatam Haldar, Sam Laing +4 more
similarity 0.56
Abstract · 5 pages, 2 figures, workshop

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$.

0.10

Neuroevolution Arena: Nested Ecological Evaluation of Update-and-Inheritance Regimes across Neural Architectures

cs.AI cs.NE ·2026-08-10 ·Yuxu Ge, Yifei Cheng
similarity 0.56
Abstract · 22 pages, 2 figures, 8 tables. Preprint

Competitive artificial-life systems can rank trained controllers differently under training and ecological evaluation. We present Neuroevolution Arena, a GPU-accelerated spatial ecology of independently parameterized neural-network cells, and an audit-tracked nested evaluation protocol. Three implementation-specific update-and-inheritance regimes (EvoEvo, EvoRL, and RLRL) are crossed with two neural architectures for 50,000 generations in three independent training runs per condition. One saved elite-controller artifact from each of the 18 runs enters an aligned-run frozen-evaluation design comprising 198 computational jobs. Pairwise effects average three seed-defined ecological contexts (two cooperation-permitting and one attack-permitting) within each aligned training-run block; the independent level remains n = 3 runs per condition. RL-enabled regimes attain higher recorded training fitness than EvoEvo, whereas pairwise outcomes show architecture-conditioned majority patterns and substantial artifact dependence. Six-way winners vary across artifacts and contexts, and the prespecified survival endpoint has a complete floor. We contribute a nested protocol that separates training-run artifacts from evaluation contexts and exposes, rather than conceals, their different sources of variation.

0.12

The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

cs.LG cs.AI ·2026-10-20 ·Hannan Mahadik, David Salinas, Jacek Golebiowski, Aaron Klein
similarity 0.55
Abstract

Direct preference optimisation (DPO) is widely used for language model post-training, but its performance depends strongly on the choice of its hyperparameters, such as the learning rate and the reference-policy regularisation parameter $β$. To better understand the DPO hyperparameter landscape, we conduct extensive experiments using a dense hyperparameter grid across different model scales from the same family. This allows us to analyse how the optimal DPO hyperparameters evolve with model size and the number of update steps, with intermediate checkpoints providing an inexpensive signal to accelerate the hyperparameter optimisation process. Furthermore, by evaluating our models on both static and dynamic benchmarks, we identify which benchmarks provide the strongest signal for hyperparameter selection. Finally, we investigate how different state-of-the-art hyperparameter optimisation strategies perform in this setting. Our experiments show that the optimal learning rate varies with model size and that hyperparameters are not transferable across training datasets. We also find that validation loss is a poor proxy for hyperparameter optimisation, as hyperparameters that minimize validation loss only weakly correlate with optimal hyperparameters for downstream benchmarks. Therefore, DPO hyperparameters need to be tuned specifically for the target model, dataset, and evaluation objective. We also show that short candidate training runs can substantially reduce the cost of this selection. We release all our training configurations and evaluation results to support reproducible analysis of selection quality and training cost.

0.19

Evolutionary Adaptive Neighborhood Learning: A Genetic Algorithm for Instance-Dependent k Optimization in Nearest Neighbor Classification

cs.LG cs.AI ·2026-11-05 ·Isaac Touza, Kaladzavi Guidedi, Kolyang
similarity 0.55
Abstract

The number of neighbours k is the central hyperparameter of the k-nearest neighbour (k-NN) classifier and is usually selected globally through cross-validation or evolutionary search. However, the bias-variance trade-off is inherently local: dense and noise-free regions may favour small neighbourhoods, whereas sparse or noisy regions may require larger ones. Since global-k optimization is one-dimensional, replacing exhaustive search with a genetic algorithm offers little computational advantage. We therefore optimize an instance-dependent neighbourhood policy instead. We propose EANK (Evolutionary Adaptive Neighborhood K), which uses a low-dimensional policy to map local neighbourhood descriptors, including density, label entropy, vote margin, and concentration, to an adaptive number of neighbours k(x). A real-coded elitist genetic algorithm optimizes this policy using leave-one-out accuracy with a neighbourhood-smoothness penalty. Precomputed neighbour votes reduce fitness evaluation costs. We investigate theoretical properties of the method, including dominance over global-k solutions on the training criterion, evolutionary convergence, policy regularity, and generalization guarantees. Experiments on synthetic data with spatially heterogeneous structure and twelve public datasets evaluate predictive performance, robustness, and computational cost. Under nested cross-validation, EANK achieves competitive performance relative to tuned global-k baselines and improves upon the default k=5. We also examine the gap between training and test performance to assess the practical value of adaptive neighbourhood selection.

0.31

Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling

cs.LG ·2026-10-27 ·Ethan Baron, Shmuel Berman, Anirudh Satheesh, Akshay Vegesna, Samip Dahal +1 more
similarity 0.54
Abstract

As growing compute budgets outpace the growth of high-quality training data, large language models must be trained with both compute and data constraints in mind. Unfortunately, existing prescriptions for data-constrained scaling suffer from two important limitations. First, past works hold key hyperparameters such as weight decay fixed with scale, despite the fact that optimal hyperparameters vary wildly with model size and number of epochs. Second, past works neglect ensembling, a promising axis for scaling under data constraints. As a surprising consequence, prior prescriptions can lead to loss that increases with compute budget. We address both limitations by fitting power laws that prescribe how to transfer hyperparameters with scale and scaling laws that prescribe how to allocate compute with ensembling. Training on 100M tokens from FineWeb, our recipe reduces validation loss by over 0.15 nats compared to prior recipes, a 14% decrease in perplexity, and achieves monotonic improvements with compute through ensembling. These gains are achieved using prescriptions fit to smaller runs and validated at up to 12$\times$ larger compute budgets.

0.09