CS Paper Radar

HeuFouFT: Task-Guided Metaheuristic Coordinate Search for Fourier Fine-Tuning

Ruiheng Wang, Yubo Hou, Yakun Zhu, Tianle Shen, Tao Wan, Zengchang Qin

cs.LG cs.CL ·submitted 2026-10-05 14:45 UTC · v1
code/dataunusual

Abstract

We introduce Heuristic-Guided Fourier Fine-Tuning (HeuFouFT), a task-guided framework for selecting trainable frequency coordinates in Fourier fine-tuning. Existing uniform and Gaussian band-pass schemes allocate a limited spectral budget through fixed, task-agnostic rules. HeuFouFT instead searches for coordinates using downstream performance. A coarse intensity map from lightweight block-level probes initializes three metaheuristic optimizers: Genetic Algorithm with Simulated Annealing (GA-SA), Particle Swarm Optimization (PSO), and Cuckoo Search (CS). During search, a Random Forest filters each population so that only the top 30% of candidates proceed to proxy fine-tuning. On E2E with GPT-2-Medium, all three variants outperform random-uniform FourierFT, Gaussian band-pass FourierFT, and LoRA across five metrics. PSO further outperforms LoCA, the best-performing baseline, on four metrics while using 37.6% fewer trainable spectral coefficients. Once coordinates are selected, HeuFouFT requires only 15--18% FLOPs of Full FT. These results show that task-guided search allocates limited spectral capacity more effectively than fixed sampling. Our code is publicly available.

Closest papers

Nearest neighbours by meaning across the whole index.

An important Bottleneck in LLM Optimizers: Principle-Level Local Optima

cs.LG ·2026-10-16 ·Nian Ran, Tianhe Jia, Zhengyang Mu, Yue Wang, Sebastian Brulin +4 more
benchmark similarity 0.54
Abstract · 10 pages under review

Feedback-driven LLM optimizers are increasingly used for evolutionary search, Bayesian optimization, heuristic design, and scientific discovery. Yet rapid initial gains often give way to prolonged plateaus despite substantial remaining evaluation budgets. We investigate \emph{principle-level search collapse}, where an optimizer's search becomes confined to a narrow set of design principles, even though it may continue to generate distinct candidates and achieve small improvements. We assemble a diagnostic benchmark spanning molecular optimization, network sentinel-node selection, and GPU-kernel optimization, and introduce a decision-tree protocol combining principle narrowing, prolonged performance plateaus, and gaps to attainable reference solutions. Across 161 assessable runs on the three problem domains, 78.3\% receive support for collapse, including 55.3\% with strong support. Candidate-level variation can nevertheless conceal restricted search at the principle level. We then test \emph{diversity injection}, a common response to search stagnation, such as higher t, multiple LLMs, and web search, and show that it provides limited and inconsistent benefits: among 65 molecular runs, still 73.8\% and 53.8\% receive support and strong support respectively. We finally test reference-principle interventions and find that they improve some runs, but their benefits vary across models and tasks: access to a better principle does not guarantee its effective use. Our benchmark and diagnostic protocol provide a basis for evaluating future optimizers on principle-level collapse. Our findings highlight the need for online collapse detection and better mechanisms for generating and maintaining alternative principles during optimization.

0.47

A Metaheuristic Optimization Framework for Discrete Optimization under Strict Time Limits

cs.NE ·2026-09-16 ·Umut Çalıkyılmaz, Nitin Nayak, Sven Groppe
benchmarkunusual similarity 0.53
Abstract

Real-time applications often rely on optimization approaches that can find high-quality solutions to hard problems on the order of milliseconds. Metaheuristic optimization frameworks (MOFs) are useful tools for such tasks, as they provide large sets of general-purpose search mechanisms that can return solutions under different computational budgets. However, existing work largely overlooks the available computation time as an explicit dimension of analysis. In this work, we introduce STILO, a MOF specifically designed for optimization under strict time limits. STILO integrates fine-grained configuration spaces for ant colony optimization (ACO), genetic algorithm (GA), and simulated annealing (SA), combining existing and novel operators. We performed experiments using both synthetic and benchmark instances of various discrete optimization problems. The results indicate that the proposed discrete distance calculation mechanism for SA is useful for optimization under strict time limits. They also show that the relative effectiveness of the proposed problem-independent graph structures for ACO can vary across time limits, even for the same instance characteristics. More generally, the results demonstrate that the effectiveness of algorithm families and operators depends not only on the problem type, but also on the characteristics of the instance and the available computational budget.

0.29

Early-Stopping Thresholds for ES-HyperNEAT: A Data-Driven Approach from Fitness Dynamics

cs.NE cs.LG ·2026-09-11 ·Romain Claret, Arthur Gygax, Michael O'Neill, Paul Cotofrei, Pascal Felber
published similarity 0.52
Abstract · 8 pages, 5 figures, 9 tables. Accepted version of a paper published at the 2026 IEEE Congress on Evolutionary Computation (CEC), part of the…

Most hyperparameter configurations for Evolvable-Substrate HyperNEAT (ES-HyperNEAT) produce networks that stagnate at random-guessing performance, wasting computational resources. We frame early stopping as binary classification on early fitness trajectories: for each trial, we compute the cumulative median of best-per-generation fitness and test it against a threshold derived by maximizing the F1 score on an initial 90-trial dataset. The resulting rule (generation G* = 3, threshold T* = 0.140) achieves F1 = 0.872 on 180 independent validation trials, retaining over 90% of successful trials while cutting computational cost by 41.6%. Compared to Hyperband, our domain-specific rule is 64% more efficient with higher mean fitness, though Hyperband occasionally discovers higher peak solutions. On a converged search population the rule becomes too aggressive (recall 31.1%), motivating adaptive thresholds. The specific thresholds are ES-HyperNEAT-specific, but the methodology, deriving stopping criteria from fitness dynamics classification, is applicable to other evolutionary algorithms with stagnation-prone hyperparameter spaces.

0.64

Hybrid metaheuristics for architectural design optimization under small evaluation and wall-clock budgets

cs.NE cs.CEmath.OC ·2026-10-19 ·Luciano Ambrosini
similarity 0.52
Abstract · 39 pages, 10 figures, 6 tables

Architectural and structural design optimization in parametric software affords about a thousand evaluations of an expensive model for problems with tens of variables, a regime for which most metaheuristics were not designed. On the side of artificial intelligence, two hybrid metaheuristics are proposed: a covariance matrix adaptation evolution strategy (CMA-ES) whose sampling is enriched with differential-evolution trial vectors, and a particle swarm with success-history parameter adaptation, differential-evolution injection and local refinement. A budget-aware delay of population-size reduction in adaptive differential evolution is also proposed. Under a hard cap of 1,200 evaluations, all counted, the CMA-ES hybrid ranks first on seven shifted and rotated functions at 10 and 30 variables and beats CMA-ES on six of seven functions at 30 variables. Ablations attribute this gain to the injection and show that a local search pays off only when its result re-enters the population. The delayed schedule improves three adaptive differential-evolution variants in 40 of 51 comparisons, with no loss. On the side of engineering application, in Grasshopper the CMA-ES hybrid and plain CMA-ES are never significantly worse than the optimizers of Galapagos, SilverEye, Opossum and Tunny on travelling-salesman instances and on a built 85-variable free-form shell, where they end 48 % below the best external tool in 15 minutes. On a 22-variable building-performance problem with binary choices, differential evolution leads instead. Shifting and rotating the functions reverses the rank of Harris hawks optimization. Solvers should therefore be selected by problem structure and stopping time, not by a single ranking.

0.24

JANUS: Online Jacobian-Aligned Infill for Black-Box Optimization

cs.NE ·2026-08-24 ·Hongyuan Yu, Pufan Xu, Jiaojiao Yi, Yiding Tian, Mingrui Sun +2 more
code/data similarity 0.52
Abstract · 27 pages, BBO method

Population optimizers such as CMA-ES, DE, and multi-objective evolutionary algorithms drive search mainly through selection signals that are scalar or rank based: such a signal indicates that one candidate outperforms another, but not the local direction responsible for the improvement. JANUS (\emph{Jacobian-Aligned Newton-Unified Search}) is a plug-and-play infill module that extracts this missing local geometric signal without replacing the host optimizer. It estimates a local Jacobian from the recent evaluation trace; the same Jacobian yields both a damped Gauss--Newton exploitation candidate and a trace-preserving exploration metric, reserving a fraction of the host's per-generation candidate slots for geometry-guided infill rather than spending evaluations on top of the host's budget. Unlike MetaBBO methods, JANUS needs no offline training or task distribution, estimating this geometry on the fly from the current run alone, while the host keeps full control of selection, survival, covariance adaptation, and step-size control. Under same-protocol comparisons, JANUS improves the CMA-ES host on \textbf{11--15/16} BBOB functions across $d\in\{30,100,500\}$. It also attains the best mean error on \textbf{13 of the 16} functions at $d{=}500$ in the complete NN-BBO/MetaBBO baseline comparison, with no training cost, and yields a $936\times$ geometric-mean improvement over the host on a $d{=}1000$ BBOB subset. On structured and multi-objective tasks, JANUS gives the best mean cost on 1135-dimensional UAV path planning ($-12.8\%$ vs.\ the strongest baseline), and it improves SMS-EMOA/AGE-MOEA2 hosts on 12/38 multi-objective tasks with zero significant regressions. Code is available at https://github.com/hongyuanyu/JANUS.

0.52

When the implementation matters more than the algorithm: NSGA-II and other multi-objective optimizers in engineering design tools

cs.NE cs.CEmath.OC ·2026-10-14 ·Luciano Ambrosini
code/dataunusual similarity 0.51
Abstract · 29 pages, 8 figures, 5 tables. Per-run data, emulator models and analysis scripts: https://doi.org/10.5281/zenodo.23157761

Engineering design tools increasingly embed optimizers as ready-made components that users select by algorithm name, while operator settings and bound, integer and duplicate handling stay fixed inside each tool. This study tests whether the name predicts the result. A tool-agnostic harness exposes one deterministic black box, an emulator of a published 22-variable mixed-integer building-performance problem, to the native optimizers of three Grasshopper plug-ins, to canonical NSGA-II, SMS-EMOA and GDE3 from Python libraries, and to the model-based RBFMOpt. Eighteen configurations were run at 5,000 evaluations under three population allocations, with statistics fixed in advance. Four NSGA-II implementations differed by up to 5% in hypervolume, more than three canonical algorithms of different families differed from one another, and their response to population size differed by up to 10 percentage points; for two MOEA/D implementations, by 28. On ZDT test functions the same NSGA-II implementations differed up to 470-fold in IGD. An ablation in one open-source NSGA-II reproduced the behaviour of one tool from its documented settings but not the gaps of the other two, so documented parameters do not identify an implementation. Counting the budget in wall-clock time reversed the ranking of the model-based optimizer, and reporting the final population instead of the archive understated some configurations by up to a third; most differences fell below the surrogate's error. For optimizers embedded in design-support systems, the implementation is part of the method; a reporting checklist is derived: implementation and version, effective settings, budget in time and evaluations, and archive-based indicators.

0.35

Targeted search shows that random-device testing underestimates worst-case error in a simulated wave-based neural operator

cs.LG cs.ETphysics.optics ·2026-10-05 ·Samrendra Roy, Jason Yoo, Souvik Chakraborty, Syed Bahauddin Alam
theory similarity 0.51
Abstract · 50 pages (19 main text and references, 31 Supplementary Information), 5 figures, 1 table

Wave-based processors promise fast, energy-efficient Fourier layers for neural operators. They are usually validated on randomly sampled devices, but using them requires knowing how large their error can become under fabrication and alignment variation. In a stylised numerical case study, a hybrid Fourier neural operator runs its four spectral layers on simulated coherent 4f processors with 32 toleranced knobs, whose half-widths are representative rather than calibrated. For 120 models (four tasks, six training methods, five seeds), we compared the worst of N random in-spec devices with a searched one. On a deterministic simulator with one frozen draw of the random static errors, the searched device's held-out error was 1.08-3.10 times the maximum over 200 Monte Carlo devices and 1.06-2.71 times that over 1000. With 20 fresh static draws, it still exceeded the maximum over 200 random devices in 116 of 120 models. Under uniform sampling, the probability of drawing such a device is at most 0.37% per model (two-sided 95% Clopper-Pearson), which says nothing about how large its error is. The gap persisted with uniform or Sobol' sampling at the search's budget, shared knobs, a second crosstalk model, box scales of 0.25-2 and a pixel-level device model. Models trained only with random static errors reached 3.7-39.9 times their nominal error on searched devices, and fine-tuning on random and gradient-searched devices gave the lowest searched error of the six in all 20 task-seed pairs. For two heat-exchanger quantities, a search targeted at each exceeded the worst of 1000 random devices in all 39 models, and hence the Wilks 95/95 limit (worst of 59). For the mean pressure of 11 models, no random device exceeded a 1% error threshold, but the searched device did. Random testing estimates how often errors exceed a threshold; worst-device search gives a lower bound on how large they can be.

0.28

Evaluation of portability and performance of an OpenMP5 offloaded Quantum-Inspired Evolutionary Optimization Across the GPU Ecosystem

cs.PF math.OC ·2026-09-25 ·Kasturi Venkata Srikanth, Ashish Singh, Ferdin Sagai Don Bosco, Aman Mittal, Abhishek Singh +2 more
similarity 0.51
Abstract

Quantum-inspired evolutionary optimization (QIEO) is a new class of population-based metaheuristic optimization algorithms which represents design variables as a set of qubits and searches a continuous, multi-dimensional landscape through rotation of the qubit's amplitude pair. Every generation rotates those amplitudes toward a single elite, which corresponds to that generation's best. The per-generation cost scales as $O(N_p N_g)$ for $N_p$ chromosomes and $N_g$ genes (decision variables). Production use of such solvers is rarely confined to a single machine class. Prototypes are run on laboratory servers- before moving to rented cloud workstations for more involved campaigns. The largest problems are reserved for leadership-class accelerators. This paper asks whether a \emph{single} OpenMP~5 source of QIEO, offloaded with \texttt{\#pragma omp target}, is a viable production path in each of those settings. We report three independent, campaigns of the 0/1 knapsack problem against a same-source multi-core Intel CPU baseline. The study comprises approximately 3,000 runs spanning varying chromosome and gene counts, evaluated using both chromosome-level and gene-level offload strategies on the NVIDIA Tesla V100 SXM2, NVIDIA A100 80GB, and AMD Instinct MI300X GPUs. Deployment-specific nuances such as Volta's constant-memory cliffs, Ampere's L2 persistence and \texttt{cp.async}, CDNA~3's Infinity Cache and XCD occupancy are addressed to ensure high performance of these platforms. Results reveal gene-parallel offload achieved geometric-mean speedups of 90$\times$, 136$\times$, and 155$\times$ over a single CPU core on the V100, A100, and MI300X, respectively, and 12$\times$, 17$\times$, and 16.6$\times$ over 72 host threads. Furthermore DetermineElite, the $O(N_p)$ selection of the generation-best chromosome, is found to be better suited to the host than to the device.

0.33