CS Paper Radar

Beyond Affine Transformations: A Soft Dominance Layer for Coordinate-Wise Neural Computation

Mariano Rivera

cs.LG cs.AI ·submitted 2026-09-30 18:38 UTC · v1

Abstract

This paper presents a preliminary study of an alternative to the affine transformation underlying conventional neural-network layers. In the proposed Soft Dominance Layer, each output unit compares input coordinates with a learnable reference vector and aggregates smooth inequality responses. A sigmoid relaxation makes the comparisons differentiable, while a sharpness parameter $α$ controls their transition toward hard threshold decisions. The aim is to examine the trainability and direct threshold interpretation of this primitive, not to claim a replacement for affine layers. In single-run MNIST experiments, the highest observed Soft Dominance accuracy is $0.9061$ without annealing and $0.9173$ with annealing, compared with $0.9827$ for the MLP baseline. These descriptive results do not establish reliable configuration rankings or a statistically supported annealing benefit. Learned reference vectors exhibit spatial structure, providing qualitative evidence of structured learning. Repeated-seed experiments and broader datasets are required to assess robustness and practical relevance beyond this proof of concept.

Comment
14 pages, 4 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Optimizing Physical Neural Networks through Device Variable Selection

cs.ET cs.ARcs.CVcs.LG ·2026-10-24 ·Sjoerd van den Belt, Manuel Escudero, Mohamadreza Zolfagharinejad, Wilfred G. van der Wiel, Nikolaos Alachiotis
similarity 0.58
Abstract

Physical neural networks (PNNs) leverage device physics in electronics, optics, and related domains to enable energy-efficient neural networks. Broken-isomorphism PNNs replace linear operations with multivariate physical transformations, modifying how input signals are mapped to device variables. We exploit network architectures emerging from these mappings by treating the routing between signals and input variables of physical devices as trainable parameters. This approach expands device-level tunability of the physical transformations while, at the system level, demonstrating improved expressivity of PNNs with negligible additional inference overhead. This discrete routing is optimized by training a categorical distribution using Gumbel-Softmax reparametrization. We evaluate this approach using experimentally calibrated differentiable models of multivariate electronic RNPUs for MNIST and CIFAR-10 classification. Using variable selection, we improve the MNIST classification accuracy for a small CNN model from 94.5% to 96.9% and the accuracy of CIFAR-10 from 62.8% to 68.0%. Our method uniquely leverages the induced biases of broken-isomorphism PNN architectures, paving the way towards more efficient physics-based neural network inference.

0.16

Input convex neural networks as surrogates in mathematical optimisation

cs.LG math.OC ·2026-08-10 ·Yu Liu, Jan Kronqvist, Fabricio Oliveira
similarity 0.57
Abstract

Embedding trained neural networks as surrogates within optimisation problems is an established practice in operations research. The prevailing approach uses feedforward neural networks (FNNs) with ReLU activations, whose piecewise-linear structure admits an exact but computationally intensive mixed-integer programming (MIP) reformulation as the networks grow. We advocate input convex neural networks (ICNNs) as structurally superior surrogates when the underlying response is approximately convex or concave. The convex architecture offers two computational advantages. First, the ICNN-MIP formulation tends to yield a tighter linear programming (LP) relaxation than its FNN-MIP counterpart, with no integrality gap in favourable instances. Second, ICNNs uniquely admit an LP-based reformulation via epigraph representations of ReLU activations, though this embedding is not always exact. When it is not, we exploit the properties of ICNNs to construct the strongest continuous relaxation over box domains, namely, the convex hull of the ICNN's graph, bounded below by the epigraph and above by the concave envelope; this construction is tractable under input convexity but hard for general ReLU networks. On this basis, we develop a branch-and-bound algorithm that builds this relaxation at each node, branches directly on input variables rather than intermediate variables as in MIP reformulations, and terminates at the root node whenever the epigraph embedding is valid. Case studies on humanitarian food aid, oil well routing, and wine blending show that ICNN surrogates match FNN accuracy and deliver gains in solve time and scalability, supporting ICNN as the default surrogate when the underlying function is convex, concave, or well-approximated as such.

0.40

Investigating Hyperparameter Optimization and Transferability for ES-HyperNEAT: A TPE Approach

cs.NE cs.AI ·2026-08-31 ·Romain Claret, Michael O'Neill, Paul Cotofrei, Kilian Stoffel
publishedhighly cited similarity 0.57
Abstract · Pages 1879 - 1887

Neuroevolution of Augmenting Topologies (NEAT) and its advanced version, Evolvable-Substrate HyperNEAT (ES-HyperNEAT), have shown great potential in developing neural networks. However, their effectiveness heavily depends on the selection of hyperparameters. This study investigates the optimization of ES-HyperNEAT hyperparameters using the Tree-structured Parzen Estimator (TPE) on the MNIST classification task, exploring a search space of over 3 billion potential combinations. TPE effectively navigates this vast space, significantly outperforming random search in terms of mean, median, and best accuracy. During the validation process, the best hyperparameter configuration found by TPE achieves an accuracy of 29.00% on MNIST, surpassing previous studies while using a smaller population size and fewer generations. The transferability of the optimized hyperparameters is explored in logic operations and Fashion-MNIST tasks, revealing successful transfer to the more complex Fashion-MNIST problem but limited to simpler logic operations. This study emphasizes a method to unlock the full potential of neuroevolutionary algorithms and provides insights into the hyperparameters' transferability across tasks of varying complexity.

0.59

Approaching Backpropagation Accuracy with Weight Perturbation on MNIST

cs.LG cs.NE ·2026-10-15 ·Patrick Inoue
similarity 0.57
Abstract

Weight perturbation (WP) estimates learning directions from random parameter perturbations and a scalar change in loss, without backpropagating derivatives. We examine how batch size, training duration, and estimator stabilization affect its attainable accuracy in fully connected MNIST networks. In the shallow configuration, large-batch training reaches approximately 97\% test accuracy, compared with an early backpropagation (BP) reference near 97.5--98\%. The same comparison exposes a substantial efficiency gap: under a ten-epoch budget, shallow WP reaches 90.2\%, whereas the reported BP reference reaches 97.4\%. Epoch-indexed curves show diminishing returns from increasing batch size beyond 4,000. Learning-rate reduction and averaging independent perturbation-based updates also stabilize deeper networks, although their recorded results do not establish BP-level accuracy. We distinguish these empirical findings from a finite-horizon approximation argument: under stated regularity conditions, population WP can approximate a fixed gradient-descent trajectory arbitrarily closely at any fixed finite depth. Positive classification margins then suffice to preserve the reference predictions on a finite test set with arbitrarily high probability. Together, the results identify near-BP accuracy as a configuration- and budget-dependent possibility, rather than evidence of equal efficiency or an unconditional accuracy guarantee across depth.

0.23

Breaking the Central Bias: Spatially Partitioned Experts for Coordinate-Based Neuroevolution

cs.NE cs.CVcs.LG ·2026-09-10 ·Romain Claret, Arthur Gygax, Michael O'Neill, Paul Cotofrei, Michael Palma Mendes +1 more
publishedbenchmark similarity 0.57
Abstract · 15 pages, 4 figures, 1 table. Author's accepted manuscript, accepted at the BIOMAP workshop (BIO-inspired Methods for Pattern Recognition) o…

Evolvable-Substrate HyperNEAT (ES-HyperNEAT), a bio-inspired indirect encoding that determines neuron placement and connection weights from spatial coordinates, exhibits a failure mode on MNIST as a diagnostic benchmark. Because input pixels map to a coordinate space centered at the origin, evolved networks converge on a small central cluster of input pixels, a spatial-concentration bias; prior work observed only 21% mean accuracy in this regime. Is this bias an optimization artifact or an architectural ceiling? Inspired by Mixture-of-Experts (MoE) principles, we partition the input into non-overlapping spatial segments, each assigned to a separately evolved specialist network. With 13 such experts, this design reaches 43% mean accuracy, a 106% relative improvement over the baseline. The architectural gain does not depend on data-driven aggregation: equal-weighted averaging, which uses no validation data, already yields a 70% improvement; the gain comes from partitioning, not the weighting. Receptive-field analysis shows the mechanism: partitioning forces evolution to discover features across the entire image, expanding active pixel coverage from 4% to 79%. Absolute accuracy stays below gradient-trained baselines, but the relative gain points to central bias, not the evolutionary search. Two tools are designed to generalize beyond MNIST: a receptive-field diagnostic for silent input-coverage collapse, and a spatial-partitioning remedy that restores coverage.

0.66

Beyond a Single Sign: BiTE for Branch-Preserving Binary Threshold Expansion

cs.LG ·2026-11-12 ·Siddharth Gupta, Mikail Yayla, Akash Kumar
similarity 0.56
Abstract · 8 pages, 5 figures

Binary neural networks (BNNs) reduce model storage and replace many arithmetic operations with efficient bitwise computations, but the single-sign activation limits their representational capacity. Each preactivation direction is compared with one threshold and reduced to one of two values, discarding variation within each sign region. We propose \textbf{BiTE} (Branch-Preserving Binary Threshold Expansion), which evaluates each feature direction against $K$ ordered thresholds and retains the resulting binary responses as separate inputs to the following learned operator. BiTE can distinguish up to $K+1$ ordered activation states while allowing the following operator to learn a separate transformation for each threshold response. It also enables a controllable trade-off between the number of independently constructed feature directions and the number of ordered responses extracted from each direction. On BinaryViT, BiTE improves mean final-epoch Top-1 accuracy from $74.60\pm0.30\%$ to $75.83\pm0.15\%$ on CIFAR-10 and from $42.80\pm0.46\%$ to $44.07\pm0.32\%$ on CIFAR-100, with nearly unchanged trainable parameter counts. A matched uniform-collapse control reaches $75.33\pm0.15\%$ on CIFAR-10, supporting the value of retaining the threshold responses separately relative to the evaluated scalar collapse. Binary VGG-16 experiments further show a favorable observed parameter--accuracy trade-off, including a configuration that achieves $0.64$ percentage points higher mean accuracy with $21.8\%$ fewer parameters than conventional width scaling. Overall, these results support branch-preserving threshold expansion as an effective way to improve binary representations and the observed parameter--accuracy trade-off.

0.18

Training Mixed-Monotone Neural Networks for Verification Without Set Propagation

cs.LG ·2026-10-22 ·Jonas Fingerhut, Lukas Koller, Matthias Althoff
theory similarity 0.55
Abstract · 20 pages, 3 figures, 9 tables

Most formal verification approaches for neural networks conservatively enclose the output set of the neural network. However, computing a tight output enclosure is in general computationally hard. We sidestep this challenge with a novel neural network architecture that realizes mixed-monotone functions as the difference of monotone subnetworks. By utilizing the theory on mixed-monotone systems, this reduces the computation of a tight output enclosure from complicated set propagation to two simple forward passes of opposite corners of an input interval. Moreover, because the size of the resulting enclosure is a closed-form function of the weights of the neural network, we can directly minimize it during training. We prove a lower bound on the size of any such enclosure, given by the total variation of the represented function generalized to several inputs, which makes it possible to compare the tightness of the enclosure against the optimum. Our evaluation on regression and dynamical-system benchmarks demonstrates that our training reduces the enclosure size to within 0.5% of the lower bound without sacrificing prediction accuracy. Computing our enclosure is orders of magnitude faster than CROWN and $α$-CROWN and on deep networks it is also tighter than zonotope propagation and CROWN, while its size exceeds that of $α$-CROWN by only 5.1%.

0.18

Structure-Preserving Neural Approximation of Convex, Semiconvex, and Quasiconvex Functions

cs.LG math.OC ·2026-11-18 ·Yeachan Park, Yeoneung Kim
similarity 0.55
Abstract

Input convex neural networks provide a natural way to enforce convexity globally through architectural constraints. We study how this structure-preserving approximation framework extends to semiconvex, quasiconvex, and convex transformable functions. For $C^{1,1}$ convex functions on compact convex domains, we establish the sharp $W^{1,p}$ minimax rate $m^{-1/d}$ under max-affine approximation with $m$ affine pieces, and show that every max-affine function admits an exact bounded-width ICNN representation with complexity linear in $m$. The same $W^{1,p}$ approximation rate extends to semiconvex functions through quadratic lifting and to convex transformable functions through monotone composition. For general Lipschitz quasiconvex functions, we develop a quantitative finite-probe approximation theory based on monotone responses to affine projections and obtain a structure-preserving IqCNN with an $L^\infty$ approximation guarantee. We further show that these approximation results extend to ReLU-transferable smooth activations, including Softplus and ELU. Numerical experiments on PDE-based approximation problems compare the proposed architectures with parameter-matched MLPs and validation-tuned shape penalties, demonstrating global structural validity without penalty tuning while maintaining competitive approximation accuracy. Overall, the results identify recoverable convex latent structure as the key mechanism enabling gradient-level structure-preserving approximation beyond convexity.

0.28