CS Paper Radar

Symmetry-quotient Flatness and Generalization

Taiki Miyagawa

cs.LG stat.ML ·submitted 2026-09-10 06:28 UTC · v1
theory

Abstract

This paper develops a theorem-level pipeline in symmetry-quotient settings: quotient linear stability implies quotient flatness, quotient flatness implies input smoothness, and input smoothness yields generalization under local covering assumptions. Flatness is often associated with generalization, and Stochastic Gradient Descent (SGD) is frequently viewed as implicitly biased toward flat solutions. However, standard flatness measures are typically defined in the raw parameter space and are therefore not invariant under function-preserving symmetries such as positive rescaling. We develop a symmetry-aware theory of quotient flatness, quotient linear stability, input smoothness, and generalization on quotient spaces of neural-network parameters. For square loss and models equipped with function-preserving group actions, we define quotient flatness as the trace of the Hessian of the empirical loss on the regular quotient manifold. We show that quotient flatness controls input smoothness through a quotient-space analogue of the flatness-to-smoothness argument. We also prove that one-step mean-square quotient linear stability of the linearized SGD dynamics implies an explicit quotient-flatness bound in terms of the batch size and learning rate, and extend this analysis to higher-order tensor moments. Finally, under local covering and boundedness assumptions, we derive population generalization bounds in terms of quotient flatness and, consequently, in terms of quotient linear stability.

Comment
Accepted to 33rd International Conference on Neural Information Processing (ICONIP 2026) as an extended abstract. This version is the full paper

Closest papers

Nearest neighbours by meaning across the whole index.

Parameter-Level Attribution of Symmetry in Trained Networks Though Parameter-Wise Functional Sensitivity

cs.LG ·2026-08-25 ·Alan Muriithi, Vedanta Thapar, Torben Berndt
similarity 0.66
Abstract

When a network has learned a function with a known symmetry, can that symmetry be moved through the parametrisation---is there a motion in parameter space realising the group action in function space? We formulate this as a lifting problem for the realisation map $Φ:θ\mapsto f_θ$, and show that a smooth parameter-space action exists only if the tangent space to the function's symmetry orbit lies within the image of $\mathrm dΦ_θ$, whose columns are the \emph{functional sensitivities} of individual parameters. This condition is also sufficient for pointwise first-order lifting. Relaxing it in least squares yields two local parameter directions: one following the symmetry orbit, one descending towards the equivariant subspace, with residuals measuring what the parametrisation cannot reach. On a rotationally invariant classifier we find these directions induce their predicted function-space motion, but only locally: recomputed directions track the orbit and reduce the equivariance defect, while directions held fixed depart from both after training. The same holds for Hamiltonian neural networks trained on a rotationally symmetric potential, even though the architecture does not explicitly enforce the symmetry.

0.16

Discovering Symmetries in Neural Network Parameter Spaces

cs.LG ·2026-09-27 ·Bo Zhao, Nima Dehmamy, Robin Walters, Rose Yu
similarity 0.65
Abstract

Parameter space symmetries are important for understanding neural networks' loss landscape, training dynamics, and generalization. However, systematically identifying these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and characterize loss invariance and the group-action axioms through infinitesimal conditions, which provide objectives for jointly learning group generators and nonlinear action maps. Our framework systematically uncovers parameter symmetries, including previously unknown ones. To study larger networks, we establish conditions under which subnetwork symmetries extend to the full model. The same construction gives an explicit family of finite-batch symmetries, providing both analytical examples and a foundation for discovery through small subnetworks. Using the infinitesimal characterization and subnetwork construction, we implement a framework for automated discovery of parameter symmetries, and successfully uncovered symmetries in various architectures, including pretrained transformer models.

0.42

Reducing Symmetry Increase in Equivariant Neural Networks

cs.LG ·2026-08-12 ·Ning Lin, Jiacheng Cen, Anyi Li, Wenbing Huang, Hao Sun
top venuetheory similarity 0.62
Abstract

Equivariant Neural Networks (ENNs) have empowered numerous applications in scientific fields. Despite their remarkable capacity for representing geometric structures, ENNs suffer from degraded expressivity when processing symmetric inputs: the output representations are invariant to transformations that extend beyond the input's symmetries. The mathematical essence of this phenomenon is that a symmetric input, after being processed by an equivariant map, experiences an increase in symmetry. While prior research has documented symmetry increase in specific cases, a rigorous understanding of its underlying causes and general reduction strategies remains lacking. In this paper, we provide a detailed and in-depth characterization of symmetry increase together with a principled framework for its reduction: (i) For any given feature space and input symmetry group, we prove that the increased symmetry admits an infimum determined by the structure of the feature space; (ii) Building on this foundation, we develop a computable algorithm to derive this infimum, and propose practical guidelines for feature design to prevent harmful symmetry increases. (iii) Under standard regularity assumptions, we demonstrate that for most equivariant maps, our guidelines effectively reduce symmetry increase. To complement our theoretical findings, we provide visualizations and experiments on both synthetic datasets and the real-world QM9 dataset. The results validate our theoretical predictions.

0.64

Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking

cs.LG ·2026-10-08 ·Mohnish Harwani
similarity 0.62
Abstract

Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.

0.10

Symmetry Breaking at the Edge of Stability via Central Flow Analysis

cs.LG ·2026-11-04 ·Naima Elosegui Borras, Georgios Arvanitidis
similarity 0.61
Abstract

Continuous symmetries in neural networks lead to conserved quantities that are invariant under gradient flow, such as layer balancedness in linear and ReLU networks. Finite-step gradient descent need not preserve these laws, particularly at the edge of stability (EOS), where the iterates oscillate around a smoother trajectory, which can be described by the central flow dynamics (Cohen et al., 2025). We study how gradient flow conservation laws evolve under the central flow description of this regime. We show that the curvature correction induces a finite-dimensional drift determined by the oscillation covariance along critical parameter directions. We thus take the first steps to analyse which conservation laws remain and are broken at the EOS according to the central flow dynamics by analysing a scalar two-layer model, two-layer matrix factorisation, and a two-layer ReLU network, where the known conservation laws are shown to be complete (Marcotte et al., 2023).

0.13

Broken scale symmetries in undercomplete linear autoencoders

cs.LG q-bio.NCstat.ML ·2026-10-02 ·Farhad Pashakhanloo, Jacob A. Zavatone-Veth
published similarity 0.61
Abstract · NeurIPS 2026 Symmetry and Geometry in Neural Representations Workshop

Neural network loss landscapes have many symmetries, which are preserved by gradient flow but broken by finite-stepsize stochastic gradient descent (SGD). A canonical example of such a symmetry is scale in homogeneous networks: one can scale up the parameters in one layer and down in the next without changing the network output. Previous work has documented cases in which SGD breaks this symmetry in favor of balancing gradient noise or minimizing fluctuations. Here, we show that the solution geometry of undercomplete linear autoencoders instead selects a preferred sign for scale drift: on the PCA solution manifold, SGD favors large decoder weights. This directed scale drift occurs on a slow timescale, and its dynamics admit an analytically-tractable effective description. However, it cannot continue indefinitely: increasing scale eventually drives the dynamics towards a finite-stepsize stability boundary. The resulting solutions are sharper than a balanced baseline in the sense of the maximum eigenvalue of the loss Hessian, but different sharpness measures can move in opposing directions. Thus, undercomplete autoencoders give a concrete illustration of how loss geometry can convert residual gradient noise into directed motion along a manifold of functionally-equivalent solutions.

0.30

Learning Operators for Symmetry Discovery: From Lie Groups to Local Symmetry Geometry

cs.LG ·2026-10-12 ·Pavan Karjol, Scindhiya Laxmi, Prathosh AP
theory similarity 0.61
Abstract

Continuous symmetry discovery is typically formulated as a separate estimation problem for each function or dataset. We instead investigate whether the map from a function to its symmetry structure can be learned across a family of functions from finite observations. Using the infinitesimal differential characterization of continuous symmetries, we formulate this map as an inverse symmetry operator, analogous to an inverse PDE problem. For global Lie-group symmetries, we establish a representation-theoretic population characterization: the symmetry Lie algebra is the nullspace of a positive-semidefinite operator constructed from representation coefficients and the infinitesimal group action. We also show that the inverse symmetry operator is equivariant under coordinate changes through the adjoint action. This characterization separates the geometric symmetry map from finite-sample estimation, motivating a hybrid approach that amortizes statistically difficult components while retaining analytic structure. On SO(5) benchmarks, amortized equivariant estimation of harmonic coefficients improves symmetry recovery from sparse observations, whereas ordinary least squares suffices for SL(2,R). We extend this viewpoint to local symmetries, where symmetry directions form smoothly varying tangent distributions rather than a fixed Lie algebra. Under the Lie-bracket consistency condition, Frobenius' theorem guarantees local integral submanifolds along which observables remain unchanged. We recover this geometry using a hybrid estimator that learns observation weights while preserving analytic differential and geometric recovery. Across global, non-compact, local, and physically motivated synthetic benchmarks, our results support learning the statistically difficult finite-sample components while preserving the known geometric structure of symmetry recovery.

0.13

On Parameter Symmetries and Conservation Laws in Gradient Flow

cs.LG math.OC ·2026-09-28 ·Khang Nguyen, Guido Montúfar
theory similarity 0.61
Abstract

Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.

0.27