CS Paper Radar

Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models

Chengzheyi Yao, Yongzhao Zhang, Yongding Tian

cs.LG ·submitted 2026-08-31 07:24 UTC · v1

Abstract

The loss landscape of Deep Neural Networks (DNNs) exhibits highly complex and non-convex properties. Recent studies have revealed the phenomenon of mode connectivity, demonstrating that independently trained network modes can be connected via a continuous low-loss path. However, existing mode connectivity research is predominantly confined to classifier-based models, leaving it an open question whether similar geometric properties exist in modern complex models. In this paper, we extend the boundaries of mode connectivity to generative and contrastive domains (specifically DDPM and NanoCLIP). Addressing the unique architecture of DDPM and CLIP, we propose an architecture-aware connection building algorithm. Extensive empirical results demonstrate for the first time that we successfully discover mode connectivity between independently trained DDPM and NanoCLIP modes. Our work provides a novel perspective for understanding the geometric properties of the loss landscapes in modern generative and contrastive models.

Closest papers

Nearest neighbours by meaning across the whole index.

Traversing the solution space of neural networks with Hessian Null Space Continuation

cs.LG q-bio.NCstat.ML ·2026-09-29 ·Ann Huang, Mitchell Ostrow, Zhouyang Lu, William T. Redman, Leo Kozachkov +1 more
code/data similarity 0.60
Abstract · 55 pages, 39 figures. Project page and code: https://ann-huang-0.github.io/Hessian-null-space-continuation/

On a single task, deep networks can learn many solutions, depending on their optimizer, training data, architecture, and hyperparameters. Many of these solutions are mode-connected: rather than isolated points in weight space, they are connected by low-loss regions. Yet how their internal computation varies within these regions is unknown. A parallel line of work has identified the degeneracy of neural representations: many networks reach similar training loss with distinct internal structures. However, it is unclear how these solutions are related in weight space. We unify these subfields and show for the first time that many different internal mechanisms exist within a local mode-connected region in weight space. To do so, we introduce Hessian Null Space Continuation (HNC), a scalable method that uses local curvature to traverse regions of weight space that preserve network function, and can be steered toward solutions with specified properties. In RNNs trained on a memory task, HNC reaches drastically different representations and dynamics with maintained behavior. In ImageNet-trained Vision Transformers, HNC finds representations that differ more from the original network than any independently trained model with a different architecture or objective. In reinforcement-learning agents, HNC uncovers a distinct navigation strategy at comparable return and exposes reward hacking in an AI Safety Gridworld. Finally, HNC measures the local geometry of the solution set, showing how model size and task complexity shape its dimension and functional sensitivity. Our results show that a surprisingly large amount of representational diversity exists near a single trained solution, unseen by standard gradient-based optimization. HNC identifies and quantifies this diversity, opening new possibilities for mechanistic understanding of solution spaces and for model merging, editing, and fine-tuning.

0.41

Mode Dynamics: A Framework for Neural Network Learning

cs.LG ·2026-10-29 ·Hossein Honarvar
similarity 0.57
Abstract · 32 pages, 17 figures, 5 tables

We introduce Mode Dynamics, a framework that connects coordinated parameter motion during neural network learning to changes in model outputs. Starting from a recorded parameter trajectory, we decompose its updates into collective modes and map each mode to its state-dependent output effect. The framework organizes this link through three mechanisms. Decomposition identifies collective modes in parameter motion and their time-varying amplitudes. Reconstruction tests whether the effects induced by those modes recover local and accumulated output change. Concentration quantifies how behavioral contribution is distributed across modes. Controlled scalar regression and next-token experiments directly test these mechanisms. They show that small sets of leading modes reconstruct behavior, induced effects depend on the training path, and modal complexity varies with task and data. We then evaluate the framework on held-out image and language-model outputs using CIFAR-100 autoencoding and a public Pythia-70M pretraining trajectory. Across these settings, the results connect compact modal subspaces of recorded parameter motion to evolving behavior.

0.20

From the Loss Landscape to Diverse Feature Learning in Neural Networks

cs.LG cs.AI ·2026-08-28 ·David Aram Yunis
similarity 0.57
Abstract · PhD dissertation. This article draws from arXiv:2408.11804

Over the course of the last decade, neural networks have grown from an academic curiosity to moving the markets of nations. Despite this explosion in both research and deployment, relatively little is understood about how they achieve the solutions they do. This is both scientifically relevant, and pressing for society. When neural networks make decisions across self-driving, construction, law, hiring and health, there have been and will continue to be unintended consequences. However, attempting to generalize the failures of the largest and most important production systems makes for a very difficult task. Yet signs of these failures exist at all scales of neural networks, so we should be able to study a much more tractable setting. All neural networks must undergo an optimization process, called training, to be useful. To a great degree, understanding neural networks is understanding their optimization: through what process and exposure to which data did they arrive at their results. Yet our knowledge on this topic as a field is quite imprecise. In particular, a curious phenomenon called mode connectivity, the ability to connect neural networks in the loss surface, defies explanation entirely. This dissertation elucidates, explains and exploits this special structure in the loss landscape...

0.25

The Tangled Web They Weave: Exploring Neural Representations with Directed Simplicial Complexes

cs.LG ·2026-10-31 ·Michael Bleher
published similarity 0.56
Abstract · 52 pages, 16 figures. Accepted at NeurReps 2026 (Symmetry and Geometry in Neural Representations)

Understanding how artificial neural networks (ANNs) represent and combine features internally is a central challenge in mechanistic interpretability. Existing neural circuit-discovery methods represent neuron interactions as pairwise edges in a graph. But distributed neural representations consist of large groups of neurons that are inherently linked by higher-order causal dependencies: suppressing a subgroup of neurons in concert may shift the activation of other neurons in the feature representation. We propose a method to construct filtered directed simplicial complexes (DSCs) that give a topological representation of such dependencies, built from the activation profiles of an ANN on a given set of inputs. Our approach treats ANNs as structural causal models and uses counterfactual ablations to test for causal links between groups of internal neurons within specific input contexts. As a proof of concept, we apply the approach to a pair of small multilayer perceptron (MLP) architectures in a synthetic two-dimensional quadrant classification task. In these examples, DSCs recover the designed feature pathways and coordinated class transitions, and the complexes of trained networks are higher-dimensional than those of their handcrafted references. As validation, we compare our method with graph-based circuit-discovery baselines and find that their 1-skeletons agree within the relations that a weight graph can represent. These results support DSCs as context-dependent descriptions of multi-neuron computation and may thus provide a novel, orthogonal framework to study the phenomenon of distributed feature representations and polysemanticity.

0.33

NanoNet: Ultra-Lightweight Network for MRI Brain Tumor Classification toward Edge AI

cs.CV ·2026-11-08 ·Tran Nguyen-Le, Huu-Thuan Huynh, Minh-Triet Tran, Trung-Nghia Le
benchmark similarity 0.56
Abstract

Automated brain tumor classification from magnetic resonance imaging (MRI) can support accessible medical image analysis, but conventional deep learning models often exceed the resource budgets of highly constrained edge-AI platforms. We present NanoNet, an ultra-lightweight convolutional neural network designed for resourceconstrained MRI classification and motivated by future edge-AI applications. Inspired by MobileNetV2, NanoNet substantially reduces network depth, channel capacity, expansion ratios, and final feature dimensionality, resulting in only five inverted residual blocks and 8,132 parameters. We further investigate dependency-aware structured pruning, knowledge distillation, and INT8 quantization-aware training for additional compression. Experiments on the BRISC 2025 benchmark show that NanoNet achieves 92.80% accuracy and 92.97% macro F1 with 8,132 parameters and 10.498 MFLOPs. Across three compression seeds, it exceeds the mean accuracy of 70%-pruned MobileNetV2 by 5.20 percentage points while using 60.2% fewer parameters and 9.8% less computation. Converted INT8 NanoNet achieves 89.70% accuracy and 90.22% macro F1 across three QAT seeds. Earlier TensorFlow Lite measurements indicate 27.7 KB of baseline model storage, motivating future edge-AI deployment.

0.12

SAM-on-the-Curve: Sharpness-Aware Mode Connectivity for Robust Weight-Space Interpolation

cs.LG ·2026-09-15 ·Alejandro Calatrava, Xu Zhang, Ren Wang
position similarity 0.55
Abstract

Deep neural networks that are independently trained to similar performance can be connected by low-loss parametric curves in weight space, a phenomenon known as Mode Connectivity (MC). This geometric property underpins practical techniques such as weight averaging, model ensembling, and model merging. We argue that low-loss connectivity is an incomplete geometric criterion: it controls loss only along a one-dimensional trajectory while leaving the surrounding weight-space neighborhood unconstrained, so the optimized curve may traverse sharp ridges that become fragile under distribution shift. We therefore reformulate mode connectivity as a neighborhood-robust path optimization problem, seeking a curve whose entire local neighborhood maintains low loss. We propose Sharp Mode Connectivity (SMC), which applies a first-order sharpness-aware approximation to the resulting minimax functional, enforcing flatness along the entire curve rather than only on it. We derive a practical optimization algorithm for connectivity paths under this sharpness-aware objective. Under severe blur corruptions from CIFAR-10-C, SMC achieves up to 6.09\% absolute accuracy improvement over standard MC. Remarkably, SMC produces negative loss barriers, meaning that models obtained at interior points of the optimized path can outperform the average endpoint loss. These results, validated across ResNet-18, VGG16-BN, and ViT-Tiny on CIFAR-10 and ImageNet-100, establish path-wise flatness as a practical principle for robust weight-space interpolation.

0.15

Connectome-to-Function: Conditional Generative Latent Representations for Reservoir Computing

cs.LG ·2026-09-05 ·Zhuolin Yu, Xingyu Liu, Yuanhao Jia, Yunhang Xiao, Hairuo Xue +2 more
similarity 0.54
Abstract · 9 pages, 4 figures. Supplementary material included

Connectomes, graph-level maps of neurons and their synaptic connections, provide a structural basis for understanding how brain circuits support function and computation. However, mapping connectome structure to computation remains difficult because these graphs are high-dimensional, sparse, and sensitive to local structural variation. Existing approaches often depend on hand-crafted structural descriptors or task-specific predictors, which limits their ability to represent connectomes in a form that is both generative and functionally meaningful. We propose a conditional generative latent framework that encodes connectome graphs into a compact structural space while using available node-level conditions to guide reconstruction and generation. From this space, the model can reconstruct observed connectivity with a mean edge-reconstruction AUC up to 0.910 and generate new candidate connectomes, enabling a unified analysis of graph structure and computational behavior. Using connectome-derived graphs as recurrent computational substrates, we found that the learned latent space captures functional variation across reservoir-computing experiments, with cross-validated $R^2$ values up to approximately 0.87. Interpretability analysis further revealed task-specific structural mechanisms: in our examples, memory performance is associated with reciprocal recurrent connectivity, whereas prediction and classification are more strongly associated with spectral properties of the recurrent network. These findings suggest an AI-for-science approach to linking neural connectivity to computation and provide a generative and interpretable basis for studying how distinct structural mechanisms shape computational capacity.

0.27

Mathematical Invariant-Enabled Topological Neural Networks for Molecular and Materials Property Prediction

cs.LG q-bio.BM ·2026-10-06 ·Yiming Ren, Xiang Liu, Mustafa Hajij, Pietro Liò, Guo-Wei Wei
similarity 0.54
Abstract

Existing molecular and materials learning approaches often rely on a limited set of structural representations, which may capture only selected aspects of complex three-dimensional structure. Here, we introduce mathematical invariant-enabled topological neural networks (MITNNs), a framework that represents complex structures through multiple complementary mathematical views and integrates them with topological neural architectures. MITNNs combine multiscale invariants from topology, spectral theory, commutative algebra, differential geometry, and discrete curvature, capturing complementary structural information from the same system. Systematic invariant-subset, architecture-subset, and ensemble analyses show that predictive performance depends on how mathematical representations and neural architectures are paired, with selected combinations outperforming individual models and the aggregation of all available components. Across protein-ligand binding, metal-organic framework properties, mutation-induced protein solubility, and molecular toxicity prediction, MITNN consistently outperforms existing methods. These results establish MITNN as a mathematically multimodal framework for scientific machine learning.

0.12