CS Paper Radar

Neural Ideals and Neural Codes: An Algebraic Framework for Neural Network Classification and Feature Interpretation

Venkata Subbaiah Yerrapati, Rahul Dixit, Ajay Kumar Shukla

cs.LG math.AC ·submitted 2026-08-20 03:51 UTC · v1
theorycode/data

Abstract

Understanding the features captured by the hidden layers of neural networks is a fundamental challenge in machine learning, despite their widespread success across various classification problems. In this work, we propose an algebraic framework for examining neural networks that model classification problems. Certain results, such as the correspondence between the neural network and neural ideals, algorithms for computing the neural ideals, and a stabilization theorem that enables approximation of the neural ideals, are first established. As an application to the framework, we present algorithms to identify and interpret the features captured by each hidden-layer neuron. Along with these theoretical developments, the practical performance has been demonstrated on the MNIST digit dataset, and the results highlight the pivotal role of neural ideals as a mathematical and computational tool for analyzing the features captured by neural networks. Further, we develop an interactive software that builds on the presented framework to visualize the features captured by each neuron. This tool is available at https://github.com/yvs1967/neural-network-representation-explorer

Closest papers

Nearest neighbours by meaning across the whole index.

The Tangled Web They Weave: Exploring Neural Representations with Directed Simplicial Complexes

cs.LG ·2026-10-31 ·Michael Bleher
published similarity 0.58
Abstract · 52 pages, 16 figures. Accepted at NeurReps 2026 (Symmetry and Geometry in Neural Representations)

Understanding how artificial neural networks (ANNs) represent and combine features internally is a central challenge in mechanistic interpretability. Existing neural circuit-discovery methods represent neuron interactions as pairwise edges in a graph. But distributed neural representations consist of large groups of neurons that are inherently linked by higher-order causal dependencies: suppressing a subgroup of neurons in concert may shift the activation of other neurons in the feature representation. We propose a method to construct filtered directed simplicial complexes (DSCs) that give a topological representation of such dependencies, built from the activation profiles of an ANN on a given set of inputs. Our approach treats ANNs as structural causal models and uses counterfactual ablations to test for causal links between groups of internal neurons within specific input contexts. As a proof of concept, we apply the approach to a pair of small multilayer perceptron (MLP) architectures in a synthetic two-dimensional quadrant classification task. In these examples, DSCs recover the designed feature pathways and coordinated class transitions, and the complexes of trained networks are higher-dimensional than those of their handcrafted references. As validation, we compare our method with graph-based circuit-discovery baselines and find that their 1-skeletons agree within the relations that a weight graph can represent. These results support DSCs as context-dependent descriptions of multi-neuron computation and may thus provide a novel, orthogonal framework to study the phenomenon of distributed feature representations and polysemanticity.

0.33

A Geometrical Approach to Constructing Neural Networks for Perfect Training-set Classification

cs.LG ·2026-10-12 ·Akos Hajnal
similarity 0.58
Abstract · 8 pages, 1 figure, 2 tables

This paper proposes a method for constructing a neural network that achieves perfect classification of all samples in a given training set, without taking regularization into account. The network is constructed on-demand, with its layer sizes and neuron parameters determined through a geometrical analysis of the input data, in which separating hyperplanes are created and mapped to neurons with step activation functions. The proposed architecture consists of two hidden layers with distinct roles: inter-class separation and transformation of the data into a linearly separable representation. Inter-class separation requires every bi-chromatic pair of samples belonging to different classes to be separated by at least one hyperplane. We show that when this separation is achieved through a hierarchical partitioning of the data, the leaf nodes of the oblique decision tree, obtained as a byproduct, directly define the neurons of the linear separability layer, whose size is only one neuron greater than that of the inter-class separation layer. Experiments on the MNIST dataset demonstrate that the proposed method achieves perfect training-set classification, and compact models. The direct correspondence between the decision tree and network neurons enhances interpretability and explainability, while the construction procedure offers opportunities for parallel implementation and improved scalability.

0.29

Łukasiewicz Neural Networks Extended: Residual Architectures and Crystallization Strategies for Interpretable Rule Extraction

cs.LG cs.AI ·2026-09-26 ·Carlos Leandro
theory similarity 0.57
Abstract · 21 pages

A feed-forward neural network whose weights are integers and whose activation is the truncated identity implements, neuron by neuron, the connectives of Łukasiewicz many-valued logic. This exact correspondence --- established theoretically by Castro and Trillas and developed into a training algorithm by Leandro --- enables \emph{symbolic knowledge extraction}: training produces not a black-box model but a logical formula. Two obstacles have limited the approach to shallow architectures and small datasets: crystallization (forcing weights to integers) succeeds only probabilistically under the original Levenberg--Marquardt training scheme, and the theoretical guarantees break down as networks grow deeper. This paper addresses both obstacles. First, we prove that \emph{residual connections} (skip connections of the kind used in ResNets) extend Łukasiewicz neural networks to arbitrary depth while preserving the symbolic correspondence \emph{at merge neurons} by construction: merge neurons in a Łukasiewicz residual block automatically satisfy the neuron-classification proposition, regardless of the inner layer weights; inner-layer neurons are trained toward representability by the crystallization strategy. Second, we analyse three crystallization strategies --- Levenberg--Marquardt (corrected), straight-through estimation (STE), and proximal regularization --- characterizing their theoretical guarantees, failure modes, and interpretability trade-offs.

0.21

The Emergent Symbolic Structure of Artificial Neural Networks

cs.CL cs.AI ·2026-08-30 ·R. Thomas McCoy, Paul Soulos, Tal Linzen, Paul Smolensky
similarity 0.56
Abstract · 30 pages, plus 29 pages of references and appendices

Modern systems in artificial intelligence (AI) somehow excel in domains for which they seem poorly suited. Intelligence has traditionally been modeled as operating over structured combinations of symbols, such as logical formulas. However, the strongest modern AI systems are based on neural networks, which instead represent information in continuous vectors. Vectors seem inadequate for capturing the structure of language, logic, and other cognitive domains, yet neural networks achieve impressive performance in these areas. How do they do it? In this work, we propose a potential answer: Despite appearances, perhaps the internal representations of neural networks implicitly realize symbolic structure. In support of this hypothesis, we show that the vector representations of a variety of neural networks can be closely approximated with symbolic structures: we can replace the network's entire representation-generating process with a closed-form equation instantiating a symbolic structure, and the network's behavior remains largely unchanged. This finding holds for both small-scale neural networks trained to manipulate lists as well as large language models (LLMs) operating in four domains that are central in symbolic traditions: arithmetic, logic, computer code, and language. Further, our symbolic approximation allows us to modify an LLM's behavior in targeted ways via precise interventions on its internal representations, showing that the LLM's behavior is reliant on the symbolic structures we have identified. This work provides a potential way to reconcile longstanding symbolic conceptions of intelligence with the vector-based nature of modern AI.

0.45

Neuron Activation-based Computation of Logical Explanations for Deep Neural Networks

cs.LO cs.LG ·2026-09-12 ·Tomáš Kolárik, Faezeh Labbaf, Fabrizio Leopardi, Grigory Fedyukovich, Michael Wand +1 more
similarity 0.56
Abstract

Formal explainability of classifying neural networks (NNs) is an active area of research, providing explanations with provable guarantees of the classification within continuous regions of the input feature space. However, the existing techniques are either limited to individual input features without guarantees on their relations or the provided solutions fail to scale to deep architectures. This paper addresses these issues by introducing a flexible symbolic framework for an efficient, guided computation of explanations of the NN behavior, parametrized by the activations of internal neurons, and using logical engines such as SMT solvers. Unlike prior methods that rely on specialized NN verifiers, our method yields explanations that are not restricted in shape. Our algorithm is implementable on top of a general-purpose logical solver, isolating the NN-specific encoding from the algorithmic framework. We experimented with a wide range of benchmarks from the domains of image recognition and medicine, illustrating the advantages of the new method, particularly in computational efficiency. Notably, our approach enables logical explanation of deep networks not amenable to prior logic-based methods.

0.24

Feature Superposition in Neural Networks: From Theory to Practice

cs.LG ·2026-09-06 ·Dai Shi, Xiaoyu Li, Andi Han, José Miguel Hernández-Lobato
code/data similarity 0.54
Abstract

Superposition refers to neural networks representing more features than they have dimensions. It was proposed to explain polysemantic neurons, which respond to several features, and also other phenomena, such as neural scaling laws and adversarial vulnerability. A feature stored in superposition is useful to the extent that its value can be read back despite interference from the other features. The theory of superposition mainly asks how wide a layer must be for the features to be read back, and by which decoder. In theoretical models, a single linear map that reads back every combination of nonzero features needs a width nearly quadratic in their number, while sparse optimization can succeed with a width linear in that number. In practice, one may hope to interpret a trained network by recovering its features in the same way. However, its features are usually unknown, and thus they are estimated from its activations or weights by recovery methods, such as sparse autoencoders and transcoders. In this work, we organize the theory of superposition by the decoder class and its practice by the strength of the claims about a feature supported by each method's results. On the theory side, we review the lower bounds on the overlap between feature directions, the conditions under which trained toy models share dimensions among features, and the separation between storing features and computing new features. On the practice side, we categorize recovery methods into four families, from probes for a named feature to decompositions of the trained weights. We then compare their evaluations, failure modes, and applications, and grade the evidence on one scale of claims about a feature. Finally, we assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks. Project page: https://eethanshi.github.io/awesome-superposition/

0.28

Revenge of Monosemanticity: Neuron Specialization as a New Form of Feature Learning in MLPs

cs.LG stat.ML ·2026-08-25 ·Amirhesam Abedsoltan, Enric Boix-Adsera, Fivos Kalogiannis, Mikhail Belkin
similarity 0.54
Abstract

Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a global low-dimensional representation. We show that this picture is incomplete. In regression problems with clustered data, we demonstrate that multilayer perceptrons (MLPs) develop specialized neurons: individual neurons become strongly aligned with specific features relevant to a particular region of the input space. A substantial fraction of these neurons exhibit monosemanticity: they activate predominantly on inputs from the corresponding cluster. Thus, rather than learning a single global low-dimensional representation, MLPs learn a collection of local low-dimensional representations. Furthermore, we show that this ability to specialize gives MLPs a provable data-efficiency advantage over feature-learning methods based on a global low-dimensional representation.

0.21

Supervised Hebbian Class Networks: Learning Without Backpropagation

cs.LG ·2026-11-18 ·John Loverich
similarity 0.53
Abstract

This paper introduces Supervised Hebbian Class Networks as a method for training multi- layer perceptrons and convolutional networks without gradient calculations or backpropa- gation. Each neuron in a learned layer is assigned a class, while connections follow the usual fully connected or convolutional pattern. The method combines a class-based supervision rule shared across layers with a Lion optimizer adapted to Hebbian activity products. All layers train together throughout the run, and their activities provide class predictions di- rectly, without a separately trained classifier. The experiments test the method on MNIST with a multilayer perceptron and on CIFAR-10 with a convolutional networ

0.24