CS Paper Radar

Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation

Meng'en Qin, Yinchen Liu, Mingxuan Cui, Youlu Xing

cs.CV cs.LGstat.ML ·submitted 2026-09-16 17:43 UTC · v3

Abstract

Visual signals require compact yet sufficient representations for robust downstream prediction. Convolutional sparse coding (CSC) provides an explicit mechanism for suppressing redundant components while preserving signal content, but its sparsity coefficient is often manually selected during training. We propose a training-adaptive convolutional sparse coding framework for robust visual signal representation. Specifically, we unfold the CSC optimization with the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) and treat the sparsity coefficient as a differentiable variable jointly learned with the network parameters. From the information bottleneck perspective, this coefficient controls the trade-off between information retention and compression: the sparsity term promotes compact representations, while the reconstruction term together with task loss preserves task-relevant signal content. We further introduce a label-free post-training strategy that adjusts the compression strength for corrupted inputs with the main network parameters fixed. Experiments on CIFAR and ImageNet demonstrate competitive clean-data recognition and greatly improved robustness under different input perturbations.

Closest papers

Nearest neighbours by meaning across the whole index.

PE-CSNet: An equivariant network architecture with learnable patch-based sparse representation

cs.CV ·2026-08-11 ·Kai Li, Haitao Long, Bo Zhang, Haiwen Zhang, Zhi Zhou
similarity 0.68
Abstract · 35 pages, 10 figures. Under review

Compressive sensing (CS) enables accurate signal reconstruction from sparse measurements and is widely applied in medical imaging, remote sensing, and image compression. However, designing an effective, task-specific sparse transform and the corresponding optimization procedure for high-quality CS remains challenging. This process typically requires expert domain knowledge and laborious parameter tuning. To address this issue, we present a Patch-based Equivariant deep unrolling architecture, termed PE-CSNet, for accurate CS recovery. While traditional CS methods generally use predefined patch-based transform sparsity, we generalize this idea by incorporating learnable transform sparsity that adapts to the specific CS task through an optimization-driven process. Specifically, we first establish a generalized patch-based CS model, which we solve via a block coordinate descent (BCD) algorithm. The BCD solver is then unrolled into a deep neural network, where all parameters of both the CS model and solver are learned through end-to-end training. To improve data efficiency, we introduce a stochastic equivariant training strategy that exploits the patch-wise structure of the network, enabling PE-CSNet to learn effectively even from limited data. We further provide a simpler, parameter-shared version of PE-CSNet and briefly discuss its convergence as an iterative solver. For practical applications, the network uses stage-specific (non-shared) parameters to enhance its expressive power and thereby improve its performance. On the tasks of CS magnetic resonance imaging (CS-MRI) and CS coded diffraction patterns (CS-CDP), PE-CSNet achieves state-of-the-art accuracy with fast computational speed, outperforming traditional methods and existing deep unrolling methods.

0.12

Inference and learning in sparse autoencoders as natural gradient flow

cs.LG cs.AI ·2026-10-05 ·Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel, Jacob L. Yates +4 more
code/data similarity 0.64
Abstract · Code: https://github.com/hadivafaii/BeFOND

Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which features explain an input and learning the dictionary that represents them. Here, we unify inference and dictionary learning as natural-gradient flows on a shared variational free energy. We instantiate this framework as BeFOND, an encoder-free sparse coding model with closed-form inference and learning dynamics. We show how recurrent explaining away reduces interference between overlapping features, while Fisher preconditioning can compensate for the slow learning of rare features. On synthetic data, BeFOND improves dictionary recovery and rare-feature detection, with a growing advantage over amortized baselines as superposition increases. On language-model activations, it improves single-feature concept detection and selective intervention, outperforming pretrained reference SAEs with substantially less training data. Its feature quality continues to improve with dictionary width, whereas the evaluated baselines largely plateau. Together, these results show how improving inference and learning within a unified probabilistic framework can make better use of data and dictionary capacity to interpret and intervene on neural representations.

0.28

Separating Interpretability and Reconstruction in Sparse Autoencoders

cs.LG ·2026-11-18 ·Anh Tien Nguyen, Dimitris Samaras, Mahdi S. Hosseini
code/data similarity 0.64
Abstract · Code: https://github.com/AtlasAnalyticsLab/DualSAE

Sparse Autoencoders (SAEs) are effective tools for mechanistic interpretability by decomposing neural representations into sparse and interpretable features. However, a single sparse representation in traditional SAEs is responsible for both reconstruction fidelity and interpretability, which are not guaranteed to align. To separate these objectives, we propose DualSAE, a modular framework that augments standard SAEs with a plug-in Residual Branch. A standard SAE serves as the Sparse Branch, learning a sparse code that contains the dominant semantics of the input. The added Residual Branch models the non-semantic information omitted by the Sparse Branch. We ensure that the contributions of the two branches towards the overall reconstruction have minimal overlap through a decorrelation loss. We also adopt a two-stage training strategy that first stabilizes sparse representation learning before coupling with the Residual Branch. Across diverse encoder-SAE configurations on two vision datasets and two language corpora, DualSAE achieves mean relative improvements of approximately 29% in reconstruction fidelity and 34% in interpretability (measured by monosemanticity score). Moreover, with more interpretable representations, DualSAE yields consistent performance gains in the downstream applications of topic modeling, hypothesis generation, and LLM steering. Qualitative analysis further suggests that the Sparse Branch indeed preserves dominant semantic content, whereas the Residual Branch captures non-semantic information to improve reconstruction fidelity.

0.29

Bottlenecked semantic compression by representation reconstruction

cs.CV cs.AI ·2026-10-12 ·Xiaoya Tang, Shilong Liu, Daniel Smolyak, Xiwen Li, Quynh Nguyen +1 more
code/data similarity 0.63
Abstract

Vision Transformers incur high computational costs when processing long visual token sequences. Existing reduction methods typically rank tokens independently and discard those deemed unimportant or redundant. We propose a reconstruction-guided framework for visual token condensation (ReConV), which summarizes the complete visual sequence with a compact Summary Bank (SMB) and retains tokens whose information is not adequately captured by the SMB, as indicated by their reconstruction residuals. At each compression layer, a lightweight position-conditioned decoder reconstructs the current tokens, leaving shared information compressed in the SMB while explicitly preserving poorly reconstructed details. Experiments on image classification and multimodal understanding demonstrate favorable performance-efficiency trade-offs: ReConV surpasses the uncompressed baseline with 48\% fewer GMACs on ImageNet and enables reusable image-level representations for efficient multi-turn inference. Code is available at TokenDrop.

0.26

Video Compression with Graph-inspired Neural Representation

cs.CV ·2026-09-02 ·Changqi Wang, Ge Gao, Fan Zhang, Yue Li, Kai Zhang +2 more
similarity 0.62
Abstract · 12 pages, 6 Figures

Implicit Neural Representations (INR) provide a compact and content-adaptive paradigm for video compression, typically representing a video through shared network parameters and frame-indexed embeddings. Compared to conventional or autoencoder-based codecs, these approaches exploit temporal redundancy within videos in an implicit manner, which potentially results in sub-optimal compression performance. In this paper, we propose G-NeRV, a graph-inspired INR that explicitly improves temporal redundancy exploitation in the implicit latent space. Motivated by the total correlation principles in information theory, we construct a temporal neighborhood over frame embeddings and perform message passing to aggregate reusable information from neighboring frames through an adaptive gate controlling the injection of neighboring information. Inspired by the reference frame buffer in conventional video coding, a memory bank mechanism has been further designed to enable efficient temporal-neighbor retrieval under random frame-index sampling in INR training. This new representation model has been integrated into an advanced representation compression framework and compared with existing conventional and neural video codecs. The results show that the G-NeRV codec outperforms the state-of-the-art INR-based codec, NVRC, and the latest standard video codec, VVC VTM, by 8.86\% and 14.68\% (in BD-rate), respectively, measured by PSNR on the UVG dataset.

0.23

Beyond SAEs: Consistent Concept Discovery with Sparse Coding

cs.AI ·2026-10-16 ·Jinho Choi, Hyesu Lim, Jaegul Choo, Steffen Schneider
hot direction similarity 0.62
Abstract

Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. This is motivated by our synthetic analysis showing that coefficients from a single linear encoder pass are suboptimal compared with those obtained by per-sample optimization, suggesting that sparse coding is a reliable mechanism for dictionary learning. Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. These gains translate to state-of-the-art cross-seed consistency across four vision and language settings, where ConSDL improves dictionary and coefficient consistency by 17.6% and 35.9%, respectively, over the average of six SAE baselines.

0.42

DAC-Loop: Distillation-Aware Compression for Remote Sensing CNN

cs.LG ·2026-10-26 ·Rico Staedeli, Lorena Raichle, Konstantinos Tzevelekakis, Aron Asefaw, Damian Borth
published similarity 0.62
Abstract

While large-scale models achieve strong results in remote sensing scene classification, deploying them on edge devices remains difficult due to limited compute, memory, and energy budgets. Compact CNN backbones are therefore a practical choice in many real-world settings, offering efficient inference with well-optimized implementations on embedded hardware. To push this efficiency further, we investigate how far a knowledge-distilled student model can be compressed while preserving high accuracy. We propose a Distillation-Aware Compression Loop (DAC-Loop) that integrates (i) knowledge distillation applied both before and after structured channel pruning, and (ii) an iterative compression schedule that alternates structured pruning with unstructured sparsification. Across EuroSAT and NWPU-RESISC45, the DAC-Loop consistently improves the accuracy-compression trade-off relative to standard one-shot pruning and single-stage KD recovery, maintaining strong accuracy even beyond 90% parameter reduction. These results demonstrate that combining distillation and pruning in a unified iterative loop offers an effective path toward compact remote sensing models.

0.29

Compressive Sensing - Introduction and Relations to Deep Learning

cs.IT ·2026-08-25 ·Hung-Hsu Chou, Johannes Maly, Holger Rauhut
similarity 0.62
Abstract

Compressive sensing predicts that sparse vectors (signals) can be recovered from a small number of linear measurements via efficient algorithms. This finding, which dates back two decades, has triggered a paradigm shift in signal processing and initiated many developments both in practical signal-processing applications, such as medical imaging, radar, and astronomy, and on the theoretical side. More recently, seminal connections to the field of deep learning have led to further advances in the field, such as the use of unrolled neural networks for sparse recovery and the discovery that common training algorithms (variants of gradient descent) favor sparsity in overparameterized scenarios -- the so-called implicit bias phenomenon. This article gives an introduction to compressive sensing and outlines connections to deep learning. In particular, we will discuss generalization for neural networks generated by unrolling sparse recovery algorithms and implicit regularization for gradient descent applied to learning simplified linear neural networks.

0.06