CS Paper Radar

FM-Mamba: An Efficient Native 2D Selective State Space Model with Stability-Aware Discretization

Shengyuan Hu, Yiyong Sun, Weihang Shan, Yongzhe Chang

eess.SY cs.AIcs.CV ·submitted 2026-11-17 01:40 UTC · v1
code/data

Abstract

When applied to two-dimensional features such as images, Mamba-based methods either flatten the feature map into 1D sequences and fuse multiple scans, which breaks spatial adjacency and multiplies computation, or recurse directly on the 2D grid but rescale the recurrence with a fixed input-independent factor and aggregate only the predecessors on a single anti-diagonal, leaving the discrete step unconstrained by 2D stability theory. We propose \textbf{FM-Mamba}, a native 2D selective state space model built on the general Fornasini--Marchesini (FM) model. Its coupled recurrence evolves along both spatial axes; cross-axis coupling induces a discrete diagonal-predecessor term $\tilde A_0$, and two independent input-dependent step sizes $Δ_1(i,j)$, $Δ_2(i,j)$ govern propagation along the two axes, so that the coefficients adapt per position and per direction. A normalized ratio $r_k=a_{0,k}/(a_{1,k}a_{2,k})$ yields a closed-form piecewise bound on the two step sizes; when a configuration falls outside the stable interval the steps are projected onto this bound, and the criterion is re-verified per position after training, which suppresses systematic local amplification without global damping. The recurrence is executed by a dual-diagonal wavefront scan: each update gathers information from two anti-diagonals, and the input enters the current state through a single projection, so that neither training nor inference flattens the feature map. Code is available at https://github.com/BitSsm/FM-Mamba-Runtime

Closest papers

Nearest neighbours by meaning across the whole index.

Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

cs.LG ·2026-09-21 ·Julien Siems, Riccardo Grazzi, Korbinian Pöppel, Jaisidh Singh, Arber Zela +6 more
code/data similarity 0.58
Abstract

Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition. We show that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate. This requires extending the parameter ranges of KDA by combining two existing range extensions: allowing gates in $[-1,1]$ and the delta-rule coefficient $β$ in $[0,2]$. We call the resulting model Complex KDA (CKDA). It preserves KDA's stability and efficiency, with transitions that remain diagonal-plus-rank-one and non-expansive, while reaching the state-tracking expressivity of DeltaProduct$_2$. We characterize the expressivity of CKDA and prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. A single CKDA layer can track every finite group isomorphic to a subgroup of $\mathrm{SO}(3)$, and many state-tracking results use one fewer layer for CKDA compared to other diagonal-plus-rank-one Linear RNNs. Empirically, combining both extensions yields the strongest length extrapolation among tested KDA range settings on $S_3$, $S_4$, and periodic audio continuation. In language modeling, CKDA outperforms Transformers and other linear RNNs, obtains similar results to a KDA baseline, and shows promising scaling behavior. Our code is open source at https://github.com/OpenEuroLLM/ComplexKDA and our models are available at https://huggingface.co/collections/openeurollm/complexkda.

0.61

Riccati State Space Models: Non-iterative Parallelization for Nonlinear Sequence Modeling

cs.LG cs.AI ·2026-09-28 ·Mónika Farsang, Ramin Hasani, Daniela Rus, Radu Grosu
similarity 0.54
Abstract

State space models (SSMs) achieve efficient sequence processing because their affine state updates are closed under composition and can therefore be evaluated with an associative parallel scan. Nonlinear recurrent models can provide richer, state-dependent dynamics, but generally lose this compositional structure: parallel evaluation then requires iterative methods that repeatedly linearize and scan the recurrence. We ask, what state-dependent nonlinear dynamics can be designed to remain exactly composable? We answer by introducing RiccatiSSM, a nonlinear SSM, in which each state dimension follows an input-conditioned Riccati differential equation. Its quadratic state dependence makes the local Jacobian explicitly state-dependent, while its exact per-step flow under piecewise-constant inputs is a Möbius transformation. Since Möbius maps are closed under composition and compose through $2\times 2$ matrix multiplication, the complete nonlinear state trajectory can be evaluated exactly with a single associative parallel scan, without iterative linearization. We further derive a constrained parameterization that ensures bounded, contractive dynamics, and avoids poles in the fractional-linear state update. Across long-sequence classification, regression, and forecasting tasks, RiccatiSSM achieves competitive predictive performance while reducing runtime by $22{-}33\%$ compared to the nonlinear LrcSSM under matched architectures. These results demonstrate that state-dependent nonlinear dynamics can retain exact composability and be evaluated efficiently within a single parallel scan.

0.23

Diagonal Equivalence in State Space Models: Characterization, Retrieval Rank, and Selective Extensions

cs.LG math.OCstat.ML ·2026-11-02 ·Dennis Rochau
code/data similarity 0.53
Abstract · 81 pages, 6 figures, 33 tables. Code: https://github.com/d-rochau/OrbitSSM

We classify real and complex state-space models (SSMs) with linear transitions, distinguishing fixed-basis diagonal equivalence from input-output equivalence and tracking coefficient-conditioning windows under coordinate changes. We characterize diagonal equivalence through commuting transition families whose members are diagonalizable over the chosen field. With time-shared token-local matrix maps, we identify this class's state-mediated kernels with unnormalized scalar-mode linear attention with multiplicative masks. Mamba 3's multiple-input, multiple-output (MIMO) construction lifts one value per local interface; SSD-MIMO routes independent values for local retrieval rank above one. HybridMamba combines Mamba 1's per-coordinate decay, Mamba 2-style heads and Mamba 3's complex rotations and two-endpoint writes, mixing real and rotating state blocks with different decay laws. They recover Mamba 1-3 single-input, single-output (SISO) configurations as exact special cases. On five large H100 bfloat16 workloads, our fused GPU kernels improve geometric-mean training time over official Mamba 1/3 in matched SISO recoveries, with Mamba 2 near parity at SSM-core scope. Memory tradeoffs depend on the baseline and metric: saved-for-backward versus peak allocation. In small-model multi-token recall experiments, both designs outperform Mamba 1-3 on the hardest tier; higher routing ranks perform best in gated or phase-diverse configurations. In single-seed language models with about 500M parameters, both improve on Mamba 1/2 in test loss at 2B tokens, but the routing gains do not translate into lower language-model loss. Our best configurations approach Mamba 3 SISO through 10B tokens while remaining behind it and Mamba 3 MIMO in language-model loss.

0.46

State-Space Unlearning for Non-Stationary Bias in Land Surface Forecasting

cs.LG ·2026-09-30 ·Anidipta Pal
code/data similarity 0.51
Abstract

Operational land surface forecasting systems built on Mamba-family Structured State Space Models absorb non-stationary confounding events (unrecorded irrigation booms, dam-operation shifts, sensor recalibrations) into their state-transition matrices, silently biasing NDVI, LST, and crop phenology predictions long after the physical cause ends. This paper introduces SSU-LSF (State-Space Unlearning for Land Surface Forecasting), the first machine-unlearning framework purpose-built for geoscientific Mamba-based SSMs. We develop EKFac influence functions specialized to the Mamba state matrices via a closed-form matrix-exponential gradient, use spectral-radius-weighted elbow thresholding to localize a temporal confounding footprint $Φ$, and apply Hessian-free projected gradient ascent within a KL-divergence trust region augmented by spatial total-variation (TV) regularization. Proposition 1 establishes that residual confounding is bounded by $\mathcal{O}\big((1-ρ(\bar{A})^{T_c})/((1-ρ(\bar{A}))μ)\big)$, which grows with the window length $T_c$. Across three heterogeneous benchmarks and eleven baselines, SSU-LSF achieves confounding reduction rates of $0.773$ (CropHarvest), $0.821$ (NDVI-LST), and $0.859$ (ERA5), with worst-case clean-domain RMSE degradation of $4.2\%$ on ERA5, converging in 3--5 epochs at $8.4\times$ lower GPU-cost per unlearning request than full retraining. Code: https://github.com/Anidipta/SSU-LSF

0.50

Balanced State Subspaces with Spectral Rank Allocation for Mamba-2 State Compression

cs.LG cs.AIcs.CLstat.AP+1 ·2026-10-27 ·Shang Tingyu, Cao Wenfei, Han Zhi
similarity 0.50
Abstract · 18 pages, 4 figures

Mamba-2 compresses all context into a fixed-size recurrent state that dominates decoding memory and bandwidth. Existing compressors quantize state values or prune coordinates with per-channel scores. Trained states are nearly rank-one and bias-like, aligned with the mean write; but the output's energy spans an order of magnitude more directions. Per-channel scores see only the diagonal of the write--read operator, so they cannot select the subspace that governs the input--output map, which is a rotation of the coordinates. We therefore propose \method{}, which keeps that subspace by applying balanced truncation exactly via Mamba-2's scalar-identity transition. Layers differ just as sharply: most are bias-dominated, while the few that carry long-range associations are the flattest and need most of their state. We therefore introduce \spar{}, which redistributes a fixed total rank budget across layers according to the Hankel spectrum. On Mamba-2 checkpoints from 130M to 2.7B, \method{}+\spar{} halves the state at an average cost of $0.3$ WikiText-2 perplexity points, against $1.6$ for GHOST, while retaining $97\%$ of the dense model's five-digit in-context recall. At a quarter of the state, giving every layer the same rank keeps $5\%$ of that recall and \spar{} $77\%$, for a $7\%$ perplexity cost.

0.22

Distributed Learning with Selective State Space Models: Architecture-Aware Convergence Analysis

cs.LG cs.AImath.OC ·2026-10-02 ·Adam Piaseczny, Md Kamran Chowdhury Shisher, Shiqiang Wang, Christopher G. Brinton
similarity 0.49
Abstract

Modern state space models (SSMs), such as Mamba2, provide a compelling alternative to transformers by combining linear-time sequence modeling with recurrent state-space dynamics. However, the behavior of SSMs in distributed learning settings remains poorly understood. In particular, the existing standard federated learning methods are largely architecture-agnostic, and do not account for the stability, selectivity, and state-space parameterization that characterize modern selective SSMs. To address this, we derive architecture-aware gradient and smoothness bounds for single- and multi-layer selective SSMs, and convergence bounds for FedAvg and FedProx, characterizing how recurrent stability, input-dependent discretization, and state projection norms affect federated optimization. We then numerically validate the single-layer bounds on sequences generated by a teacher SSM, using a learner that follows the analyzed recurrence. We use this analysis to formulate expectations about the effects of local training and client heterogeneity, and examine these expectations by comparing nine federated learning algorithms on Mamba2 language modeling across six text domains. These experiments illustrate how SSM-specific bounds can provide a basis for interpreting the behavior of practical federated learning algorithms.

0.48

The Attention Within: Consensus Dynamics in Selective State Space Models

cs.LG cs.AIeess.SYmath.DS+1 ·2026-09-16 ·João Pedro Silvestre, Álvaro Rodríguez Abella, Paulo Tabuada
similarity 0.48
Abstract

Selective state space models (SSMs) have recently emerged as a compelling alternative to transformers, combining competitive performance with substantially improved inference efficiency. At each SSM layer, a sequence of hidden states are propagated by a recurrence, mixing information of different tokens. Despite using a different mechanism, this mixing plays a role analogous to attention in transformers. In fact, recent works have shown that the two architectures may be closer than they first appear, as this recurrence admits a formulation akin to linear attention. In transformers, attention is known to drive the tokens to cluster, i.e., to reach consensus, collapsing in the limit to a single direction. Thus, we ask: does the recurrence at the core of SSMs drive the tokens to consensus, as attention does in transformers? To answer this question, we take a dynamical systems perspective on SSMs, modeling the evolution of tokens across layers as an ordinary differential equation. By exploiting input-to-state stability arguments, we establish local exponential stability of the consensus equilibria and characterize their domain of attraction for time-varying weight matrices, a setting not addressed by previous results. We thereby show that the resemblance between SSMs and transformers does run deeper: the recurrence at the core of SSMs aggregates tokens just as attention does. Numerical experiments on a pretrained Mamba-2 model point to the output gate as the component that regulates the extent of this consensus, preventing the tokens from reaching it in full.

0.45

MambaLIFT: Reference-Based Mechanism-Aware Attribution for VMamba

cs.AI cs.CVeess.IV ·2026-11-19 ·Jintu Kakati, Niladri B. Puhan
hot direction similarity 0.48
Abstract

Attributing predictions of visual state-space models remains challenging because input-dependent selective dynamics couple feature selection, recurrent state propagation, and shared computational pathways. We introduce MambaLIFT, a reference-based, multiplier-first attribution framework for VMamba that propagates contributions through its computational graph using interaction-aware rules for selective-scan injection, recurrent memory, LayerNorm, and shared pathways. MambaLIFT decomposes the reference-relative class-logit difference, \(f_c(x)-f_c(x^0)\), by spatial position, VSS block, scan direction, and computational mechanism. Experiments on ImageNet-1K evaluate VMamba-T and VMamba-B, with hyperparameters selected on a calibration set and faithfulness assessed on a disjoint held-out evaluation set. MambaLIFT achieves numerical conservation to floating-point precision across audited attribution interfaces in both configurations. Under mean-colour pixel flipping, it achieves significantly higher mean faithfulness scores than MambaLRP on both configurations and the highest scores among the evaluated methods, including Integrated Gradients with matched references. Under blurred replacement, MambaLIFT is non-inferior to MambaLRP on VMamba-T and equivalent on VMamba-B, indicating that its advantage depends on the replacement protocol. Ablations on VMamba-T show that interaction allocation, temporal propagation, and pathway composition affect attribution maps. Mechanism-level analyses further show that exact relevance conservation does not imply additive effects under finite interventions: the mechanism decomposition accounts for the reference-relative logit difference but does not estimate intervention effects or establish causal validity.

0.36