CS Paper Radar

MulClip: A table-free activation function with monotonic and folding regimes for reservoir computing on microcontrollers

Guren Matsumura, Masato Takemoto, Shinzo Koyama

cs.LG cs.NE ·submitted 2026-10-29 04:32 UTC · v1

Abstract

This letter proposes MulClip, a lightweight activation function for reservoir computing (RC) on microcontrollers (MCUs). In a ring reservoir, a representative sparse RC topology, multiply-accumulate operations are few, and evaluating tanh accounted for 75% of the state-update time on a Cortex-M4. MulClip introduces curvature into the response using only comparison, absolute value, multiplication, and subtraction, and places its clipping threshold not at the extremum but at the point where the response returns to zero after passing the extremum. As a result, a single function has both a monotonic regime and a nonmonotonic folding regime that increases the expressiveness of a single unit. Which regime is used is determined by the usual hyperparameter search, e.g., over the input scale, so no task-specific threshold tuning is required. The state-update time measured on hardware was 0.34 times that of tanh and shorter than that of any of the 11 evaluated tanh approximations. In PC-based evaluation, third-order MulClip achieved a 22% lower NMSE than tanh on NARMA-20, where the pre-activations remained in the monotonic regime, and on 8-bit parity it raised the accuracy from 0.76-0.78 for the monotonic-clip variant without folding to 0.97-0.98. For these two tasks, on-device inference yielded nearly the same values as on the PC.

Comment
15 pages (7 pages main text, 8 pages appendices), 2 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Sector-Ring Linear Reservoirs: Sector-Wise Decay-Timescale Allocation With State Reuse Across Candidate Settings

cs.LG cs.NE ·2026-10-19 ·Yudai Ebato, Sou Nobukawa, Yusuke Sakemi, Takashi Kanamaru, Nina Sviridova +1 more
similarity 0.57
Abstract · 19 pages, 9 figures, 6 tables. Submitted to IEEE Access

In reservoir computing, performance depends on whether the fixed dynamics of the reservoir suit the task, and the appropriate memory length and frequency response differ across tasks. The reservoir settings are found by a search on each task, and driving the reservoir with the input and collecting its states for every candidate setting dominates the cost of the search. In a reservoir with a linear recurrence, the dynamics decompose into modes, one per eigenvalue, and the eigenvalue distribution becomes a design variable. We propose the Sector-Ring Linear Reservoir (SRLR), in which the eigenvalue angles are fixed at evenly spaced positions around a circle and one decay timescale, chosen from a few values, is assigned to each sector of consecutive angles. To keep the scale of each mode's state equal across candidates, the input weights are scaled using the power spectrum of the training input. To reduce the cost of the search, the states of the modes for each decay-timescale value are computed once and reused across candidates. On prediction and reconstruction benchmarks built from chaotic systems, the SRLR achieved, in all eight cases, a lower mean test error than a reference model in which all modes share one decay timescale, and was competitive with nonlinear echo state network and leaky-integrator echo state network baselines. The selected decay-timescale profiles changed with the prediction target and horizon. The state reuse reduced the number of computed mode trajectories in the search by 99.8% relative to per-candidate recomputation.

0.28

Free-Probability Kernels for Zero-Rollout Hyperparameter Selection in Reservoir Computing

cs.LG cs.NEstat.ML ·2026-08-21 ·Sara Malacarne, Andrea Ceni, Claudio Gallicchio
similarity 0.56
Abstract · Submitted to IEEE Transactions on Neural Networks and Learning Systems (TNNLS)

Reservoir computing (RC) couples a fixed recurrent dynamical system with a trained lightweight readout, but this efficiency is partly lost during hyperparameter selection: the recurrent gain, input scale, and leakage rate determine the reservoir's stability and temporal processing regime and are usually tuned through many rollouts. We introduce a deterministic, pilot-informed selector for leaky linear reservoirs followed by coordinate-wise nonlinear features. Free probability yields cross-lag propagation coefficients that summarize how the reservoir mixes past inputs. In the large-width limit, these coefficients define a deterministic temporal kernel that approximates the finite-reservoir feature geometry. Kernel ridge regression on a short labelled pilot sequence therefore ranks candidate operating regimes without instantiating or rolling out a reservoir, and the selected configuration transfers across widths. Across ten synthetic temporal benchmarks, zero-rollout selection obtains a mean deployment score of $0.772$, compared with $0.774$ for exhaustive simulation-based search, while avoiding $156\,600$ selection rollouts. With a small rollout budget, the proposed ranking provides the strongest mean performance at every tested budget and reaches the exhaustive reference using $4.8\%$ of its rollout cost. On four public electricity-transformer-temperature (ETT) forecasting datasets, five retained candidates recover the exhaustive operating point on three datasets. On multivariate cellular-traffic forecasting, 15 rollouts per cell reach the 462-rollout exhaustive reference and outperform random search and Bayesian optimization at low budgets. These results position free-probability kernels as deterministic surrogates for selecting reservoir operating regimes when validation rollouts are scarce.

0.45

Activation-Flexible ANN-to-SNN Conversion with Finite-State Markov Neurons

cs.NE ·2026-09-24 ·Ruiyu Jia, Zhuo-Cheng Xiao
theory similarity 0.55
Abstract

Most ANN-to-SNN conversion methods rely on a specific correspondence between the source activation and the spiking neuron dynamics. We propose a finite-state continuous-time Markov chain (CTMC) neuron framework whose stationary spike flux can approximate every continuous nonnegative monotone activation function on a compact interval. For a generalized CTMC family with affine input-dependent transitions, we prove uniform approximation to arbitrary accuracy over this function class and derive an explicit approximation error bound. In practice, two- and three-state CTMCs fit ReLU, sigmoid, softplus, and clipped ReLU on the evaluated input ranges, and we evaluate corresponding MLP conversions for each activation with layerwise rate scaling. Moderate clipping improves the conversion cost-accuracy tradeoff on the MNIST MLP and reduces SynOps by 27% on VGG-11/MNIST at matched ANN-SNN accuracy gap criteria, whereas the trend reverses on VGG-11/CIFAR-10. Mean-field and layerwise diagnostics indicate that finite-window sampling and terminal-layer mismatch are the main residual errors. Overall, our results establish finite-state CTMC neurons as a theoretically grounded framework for activation-flexible ANN-to-SNN conversion beyond fixed activation-neuron correspondences.

0.05

FlexSpIM: An Event-Based Digital Compute-In-Memory Accelerator with Flexible Operand Resolution and Layer-Wise Hybrid Stationarity

cs.AR ·2026-09-08 ·Nicolas Chauvaux, Adrian Kneip, Charlotte Frenkel
similarity 0.54
Abstract · 14 pages, 17 figures, 2 tables

Compute-in-memory (CIM) accelerators for spiking neural networks (SNNs) offer a promising solution for achieving $μ$s-level inference latency and ultra-low energy in edge vision applications. However, their limited flexibility at both circuit and system levels restricts their deployment across diverse workloads. This work introduces FlexSpIM, a digital CIM architecture supporting arbitrary operand resolution and shape within a unified storage for weights and neuron states (i.e., membrane potentials). These circuit-level capabilities enable a layer-level hybrid weight- and output-stationary dataflow, maximizing operand reuse and reducing costly on- and off-chip data movement during SNN execution. Measurement results from a fabricated FlexSpIM prototype in 40-nm CMOS demonstrate competitive 1-bit-normalized energy efficiency and higher throughput compared with prior fixed-precision digital CIM-based SNN accelerators, while providing bitwise resolution reconfiguration. Evaluated on the IBM DVS gesture dataset, FlexSpIM achieves 95.8% accuracy while enabling up to 45% energy and 52% latency reductions in large-scale systems compared with fixed stationarity approaches.

0.13

HyDra: Demystifying and Taming Dynamic Context Parallelism at Production Scale

cs.DC ·2026-09-28 ·Zihao Fan, Yunzhuo Liu, Bo Jiang, Changgang Zheng, Lin Zheng +2 more
similarity 0.53
Abstract · 15 pages, 25 figures, 1 table

Long-context training runs on sequences whose lengths span orders of magnitude, and dynamic context parallelism (DCP) gives each sequence its own CP degree. Existing DCP systems either do not scale or perform poorly on mainstream models, leaving Megatron-Core (Mcore) DCP as the only option at production scale. Mcore DCP, however, sizes each degree to fit memory, which grows linearly with length while attention grows quadratically, so comparable token counts hide unequal computation. In our production 256K-context training job on more than 11K GPUs under Mcore DCP, per-rank microbatch times differ by up to 5x at similar token counts. The skew leads to a 46% pipeline bubble and a 13% data-parallel bubble. We present HyDra, a scalable load-driven DCP system. Its scheduler balances computation by pulling every rank toward one load target, and balance in turn makes that target solvable in closed form. It places sequences with lazy heaps rather than whole-pool scans. That balance asks for CP degrees larger than memory requires, so its CP engine nests an inner Ulysses group in a shallow outer ring, letting a higher degree lower computation and communication together. Evaluation at both scales shows consistent gains. On a 512-GPU testbed, HyDra raises throughput over Mcore DCP by 1.18x on average at 32K context and 2.48x at 256K. On a 2,048-GPU production job, it shrinks the pipeline bubble from 36% to 14%, cuts scheduling time by 2.3x, and raises throughput by 1.10-1.43x (avg. 1.25x) over Mcore DCP and 1.33-1.90x (avg. 1.59x) over static CP.

0.27

RC Prediction: A Python Package for Parameter-Aware Reservoir Computing

cs.LG eess.SYmath.DS ·2026-10-29 ·Jin-Cheng Rao, Zheng-Meng Zhai, Ying-Cheng Lai
benchmarkcode/data similarity 0.53
Abstract

A nonlinear dynamical system can be characterized by the autonomous evolution of its state in time and its dependence on system parameters. As a parameter varies, the dynamics can change quantitatively and, at bifurcation points, qualitatively. When the governing equations are unknown, a data-driven model should therefore capture both elements from observational data, rather than reproduce the dynamics at a single operating condition only. Parameter-aware reservoir computing (PARC) is a model-free framework for this purpose, where the parameter is supplied to a reservoir computer through an additional input channel. PARC is trained on time series recorded at several parameter values, and the trained reservoir, run in closed loop as an autonomous dynamical system, generates trajectories at parameter values that were never observed. rc_prediction is an open-source Python package that implements this workflow. It provides multi-parameter training of a reservoir, autonomous closed-loop prediction at user-specified parameter values, parameter scans, and a standard echo state network for forecasting at a fixed parameter. Hyperparameter optimization and optional GPU acceleration through PyTorch are also provided. We use three benchmark dynamical systems as examples to demonstrate its usage. As one of the featured applications, here we illustrate the prediction of critical transitions and system collapse: trained only on pre-critical data, the model is queried beyond the training range to predict whether the system collapses, to estimate the critical parameter $p_c^{*}$, and to characterize transient lifetimes.

0.19

Towards Design of Efficient Neuromorphic Controllers

eess.SY ·2026-11-10 ·Koen Scheres, Erik Steur, Maurice Heemels
similarity 0.52
Abstract · Submitted to European Control Conference 2027

One of the major sources of inspiration for neuromorphic engineering is the central nervous system, which is orders of magnitude more efficient than classical digital clock-based computers. Neuromorphic control aims at recreating this efficiency in control systems, combined with (formal) stability and performance guarantees. In our previous work [1], we have established formal guarantees for the emulation of static output feedback controllers implemented using integrate-and-fire neurons. In this paper, we investigate the ``efficiency'' of the resulting controllers in terms of sparse spiking activity. By changing the network structure of our original implementation, foregoing the unit gain assumption made in our previous work, we will establish that the chosen architecture of the spiking controller significantly affects its sparsity and discuss the implications of these observations for the efficient implementation of spiking controllers.

0.28

TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

cs.LG math.OC ·2026-10-01 ·Jichao Jiang, Cristian McGee, El Houcine Bergou, HanQin Cai, Aritra Dutta
code/data similarity 0.52
Abstract · 24 pages, 7 figures, 10 tables. Code available at https://github.com/Jichao2357/TACO_optimizer

Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficiency in LLM fine-tuning, we propose Ternary Absolute-max Column-wise One-sparse optimizer, or TACO, which follows Muon's operator-norm steepest-descent view but takes the geometric route further. TACO computes the exact steepest-descent direction under a dimension-normalized $1\to1$ operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This retains first-order gradients while making optimizer state memory nearly negligible. Our practical TACO optimizer maintains only a small set of low precision gradient components per column, reducing persistent optimizer state by $174\times$ relative to AdamW8bit (from 27.7 GB to 0.16 GB) and peak training memory by $2.9\times$ (from 80.6 GB to 27.5 GB) on OPT-13B, while achieving comparable accuracy and runtime. TACO further enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks.

0.30