CS Paper Radar

REACT: A Fully Spiking State-Space Model for Real-Time Event-Driven Temporal Perception

Geoffroy Keime, Nicolas Cuperlier, Benoit R. Cottereau

cs.RO cs.AI ·submitted 2026-09-16 10:06 UTC · v1

Abstract

Robotic systems operating in dynamic environments require visual perception that evolves continuously with the incoming sensory stream. Event cameras provide microsecond temporal resolution and asynchronous sensing, but most learning-based methods accumulate events into frames or temporal bins, introducing an integration delay that can limit fast reaction. Here we propose REACT, a fully spiking state-space model for event-driven temporal perception that processes raw events one by one, without temporal accumulation. REACT uses a complex-valued spiking neuron, C-SiLIF, whose continuous-time dynamics are driven by the physical inter-event interval, allowing its internal state to evolve at the temporal resolution of individual events. We evaluate REACT on gesture recognition and time-to-collision (TTC) estimation from full-field event streams, without a target bounding box or localization input. On EvTTC, REACT achieves a 9.59% relative TTC error with 4.6 ms end-to-end inference latency, within 0.15 percentage points of the best learned method while requiring no target prior. At the dataset's mean approach speed, this latency corresponds to only 4 cm of vehicle motion, compared with 1 m for the fastest competing learned method. REACT further supports anytime TTC prediction, zero-shot transfer to a different driving sequence, and INT8 quantization, reducing the estimated energy consumption from 18.5 to 2.8 mJ per 32,768 events. These results show that event-driven spiking state-space dynamics can provide low-latency, continuously updated temporal perception for reactive robotic systems.

Closest papers

Nearest neighbours by meaning across the whole index.

Can Spiking Neural Networks play pinball? A neuromorphic motion detector for target tracking

cs.CV ·2026-09-21 ·Mazdak Fatahi, Šárka Pryjmaková, Pierre Boulet, Giulia D'Angelo
similarity 0.72
Abstract

Biological visual systems achieve continuous, low-latency motion perception by processing sparse, asynchronous spiking signals, enabling real-time tracking under strict energy constraints. Event-based cameras, inspired by the mammalian retina, replicate this efficiency by capturing only local brightness changes as asynchronous events, offering a natural substrate for spiking neural networks (SNNs) to parallelise computation and adapt to fast-changing scenes. Pinball provides a controlled yet dynamic testbed, requiring precise motion estimation and fast reaction to a small, rapidly moving target. This work presents a fully spiking, real-time perception-to-action pipeline for closed-loop pinball gameplay. A dynamic vision sensor observes a small, fast-moving ball, and a network of spiking Time-Difference Encoders on the SpiNNaker neuromorphic platform jointly estimates its position, speed, and direction. The system is characterised across receptive field size, accumulation window, and angular tuning width for real-time operation, and benchmarked in closed loop against human players across two flipper regimes of increasing physical realism. It achieves a hit rate of 56.1%, nearly double the human average, reacting within 21.7 ms (5 ms network latency) and consuming an estimated 148 μW using fewer than 25k neurons, among the fastest and most energy-efficient event-based closed-loop demonstrators benchmarked. Under more realistic flipper dynamics, tuning a single interpretable policy parameter reproduces the full spectrum of human play styles, from cautious to aggressive, with no change to the perception pipeline. A physical demonstrator, tracking a real ball and actuating real flippers in closed loop, confirms the principle operates beyond simulation. Its fully spiking, learning-free design offers a compact, energy-efficient example of real-time neuromorphic perception-to-action.

0.10

Removing the Compute Bottleneck for Event-based Visual Place Recognition with Spiking Neural Networks

cs.CV ·2026-10-12 ·Somayeh Hussaini, Tobias Fischer, Michael Milford
hot direction similarity 0.70
Abstract · 8 pages, 6 figures, under review

Neuromorphic sensing and computing promise low latency and high energy efficiency for robotics, yet most event-based visual place recognition (VPR) pipelines reintroduce dense, frame-like computation downstream of an inherently sparse sensor, forfeiting the sensor's latency advantage. In this paper, we address the problem of resolving an event-based VPR query in tens of microseconds on a commodity CPU. We train a two-layer spiking neural network (SNN) with additive spike-timing-dependent plasticity (STDP), which drives the learned weight matrix towards a strongly bimodal distribution, amenable to aggressive magnitude pruning. At inference, our pipeline exploits this sparsity by reading only the pruned weights corresponding to the query's active pixels. As surviving weights share a near-uniform magnitude, each neuron's score reduces to an integer count of matching synapses instead of a weighted sum, lowering per-query latency. This computation is small enough to remain resident in a CPU's fast on-chip cache. On three event-based VPR benchmarks, our sparse SNN exceeds the recall of LENS, an existing event-based SNN, while reducing per-query latency by 5.5-6.2x to tens of microseconds. It matches the recall of a dense variant of our network and of the sum-of-absolute-differences (SAD) baseline, while reading 10x fewer weights than the dense variant. We further characterise how recall and latency scale with the number of active pixels and reference places. These results suggest that the sparsity of both event-based input and learned weights can already be exploited for low-latency inference on conventional processors.

0.27

SpikingVLA: Asynchronous Spiking Vision-Language-Action Models

cs.CL ·2026-10-07 ·Jingya Wang, Dehao Zhang, Shuai Wang, Malu Zhang, Yang Yang +1 more
similarity 0.69
Abstract

ANN-to-SNN conversion offers a practical route toward energy-efficient spiking Vision-Language-Action (VLA) models by bypassing the substantial cost of training large-scale SNNs from scratch. However, existing methods often require many timesteps to maintain competitive performance, resulting in substantial inference latency for real-time VLA deployment. To address this challenge, we introduce SpikingVLA, an ANN-to-SNN conversion framework that enables accurate and low-latency spiking VLA inference. Specifically, we propose a Dendritic Integrate-and-Fire (DIF) neuron that alleviates channel-wise activation outliers through dendritic mixing and adaptive somatic firing, enabling accurate ANN-to-SNN conversion with fewer timesteps. Building on DIF neurons, we further introduce an asynchronous execution mechanism that overlaps temporal computation across VLA components, reducing synchronization overhead and latency. Extensive experiments demonstrate that SpikingVLA achieves competitive navigation performance with substantially improved inference efficiency. Compared with existing spiking VLA methods, SpikingVLA improves SR and SPL by 11.9\% and 12.6\%, respectively, while reducing first-action latency by 11.2$\times$. These results establish SpikingVLA as a practical framework for deploying pretrained VLA models with high-performance and low-latency spiking inference.

0.28

Toward On-Chip Training of Spiking Neural Networks for Dense Event-Based Vision

cs.CV ·2026-09-26 ·Maxime Vaillant, Axel Carlier, Lai Xing Ng, Christophe Hurter, Benoit R. Cottereau
benchmark similarity 0.68
Abstract

Event cameras provide low-latency, asynchronous visual sensing for resource-constrained robotics. Spiking neural networks (SNNs) process event streams naturally, but training deep SNNs with backpropagation through time (BPTT) requires substantial memory and remains difficult on neuromorphic hardware. Local learning avoids this by restricting error propagation to local blocks, but existing methods mainly target classification rather than dense prediction. We introduce DELL (Dense Event-driven Local Learning), a block wise scheme for dense event-based vision that replaces global gradient propagation with local dense supervision. Learnable, spatially structured local heads supervise each block at its appropriate resolution while preserving temporal dynamics within blocks. We evaluate DELL on optical-flow regression and semantic segmentation with a fully spiking U-shaped architecture. On DSEC optical flow, DELL reduces peak training memory by 39.6% relative to end-to-end BPTT while improving accuracy, reaching 1.670 px endpoint error on the official test benchmark versus 1.941 px for the same backbone trained end-to-end. Block detachment behaves more like a regularizer than a constraint. DECOLLE, the existing local-learning baseline, relies on fixed random local read-outs poorly suited to dense regression, resulting in a 3.9x higher endpoint error; learnable local heads recover this loss and outperform end-to-end training across all optical-flow metrics. On segmentation, they recover most of the performance gap, although DELL remains a few mIoU points behind end-to-end training. With 2.3M parameters, 24x fewer than the strongest SNN baseline, the backbone remains competitive with the SNN state of the art on DSEC. These results extend local learning to dense event-based prediction while substantially reducing training memory.

0.21

Spike-driven Vision-Language-Action Model

cs.CL ·2026-09-30 ·Shuai Wang, Malu Zhang, Mingquan Liu, Weihui Dai, Dehao Zhang +4 more
similarity 0.67
Abstract

Vision-language-action (VLA) models bridge multimodal understanding and robotic control, advancing the dominant paradigm for embodied intelligence. However, most existing models rely on large Transformers, whose latency and energy costs hinder deployment on resource-constrained platforms. Through sparse event-driven computation, spiking neural networks offer a promising paradigm for high-performance and energy-efficient computing. Here, we propose the first Spike-driven VLA framework enabling end-to-end direct training for robotic manipulation, which mainly comprises three core components. First, we develop spiking visual and instruction encoders for multimodal perception, encoding visual observations and language instructions into sparse, reliable spike representations for subsequent cross-modal fusion. Then, we introduce Multi-Winner Spike Fusion for instruction-guided scene understanding, using bidirectional top-$k$ winner-take-all spike routing to suppress background interference and yield fused memory. Finally, we propose a Spike Action Chunking Transformer that incorporates spiking cross-attention over the fused memory and the current robot state, enabling efficient end-to-end generation of continuous action chunks for robotic control. Extensive experiments on LIBERO and Meta-World demonstrate that Spike-driven VLA achieves competitive performance with fewer parameters and lower estimated inference energy than conventional VLA models. This work establishes a foundational framework for neuromorphic VLA modeling, paving the way for future advances in resource-efficient embodied intelligence.

0.44

Controlling the latency in temporal coded spiking models for neuromorphic datasets

cs.NE ·2026-10-14 ·Alban Gattepaille, Alexandre Muzy
theory similarity 0.66
Abstract

Spiking neural networks (SNNs) promise energy-efficient, event-driven computation on neuromorphic hardware, where response time, or latency, matters as much as energy for real-time applications. Time-to-first-spike (TTFS) coding with learnable synaptic delays is a natural way to exploit the timing of events. However, in state-of-the-art spiking neuron models, spike times emerge from nonlinear membrane dynamics and thresholds: no closed-form expression links network parameters to spike times, so latency can only be measured empirically, not specified or guaranteed at design time. Moreover, TTFS models are mostly evaluated on static datasets, which hides their latency on truly temporal data. Our hypothesis is that, if spike times are computed directly as closed-form functions of presynaptic spike times and synaptic delays, bypassing membrane potential and thresholding, latency becomes an explicit function of the learnable parameters that can be bounded, and therefore controlled, analytically. We build on this hypothesis with the Delay Neural Network (DeNN) in real time, which turns event streams into successive single-spike images, computes each spike time from standardised inter-spike intervals and synaptic delays, and links successive images through a long-term memory. We prove lower and upper bounds on the latency of every neuron of a layer, and show that bounding the synaptic parameters after training with a single hyperparameter places the latency anywhere between these bounds, without retraining. On a ten-class subset of the Google Speech Commands dataset, this post-training control exposes the accuracy--latency trade-off: a three-hidden-layer network reaches 94.25\% accuracy while deciding, on average, 477~ms after the end of the one-second input, and about 250~ms with confidence-based early decisions. Learning is also shown to concentrate the long-term memory on a few neurons. Latency thus becomes a design parameter of spiking models rather than a quantity measured after the fact, a step towards neuromorphic systems with real-time guarantees.

0.16

SpikWM: Efficient End-to-End Spiking World Modeling for Visual Planning

cs.CV ·2026-11-17 ·Shuhan Ye, Hongbin Yu, Chong Wang, Hui Lu, Jiaming He +3 more
code/data similarity 0.66
Abstract

Goal-conditioned planning with image observations and goals requires predictions that distinguish visual states and action consequences. Spiking neural networks offer sparse computation, but single-step spikes can obscure motion cues and merge actions with different physical outcomes. We propose the Spiking Latent World Model (SpikWM), an end-to-end joint-embedding predictive architecture (JEPA) with one simulation step per image and one neural update per latent position. Spiking Regional Encoding retains spatial cues through regional pooling. Bi-Spike Action Encoding combines positive and negative binary responses, while Causal Membrane Prediction carries subthreshold evidence across the latent history with learned group-wise time constants. The encoder and predictor are trained jointly on reward-free image--action trajectories using next-embedding prediction and SIGReg. With a common planner, SpikWM achieves mean success rates of $88.67\%$, $97.33\%$, $99.33\%$, and $86.67\%$ on Push-T, Two-Room, Reacher, and OGBench-Cube, respectively, outperforming two spiking baselines. Push-T ablations support each module, and controlled replays examine responses to appearance and physical state changes. Under a matched planning budget, SpikWM reduces estimated arithmetic energy by $75.6\%$ relative to LeWM. Code will be released at https://github.com/ShawnYE618/SpikWM.

0.37

Power-of-Two Leaky Integrate-and-Fire Neurons for Event-Driven Vision-Language-Action Inference

cs.AI ·2026-10-24 ·Rui Liu, He Lv, Ziyao Zhang, Rui Ma, Rongxiang Nie
similarity 0.65
Abstract

Vision-language-action (VLA) models have demonstrated strong capabilities in robotic manipulation, yet their reliance on dense, continuous-valued computation presents a fundamental challenge for energy-constrained deployment. Inspired by the sparse, event-driven signaling and temporal integration mechanisms of biological neurons, a promising alternative is to introduce spiking computation into VLA models. However, enabling such computation without restructuring pretrained policies or compromising closed-loop control remains an open challenge. To address this challenge, we introduce the Power-of-Two Leaky Integrate-and-Fire (PoT-LIF) neuron, which emits signed spikes with power-of-two amplitudes. By combining magnitude-coded spike events with residual-preserving membrane dynamics, PoT-LIF enables continuous information to be progressively represented through discrete spikes. The Heartbeat formulation further localizes this process within weighted projections, bridging biologically inspired spiking computation and pretrained VLA architectures without modifying their external inference interfaces. We integrate PoT-LIF into the vision projection, vision-language projection, and action projection of $π_{0.5}$ and evaluate both operating modes over 2,000 paired closed-loop LIBERO episodes per configuration. Standard-mode PoT-LIF VLA achieves success rates of 96.90% and 96.35% at $T = 1$ and $T = 2$, respectively, while Heartbeat mode achieves 97.25% and 96.90% at $T = 2$ and $T = 4$, compared with 96.70% for the original continuous-activation $π_{0.5}$ baseline. At $T = 1$ with a 60.01% firing rate, PoT-LIF uses approximately 1/7.67 of the dense-reference arithmetic energy, corresponding to a modeled reduction of 86.95%.

0.15