CS Paper Radar

Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models

Hanfei Liu, Shuchang Feng, Yanxia Chen, Changzeng Fu, Shiqi Zhao

cs.LG cs.AI ·submitted 2026-10-07 21:41 UTC · v1

Abstract

Spiking language models face a tradeoff between representing continuous semantic features over short temporal windows and retaining costly nonlinear attention operations. We introduce Spora, which jointly designs spike encodings and attention operators. Binary temporal weights let $T$ spikes represent compositional values with up to $T$ bits of capacity, compared with $O(\log_2 T)$ bits for spike-count readout. Unipolar Binary Spiking (UBS) uses thresholds and spike-triggered residual decay to produce non-negative integer codes; Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations. These representations support accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora achieves 76.6 average GLUE score and 44.1 CoLA MCC, improving over SpikeLM by 1.2 and 6.2 points, respectively. Extending BBS to six steps raises these scores to 78.2 and 47.4. Conditional-decay analysis, matched-budget activation-quantization comparisons, event-workload statistics, and fixed-point evaluation further characterize the connection between encoding fidelity and computational cost.

Closest papers

Nearest neighbours by meaning across the whole index.

Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage

cs.NE ·2026-08-12 ·Kaiwen Tang, Jiaqi Zheng, Zixuan Zhu, Yiqun Wang, Zhanglu Yan +1 more
similarity 0.71
Abstract · 12 pages, 2 figures

Self-attention has become central to spiking vision transformers, yet its query-key scoring is still largely inherited from dense networks. Existing spiking variants either simplify dot product scoring or replace it with discrete operators, but spike timing, the native variable of a spiking network, does not directly define how tokens are related. We propose Lapis, a spiking attention mechanism that scores each token pair by the L1 distance between its query and key first-spike latency vectors under time-to-first-spike coding, and maps this distance to an affinity through a Laplacian kernel. The kernel's exponential decay matches the impulse response of a leaky integrate-and-fire membrane, so the accumulated latency difference determines the decay of a membrane trace, while row normalization reduces to a bit shift under power-of-two rounding. Scoring therefore needs only subtraction, absolute value, and accumulation, and removes all multiplication between query and key channels. Under a matched backbone and training schedule, Lapis reaches 96.56% top-1 accuracy on CIFAR-10, within 0.53 points of dot-product scoring. On ImageNet-1K, it reduces the estimated arithmetic energy of the attention path by 14.5x relative to dense dot-product attention. The deployed 6-bit model attains 83.25% top-1 accuracy at an estimated arithmetic energy of 3.28mJ per image.

0.10

Parallel Time-Aligned Spiking Self-Attention for Consistent Integer-Valued Training and Spike-Driven Inference

cs.NE ·2026-10-02 ·Peng Xue, Wei Fang, Kaiwei Che, Qingyan Meng, Zhengyu Ma +2 more
similarity 0.69
Abstract

Integer-valued leaky integrate-and-fire (I-LIF) neurons and spike firing approximation (SFA) reduce temporal training cost by representing spike trains as firing counts and normalized firing rates, respectively. However, applying spiking self-attention (SSA) directly to these compressed query, key, and value representations introduces cross-time interactions that are absent during spike-driven inference. We term this operator-level discrepancy Temporal Interaction Mismatch (TIM). We propose Parallel Time-Aligned Spiking Self-Attention (PT-SSA), which reconstructs consecutive virtual spike slices from either I-LIF counts or SFA firing rates, computes attention only between time-aligned slices in parallel, and sums the per-step outputs. To accommodate the reduced attention output scale under SFA, we further introduce Adaptive PT-SSA, which learns a positive per-block rescaling before the output SFA neuron to improve firing-level utilization. Experiments on CIFAR-10, CIFAR-100, and ImageNet-1K show that the proposed methods substantially reduce train--inference mismatch. On CIFAR-100 with I-LIF, PT-SSA reduces the mean Top-1 gap from 1.85 to 0.29 percentage points. On ImageNet-1K, Adaptive PT-SSA reduces the Top-1 gap from 27.78 to 0.06 percentage points and achieves 74.53\% spike-driven Top-1 accuracy. A Triton-fused PT-SSA training kernel retains a $2.91\times$ throughput advantage over recurrent LIF SSA.

0.19

QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models

cs.AI ·2026-09-28 ·Bang Hu, Guowei Zhu, Changze Lv, Xiaoqing Zheng, Fengzhe Zhang +2 more
similarity 0.68
Abstract

Large language models (LLMs) achieve strong performance across many tasks but rely on dense multiply-accumulate (MAC) operations during inference, resulting in high energy cost. Spiking neural networks (SNNs) offer an event-driven alternative in which synaptic integration uses lightweight accumulation. However, spike-driven LLM inference remains difficult because outlier-heavy activations typically require long firing windows or auxiliary non-spiking paths. We propose QuantaSpike, a short-window spike-driven quantization framework for LLMs built around Logarithmic Ternary Integrate-and-Fire (LTIF) neurons. LTIF uses ternary events with power-of-two membrane-response quanta, improving the information represented by each firing step while retaining shift-ACC-compatible computation. QuantaSpike combines this neuron with group-adaptive gain and selective outlier admission: normal values use residual LTIF steps, whereas admitted outliers receive one additional onset spike before entering the same residual dynamics. Across OPT and Llama-2, QuantaSpike achieves state-of-the-art or competitive perplexity and zero-shot accuracy among spike-driven LLM quantization methods. It also transfers to newer dense LLMs, remaining close to the FP16 reference on Llama-3-8B and Qwen3-8B under the same four-step firing window. Analytical linear-energy projections show that QuantaSpike reduces the energy of one linear transformation by about $80.0\%$ on OPT models and $67.1\%$ on Llama-2 models relative to SpikeQuant, providing an accurate and energy-efficient spike-driven path for LLM inference.

0.22

Large Language Models with At Most One Spike per Neuron

cs.NE cs.CL ·2026-09-04 ·Zhuoya Zhao, Parsa Omidi, Aref Jafari, Richard Naud
similarity 0.68
Abstract

Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.

0.15

SpikCTU: A Compact Spiking Network with Adaptive Recurrent Computation for Speech Command Recognition

cs.SD ·2026-10-10 ·Jin Li, Wenbin Jiang, Tao Chen, Fei Wen
code/data similarity 0.68
Abstract

Spiking Neural Networks offer event-driven, energy-efficient computation for speech command recognition, yet most existing methods follow fixed computation paths, allocating redundant computation to easy samples while providing limited insight into how evidence evolves during inference. To address these challenges, we introduce SpikCTU, a compact spiking network with adaptive recurrent computation for speech command recognition. Specifically, a hybrid spiking embedding module preserves continuous input information alongside spike features to construct a fixed temporal memory. A shared Continuous Thinking Unit (CTU) repeatedly reads this memory using keys derived from spikes and continuous values, and updates a binary recurrent state with lightweight stage adapters. Furthermore, combining a fixed base prediction with accumulated corrections at each step enables learned aggregation over all steps or confidence-based early exit using only the predictions available so far. Experiments on three speech command recognition benchmarks show that SpikCTU achieves competitive accuracy with only 0.16--0.17M parameters, and the early exit strategy can substantially further reduce computational cost without compromising accuracy. In addition, trajectory and attention analyses reveal that recurrent steps enable error correction and progressively refine predictions. Code is available at https://github.com/Rivflyyy/spikctu.

0.31

Contrastive Attention Mitigates Spectral Bias in Spiking Transformers

cs.AI ·2026-10-01 ·Xiaoli Liu, Malu Zhang, Yang Yang
similarity 0.65
Abstract · Spiking Neural Networks

Spiking Transformers merge the energy-efficiency of spiking neural networks (SNNs) with the representational power of self-attention, creating a promising architecture for high-performance, energy-efficient computation. However, a performance gap persists versus its counterparts in artificial neural networks (ANNs). Unlike prior works attributing this to binary activations, we reveal that both spiking neurons and spiking self-attention (SSA) act as low-pass filters through multiscale spectral analysis. This characteristic leads to the dissipation of high-frequency components. To address this issue, we propose the Spiking Contrastive Attention (SCA) paradigm, which draw inspiration from the edge-detection and differential sensing properties of biological visual system. By extracting contrast prototypes via global contrastive aggregation and applying local differential refinement, SCA effectively enhances high-frequency information. Extensive experiments show that SCA is a general module that consistently boosts Spiking Transformers across image classification, semantic segmentation, and event-based tracking. Furthermore, it achieves lower complexity, offering superior efficiency over original SSA. These results establish its potential as a fundamental building block for energy-efficient Spiking Transformers.

0.16

BASC : Behavior-Aligned Quantization and Pruning for Low-Bit Spiking Neural Networks

cs.NE ·2026-08-12 ·Linliang Chen, Yan Zhong, Xin Liu, Sai Li, Wang Kang
similarity 0.64
Abstract

Spiking Neural Networks (SNNs) encode information through binary spikes and compute in an event-driven manner, offering an energy-efficient paradigm for machine intelligence. However, high-performance SNNs incur substantial memory and timestep-wise computation costs that hinder deployment on resource-constrained devices. Quantization and pruning provide complementary routes to reducing these costs, yet both make their decisions with local criteria that overlook temporal task feedback in quantization and inter-channel dependencies in pruning. Consequently, optimizing either criterion can still yield suboptimal compression performance. We refer to this discrepancy as criterion-behavior mismatch and propose Behavior-Aligned SNN Compression (BASC), a unified framework with two lightweight modules. For quantization, the scale is applied to synaptic current at every timestep and therefore shifts spike timing. Temporal-Behavior Scale Correction (TSC) makes the scale learnable under a temporal loss, allowing firing behavior to inform scale optimization. For pruning, channel importance depends on how channels jointly drive the membrane potential across the firing threshold. Boundary-Level Inter-Channel Correction (BIC) uses channelwise importance scores for initial selection and inter-channel information to re-evaluate only channels near the pruning threshold. Extensive experiments on static and neuromorphic benchmarks show that lower-bit BASC models match or outperform higher-bit baselines and retain this accuracy advantage after structured pruning, while further reducing model storage and synaptic operations.

0.04

SpikeLite: Lightweight Spiking Neural Networks for Time-Series Forecasting

cs.LG cs.AI ·2026-09-28 ·Bang Hu, Changze Lv, Mingjie Li, Xiaoqing Zheng, Wei cao +1 more
similarity 0.64
Abstract

Spiking neural networks (SNNs) offer an energy-efficient paradigm for time-series forecasting through spike-driven computation. However, recent SNN forecasters often pursue higher accuracy through increasingly complex attention mechanisms, or specialized neuronal dynamics, weakening the lightweight motivation of SNNs. We introduce SpikeLite, a spiking forecasting framework built around two modules: a Frequency-Selective Spiking Encoder (FSSE) for frequency-sensitive temporal encoding and a Sparse Spiking Channel Attention (SSCA) module for selective cross-channel interaction. FSSE exploits the low-pass filtering behavior of LIF dynamics to reorganize each input sequence into frequency-sensitive components while collectively preserving the input at the decomposition stage. SSCA then learns a binary mask from encoded channel representations and uses it to selectively exchange information within spike-driven self-attention, retaining informative cross-channel interactions while suppressing redundant ones. When explicit channel interaction is unnecessary, SpikeLite uses the lighter FSSE-only channel-independent path. Experiments under the SeqSNN and SpikF protocols cover four standard multivariate and eight long-term forecasting benchmarks. SpikeLite achieves the best aggregate performance under both protocols, with an average $R^2$ of 0.790 and RSE of 0.440, and lowest average MSE/MAE of 0.343/0.345 in long-term forecasting. Moreover, evaluation on the ECL dataset shows that SpikeLite achieves the lowest reported energy consumption, further demonstrating its potential for energy-efficient time-series forecasting.

0.23