CS Paper Radar

Event-Frame Fusion for Inter-Frame Segmentation via Event-Guided Motion

Dalia Hareb, Jean Martinet, Benoit Miramond, Elisabetta Chicca

cs.CV ·submitted 2026-09-18 19:02 UTC · v1
published

Abstract

Autonomous navigation requires precise and efficient semantic segmentation, yet existing frame-based approaches remain limited by motion blur, glare, latency, and the low temporal resolution (20-30 FPS) of conventional cameras, which leads to information loss between frames. Event cameras have emerged as an alternative sensing modality, capturing intensity changes asynchronously with high temporal resolution, high dynamic range, and sparse outputs. However, event-based algorithms still fall short of frame-based ones in accuracy, as most segmentation methods are designed for dense frame data. To overcome these limitations, we propose a hybrid vision architecture that combines conventional frame-based and event-based cameras. The system integrates two complementary components: (1) a compact Spiking Neural Network (SNN) with 42k parameters for motion estimation, and (2) a lightweight event-driven SNN with 0.84M parameters for frame-based semantic segmentation, which interpolates motion between frames to refine segmentation results. By predicting inter-frame segmentations, the framework achieves segmentation rates of up to 500 Hz with an energy consumption below 1.87 mJ per inference, while maintaining real-time GPU execution at frequencies up to 200 Hz. Additionally, our approach compensates for information loss in frames affected by blur or overexposure, enabling more robust perception in challenging conditions.

Journal ref
ICONIP 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Toward On-Chip Training of Spiking Neural Networks for Dense Event-Based Vision

cs.CV ·2026-09-26 ·Maxime Vaillant, Axel Carlier, Lai Xing Ng, Christophe Hurter, Benoit R. Cottereau
benchmark similarity 0.70
Abstract

Event cameras provide low-latency, asynchronous visual sensing for resource-constrained robotics. Spiking neural networks (SNNs) process event streams naturally, but training deep SNNs with backpropagation through time (BPTT) requires substantial memory and remains difficult on neuromorphic hardware. Local learning avoids this by restricting error propagation to local blocks, but existing methods mainly target classification rather than dense prediction. We introduce DELL (Dense Event-driven Local Learning), a block wise scheme for dense event-based vision that replaces global gradient propagation with local dense supervision. Learnable, spatially structured local heads supervise each block at its appropriate resolution while preserving temporal dynamics within blocks. We evaluate DELL on optical-flow regression and semantic segmentation with a fully spiking U-shaped architecture. On DSEC optical flow, DELL reduces peak training memory by 39.6% relative to end-to-end BPTT while improving accuracy, reaching 1.670 px endpoint error on the official test benchmark versus 1.941 px for the same backbone trained end-to-end. Block detachment behaves more like a regularizer than a constraint. DECOLLE, the existing local-learning baseline, relies on fixed random local read-outs poorly suited to dense regression, resulting in a 3.9x higher endpoint error; learnable local heads recover this loss and outperform end-to-end training across all optical-flow metrics. On segmentation, they recover most of the performance gap, although DELL remains a few mIoU points behind end-to-end training. With 2.3M parameters, 24x fewer than the strongest SNN baseline, the backbone remains competitive with the SNN state of the art on DSEC. These results extend local learning to dense event-based prediction while substantially reducing training memory.

0.21

A 25-$μ$s/inf Event-driven Graph Neural Network Processor with Spatiotemporal Caching and Spline Convolution for Ultra-low-latency AI at the Edge

cs.CV cs.AR ·2026-09-14 ·Adrian Kneip, Martin Lefebvre, Daniel Gehrig, Victoria Catalán Pastor, Davide Scaramuzza +2 more
published similarity 0.69
Abstract · This paper was accepted for presentation at the IEEE ESSERC 2026 conference

Dynamic-vision-sensor (DVS) cameras generate events on a per-pixel basis with a $μ$s-level temporal resolution, calling for new algorithm-hardware co-design approaches compared to standard frame-based vision. While event-driven graph neural networks (EV-GNNs) emerge as a promising algorithmic solution, they raise new HW challenges by mixing dense-regular compute operations and sparse-irregular memory accesses. We present ETHEREAL, the first EV-GNN accelerator that scales to 640$\times$480 resolutions, thanks to a neighbor-parallel spline convolution engine and a 2D/3D-split memory hierarchy with a novel region-of-interest spatiotemporal caching mechanism. Measurement results demonstrate end-to-end inference with 25.6$μ$s latency and 1.7$μ$J energy per event on state-of-the-art workloads

0.25

LiFR v2: Completion-Augmented Event Propagation for High-Rate Dense Prediction

cs.CV ·2026-09-22 ·Tao Wan, Xiaoshan Wu, Yifei Yu, Bo Wang, Xiaoyang Lyu +4 more
similarity 0.69
Abstract · 15 pages, 9 figures, 6 tables

High-rate dense perception in dynamic environments is limited by the low update rate of RGB cameras, as rapid scene changes can occur between frames. Event cameras offer temporally dense but spatially sparse measurements, complementary to spatially dense RGB observations. Direct fusion cannot fully exploit this complementarity, while event-guided propagation fails on newly appearing or disoccluded regions without valid RGB support. We present LiFR v2, a unified propagation-completion-memory framework for causal anytime and streaming dense prediction from an RGB keyframe and events. LiFR v2 introduces an Event-Guided Completion Module (EGCM) to recover task-relevant representations where propagation is unsupported, and a History Retrieval Module (HRM) to reuse completed representations across successive queries. The framework supports semantic segmentation, monocular depth estimation, and multi-task dense prediction, and we further introduce SHF-Emerge to evaluate rapid object emergence and disocclusion. LiFR v2 achieves 74.37% mIoU on DSEC and 56.13% on SHF-Emerge, improving LiFR-Seg by 1.85 percentage points on the latter, while reducing SHF-Emerge depth RMSE from 1.564 m to 1.118 m over the propagation baseline. It also exceeds 100 FPS for both segmentation and depth, demonstrating accurate and efficient high-rate perception beyond RGB frame rates.

0.47

Tight Learned Event-Inertial Odometry (TLEIO): a Transformer-Based Architecture for End-to-End Motion from Event Streams

cs.CV ·2026-10-23 ·Matteo Cascone, David Jensen, Alessandro Pirini, Angelo Stefanini, Pietro Venè +2 more
benchmark similarity 0.68
Abstract · 14 pages

Event cameras provide microsecond temporal resolution, high dynamic range, and sparse asynchronous measurements, making them compelling for odometry under fast motion and challenging illumination. Yet, accurate event-based state estimation remains difficult: existing systems often use events as a complement to frames or rely on feature tracking, correspondence search, and optimization-heavy backends, limiting the efficiency promised by event sensing. We introduce Tight Learned Event-Inertial Odometry (TLEIO), a compact filter-based pipeline that learns short-window relative motion directly from voxelized event streams and tightly fuses the resulting constraints with IMU propagation. Its learned front-end, EventsFormer, predicts local displacements and per-axis uncertainty from event voxels in a single forward pass, while a stochastic-cloning EKF fuses these overlapping predictions with IMU propagation, producing metric trajectories without explicit feature tracks and pose optimization at inference time. The network is trained directly from pose displacements, without requiring event-flow labels. We evaluate TLEIO on the TartanAir/TartanEvent benchmark and on real DAVIS240c sequences. On TartanAir, TLEIO reduces average ATE from 2.26 m to 1.10 m over the IMU-only TLIO baseline, while achieving up to 48.9% lower runtime than optimization-based event odometry methods.

0.11

Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction

cs.CV ·2026-10-26 ·Jingxue Huang, Xiaosong Li, Yi Lin
benchmark similarity 0.68
Abstract

Multi-modal video fusion aims to generate temporally coherent videos with salient targets and rich structural details by integrating complementary information from heterogeneous sensors. However, existing video fusion methods commonly rely on optical flow estimation and feature warping for inter-frame alignment and cross-frame information aggregation to maintain temporal consistency, thereby introducing considerable computational overhead and making the fusion performance vulnerable to flow estimation errors. To address these limitations, this paper proposes an efficient multi-modal video fusion network with dynamic region-guided temporal interaction. Specifically, the proposed spatio-temporal interaction module leverages the Test-Time Training paradigm to model temporal dependencies with linear complexity. Building on this design, the module restricts temporal modeling to key dynamic regions in the video sequence, as static background regions provide limited temporal cues. By excluding low-variation tokens from unnecessary temporal interaction, this selective strategy contributes to improved fusion quality and temporal consistency. In addition, leveraging the strong ability of pretrained MAE features to capture global structural dependencies, we introduce a feature-level temporal consistency constraint to suppress local flickering by aligning the inter-frame variation directions of the fused and source videos. We further propose a new MAE-based temporal metric that quantifies directional distance between their temporal variations within dynamic regions. Experimental results demonstrate that, compared with existing state-of-the-art methods, our method achieves superior fusion quality and temporal consistency on multiple benchmark datasets, while exhibiting high computational efficiency under edge-device deployment settings.

0.13

ETHEREAL: A 25.6-$μ$s/inf. Low-latency Event-driven Graph-neural-network Processor for High-resolution Vision at the Edge

cs.AR cs.CV ·2026-08-18 ·Adrian Kneip, Martin Lefebvre, Daniel Gehrig, Victoria Catalán Pastor, Davide Scaramuzza +2 more
similarity 0.67
Abstract · This work has been submitted to the IEEE JSSC for possible publication

Dynamic vision sensors (DVS) are enticing candidates to reach the low-latency, sub-ms target of edge-vision applications, as they generate events with a $μ$s-level time resolution. However, using DVS front ends also calls for novel algorithm/hardware back ends capable of efficiently handling streams of sparse spatiotemporal events. While event-driven graph neural networks (EV-GNNs) have emerged as a solution on the algorithmic side that is both accurate and efficient, there is no dedicated hardware to date capable of efficiently supporting their mixed requirements of dense-regular compute operations and sparse-irregular memory accesses. We therefore introduce ETHEREAL, the first EV-GNN processor chip, capable of bridging this gap by means of a neighbor-parallel spline-convolution engine combined with a split-2D/3D memory hierarchy that introduces a novel spatiotemporal event-caching mechanism. Measurement results demonstrate a 25.6$μ$s latency and a 1.6$μ$J energy per end-to-end event-wise inference on the state-of-the art DAGr-GNN workload and VGA-resolution (640x480 pixels) DSEC dataset.

0.17

MotionDLO: Hybrid Event- and Frame-Based Tracking of Deformable Linear Objects

cs.RO cs.CV ·2026-08-23 ·Annalena Hartmann, Priyamvada Ajithkumar, Patrick Bründl, Jörg Franke
code/data similarity 0.66
Abstract

Reliably tracking moving deformable linear objects (DLOs) while simultaneously ensuring robustness, accuracy, and temporally consistent state estimation remains a fundamental challenge in robot perception. We introduce MotionDLO, a real-time tracking framework specifically designed to overcome these limitations in temporal continuity and latency. The method exploits the high temporal resolution and sparsity of event-based cameras and combines segmentation with the Coherent Point Drift (CPD) algorithm under the principles of Motion Coherence Theory. This integration enables temporally consistent shape estimation while maintaining a low computational overhead. Existing event-based tracking methods are typically computationally efficient but exhibit reduced accuracy compared to frame-based approaches, or alternatively compromise event sparsity to achieve competitive performance. To resolve this trade-off, we propose a hybrid event- and frame-based tracking architecture that preserves the complementary strengths of both sensing modalities. The event stream ensures high-frequency motion updates, while frame-based information stabilizes spatial accuracy and object identity. We demonstrate that the proposed framework reliably associates DLO instances across video sequences, enabling robust perception for robotic manipulation tasks. Experimental results validate real-time performance at 12 ms update rates and accurate shape tracking with an point-to-curve error as measurement of accuracy of up to 0.43 mm, supporting dynamic path adaptation during manipulation. The source code and demonstration datasets are publicly available.

0.20

Bridging Event Streams and DiT: Event-Guided Video Frame Interpolation

cs.CV ·2026-08-11 ·Guixu Lin, Yuyang Yu, Xiang Ji, Linyao Chen, Zhengwei Yin +4 more
code/data similarity 0.66
Abstract · https://joseph-lin-tech.github.io/BridgeEventDiT-VFI/

Latent diffusion models have recently advanced video frame interpolation by synthesizing intermediate frames between input images. However, handling large temporal gaps and complex motion remains challenging, often resulting in motion blur, structural distortions, and temporal inconsistencies. Event cameras provide high-temporal-resolution motion cues that are well suited for bridging these gaps and improving interpolation quality. To exploit this advantage without training an event-assisted model from scratch, we propose an adapter-based framework that incorporates event-derived cues into a pre-trained image-to-video diffusion model with minimal architectural changes. Specifically, our method leverages Image Warped Events (IWEs) and bidirectional sparse optical flow to provide spatially and temporally aligned guidance during generation. By injecting these event-guided structural and motion cues into the diffusion process, our approach reduces interpolation artifacts and improves both reconstruction fidelity and temporal coherence. Experimental results on real and synthetic benchmarks show that our method consistently outperforms existing state-of-the-art approaches. The project page is at https://joseph-lin-tech.github.io/BridgeEventDiT-VFI/.

0.20