CS Paper Radar

Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking

Xiangyu Liu, Yijie Shen, Yue Yin, Zhitong Yin, Yongqi Pan, Wentao Fan

cs.CV ·submitted 2026-10-26 07:06 UTC · v1

Abstract

RGB-X tracking complements RGB appearance with thermal, depth, or event observations, but centralized training requires collecting multimodal videos at a single location. Federated training keeps raw videos local, but sensor heterogeneity can create a mismatch between parameter-sharing domains and the clients contributing to them. This mismatch is particularly relevant when RGB-only clients coexist with sensor-equipped clients: their RGB observations remain useful, but they provide no auxiliary measurements for learning sensor-dependent interactions. We propose Fed-MAPTrack, a modality-aware federated prompt tuning framework for RGB-X tracking. The pretrained tracking backbone and prediction head are frozen, and clients optimize and exchange only prompt-related parameters. The server aggregates RGB-shared parameters globally and maintains separate thermal, event, and depth banks for auxiliary-specific, fusion, and normalization parameters. Selective participation retains RGB-only clients' contributions to the global RGB bank while excluding their fusion and normalization updates from modality-specific aggregation. Across three matched training repetitions under a fixed client partition, Fed-MAPTrack improves the mean FedAvg scores by 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack. Among the nine evaluated methods, it achieves the highest mean thermal AUC and depth F-score, while its mean event SR is 0.07 points below that of Independent FL. Controlled ablations show that extending the bucketing scope without a compatible participation rule can reduce performance. These results highlight the need to jointly define the sharing domain and eligible contributors for each parameter group.

Closest papers

Nearest neighbours by meaning across the whole index.

UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking

cs.CV ·2026-10-09 ·Shiao Wang, Xiao Wang, Yuhe Ding, Jiarui Xu, Zikang Yan +3 more
code/data similarity 0.69
Abstract

RGB-X tracking leverages complementary modalities, such as thermal infrared, event, or depth, to improve robustness under challenging conditions. Existing general-purpose RGB-X trackers typically follow two paradigms: independent task-specific training and joint training across modality combinations. However, the former suffers from knowledge isolation and model redundancy, while the latter introduces optimization interference across heterogeneous tasks. In this work, we present UniMergeTrack, a new model merging paradigm that preserves task-specific optimization while enabling effective cross-task knowledge integration. UniMergeTrack first trains modality-specific trackers independently and then performs hierarchical model merging. Specifically, transferable backbones are merged across RGB-X tasks via high-pass-filtered merging, and complemented by lightweight task-specific experts that preserve modality-specific cues. Tracking heads are merged within each task using optimal transport-guided channel alignment to mitigate hidden channel mismatch. A subsequent distribution alignment fine-tuning stage further improves compatibility between the unified backbone and task-specific heads. Extensive experiments on the LasHeR, RGBT234, VisEvent, VOT-RGBD22, COESOT, and CDTB benchmarks demonstrate that UniMergeTrack achieves state-of-the-art or competitive performance across RGB-Thermal, RGB-Event, and RGB-Depth tracking tasks, while exhibiting strong generalization to unseen datasets. The source code will be released upon acceptance. The source code of this paper will be released on https://github.com/Event-AHU/OpenEvTracking.

0.36

End-to-End Self-Supervised RGB-T Tracking without Modality Misleading

cs.CV ·2026-09-29 ·Shenglan Li, Rui Yao, Kunyang Sun, Hong Jia, Yong Zhou +2 more
code/data similarity 0.66
Abstract

RGB-T object tracking leverages the complementary characteristics of visible and thermal infrared modalities to improve robustness under adverse conditions. Existing supervised methods typically rely on costly modality-aligned bounding box annotations, while most self-supervised approaches follow a two-stage pseudo-labeling paradigm, making tracker training sensitive to pseudo-label quality and preventing joint end-to-end optimization. In this paper, we propose ESMTrack, a fully end-to-end self-supervised RGB-T tracking framework without offline pseudo-label generation or dense frame-level bounding box annotations. Given only the standard initial-frame annotation used in visual tracking, ESMTrack learns discriminative and temporally consistent representations through two complementary objectives: a grounding triplet loss on annotated initial frames and a cross-frame temporal triplet loss on unlabeled search frames, with reliable samples selected by forward-backward consistency. To address modality dominance bias, ESMTrack employs a three-branch architecture consisting of a fusion branch and two unimodal branches for RGB and thermal inputs. We quantify modality contributions using the Average Peak-to-Correlation Energy by measuring response discrepancies between the fusion and unimodal branches. The resulting reliability estimates guide a training-time modality decoupling mechanism that suppresses dominant-modality shortcuts and adaptively weights cross-modal contrastive learning for task-level alignment. Extensive experiments on five RGB-T tracking benchmarks show that ESMTrack achieves competitive state-of-the-art performance, strong cross-dataset generalization, and real-time inference speed. The source code is available at https://github.com/LiShenglana/ESMTrack.

0.43

SFE-VGGT: Source-Free VGGT Distillation for Event-Based Monocular Depth Estimation

cs.CV ·2026-09-29 ·Thai Duy Nguyen, Addison Lin Wang
similarity 0.63
Abstract

Recent event-based depth estimation methods successfully transfer geometric priors from vision foundation models via cross-modal distillation. However, their reliance on synchronized RGB-event pairs or depth annotations during training severely restricts practical deployment. To overcome this bottleneck, we propose SFE-VGGT, a novel source-free framework that distills the geometric priors of VGGT to the event domain without any paired RGB observations. Our core idea is to reconstruct surrogate frames directly from the target event stream to act as a frozen geometric teacher, entirely eliminating the need for genuine source RGB data. Crucially, as these surrogate frames inherently yield imperfect and spatially varying supervision, directly distilling from them propagates artifacts. To resolve this, we introduce a novel reliability-aware distillation strategy. This includes Density-Aware Feature Distillation to emphasize informative event regions, and Confidence-Weighted Depth Distillation to dynamically regulate supervision based on relative teacher-student prediction confidence. Meanwhile, we propose a Cross-Frame Relational Consistency loss that enforces temporal geometric stability using reliable inter-frame correspondences, bypassing the need for temporally consistent teacher's depth. Extensive experiments demonstrate that, despite source-free, our SFE-VGGT closely matches the accuracy of RGB-dependent baselines under standard conditions and significantly surpasses them in challenging nighttime scenarios. Across MVSEC nighttime sequences, SFE-VGGT reduces the average 10 m depth error by 15.3% compared with EventVGGT. Moreover, our method exhibits robust zero-shot generalization across real-world datasets, proving that highly effective geometric priors can be transferred to event cameras using strictly source-free supervision.

0.19

LC-ISP Benchmark: Learning Advanced ISP with Live and Capture Streams

cs.CV ·2026-11-14 ·Kai Chen, Junyu Lou, Jinhua Zhang, Weiyi You, Shuhang Gu
benchmarkcode/data similarity 0.62
Abstract · Code: https://github.com/CVL-UESTC/LC-ISP

High-quality paired RAW-to-sRGB datasets are essential for data-driven ISP, yet existing datasets often rely on cross-device collection, causing misalignment and failing to faithfully reflect real capture conditions. We therefore introduce LC-ISP, a large-scale benchmark built from the intrinsic gap between a camera's live preview and offline capture streams. LC-ISP contains 53.9k spatially aligned groups, each comprising live RAW/live sRGB and multi-frame fused RGB/capture sRGB, and covers day, night, indoor, outdoor, and HDR scenes. It also provides capture-stream intermediates for modular supervision. We further develop a compact two-stage baseline that uses these intermediates as auxiliary targets. Under a 2.5-GFLOP budget for 448$\times$448 RAW inputs, it outperforms the strongest single-frame baseline by over 2 dB PSNR while improving texture, denoising, and color reproduction. LC-ISP supports realistic benchmarking, the design of supervision strategies, and the development of lightweight ISP models.

0.37

Latent Spectral Prompting for Multimodal Hyperspectral Object Tracking

cs.CV cs.AI ·2026-11-02 ·Wangshu Cai, Jun Zhou, Lei Wang, Mohammad Aminul Islam, Xu Han +1 more
published similarity 0.61
Abstract · Accepted at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2027)

Hyperspectral videos provide rich spectral signatures for robust tracking under challenging conditions such as camouflage and low illumination. However, limited hyperspectral annotations and the large RGB-hyperspectral domain gap hinder the direct use of modern temporally pretrained RGB trackers. We propose SProTrack, a multimodal framework that transfers temporal knowledge from pretrained RGB trackers to the hyperspectral domain through latent spectral prompting. The key idea is to decouple temporal modeling from spectral adaptation, allowing temporal dynamics to be inherited from RGB foundation models while hyperspectral information is learned from unlabeled data in a self-supervised manner. Unlike band-selection methods that discard most spectral information or prompt-based approaches that rely on local-search crops, SProTrack learns a global, scene-level spectral representation from the full hyperspectral frame using an autoencoder. This latent representation is used to guide spectral prompt extraction while a pretrained RGB tracker preserves strong temporal modeling on RGB-compatible inputs. By encoding spectral-spatial structure into compact latent prompts, SProTrack bridges the RGB-hyperspectral modality gap without extensive retraining or dense supervision. Experiments on HOT2020 and HOT2023 demonstrate that SProTrack achieves the best overall performance, with strong results across all spectral domains.

0.32

ReTailor: A Residual Feature Tailoring Framework for Real-Time Cross-Sensor Pansharpening Adaptation

cs.CV ·2026-11-04 ·Jing-Ze Liang, Wei-Jie Xiao, Zi-Han Cao, Yang-Dong Li, Tian-Yi Hu +1 more
code/data similarity 0.61
Abstract · 18 pages, 12 figures, including appendices. The first three authors contributed equally

Recent pansharpening models achieve strong fusion performance on trained sensors, yet often generalize poorly to unseen sensors. Existing cross-sensor adaptation is limited by target-sensor retraining, which requires target-sensor training pairs and a dedicated training stage, and by per-image zero-shot adaptation, which is computationally expensive due to whole-network optimization for each test image. To address these problems, we propose ReTailor, a real-time sequential cross-sensor adaptation framework. Only a lightweight residual Feature Tailor is rapidly optimized under self-supervised, physics-aware constraints to correct cross-sensor shifts without modifying the well-learned fusion prior. For efficient and effective deployment, ReTailor combines content-aware patch selection, which identifies a few informative patches per image for optimization at negligible cost to improve both quality and efficiency, with rolling residual bootstrap, which inherits sensor-related knowledge across the image stream to further improve quality. Extensive experiments on cross-sensor adaptation settings show that ReTailor mitigates cross-sensor degradation both effectively and efficiently. Moreover, we compare ReTailor with Low-Rank Adaptation (LoRA), a widely used method for fine-tuning, to demonstrate the advantages of our approach. We further evaluate the core principle of ReTailor on two additional remote sensing tasks: super-resolution pansharpening and image segmentation, where it consistently improves performance. In general, ReTailor provides a highly effective and efficient strategy for deploying pansharpening models under real-world cross-sensor degradation, while also offering useful insights into managing cross-sensor degradation in other remote sensing tasks. Code will be released at https://github.com/JingZe-Liang/ReTailor .

0.44

SenseWeather: Multimodal Sensing for Robust Robotic Perception under Extreme Conditions

cs.RO cs.CV ·2026-10-08 ·Fucai Ke, Sukai Huang, Junyi Hao, Kal Backman, Naim Rastgoo +2 more
benchmark similarity 0.61
Abstract

Robotic perception must remain reliable when extreme environmental conditions severely degrade the primary RGB observation. Although robotic platforms commonly carry complementary sensors such as depth and infrared, much of the modern perception stack remains RGB-centric, creating a mismatch between multimodal onboard sensing and pretrained downstream models. We study whether complementary sensing can be used upstream to recover a task-compatible RGB observation that preserves existing perception capabilities without modifying or retraining the downstream stack. To this end, we introduce SenseWeather, a controlled multimodal benchmark for robotic perception under severe rain, snow, dust, fog, and darkness, with registered RGB, depth, and infrared observations, clean references, and grounded high-level perception tasks. We further develop Fusion-to-RGB, with two parameter-efficient sensor-fusion realizations based on token-space fusion for a diffusion transformer and residual latent conditioning for an inpainting UNet. Across frozen open-vocabulary grounding and vision-language models, RGB+D+IR fusion improves all paired downstream comparisons over the corresponding RGB-only restorers, including AP50 gains across all ten detector comparisons. Importantly, these perception gains do not consistently follow conventional image-fidelity metrics, and increasing weather severity does not induce monotonic downstream degradation. These results show that complementary sensing can preserve reusable robotic perception under extreme conditions, while highlighting task-level perception robustness as an objective distinct from visual restoration quality.

0.24

TriNetra: Thermal-Reliability-Guided RGB-Thermal Fusion and Tracking for Surveillance

cs.CV ·2026-11-06 ·Sadique Khatib, Aditya Ahirrao, Vedant Sonawane, Disha Kapse, Y. D. Bhise
benchmark similarity 0.60
Abstract

Visible cameras fail at night and thermal sensors fail under saturation, blur or loss of contrast, yet most RGB-thermal (RGB-T) detectors mix the two streams with fixed weights and cannot tell when one modality has become unreliable. TriNetra is a reliability-aware surveillance framework built around this problem. The proposed solution computes a frame-level Thermal Reliability Coefficient (TRC) from five label-free image cues, uses it to gate the thermal features of a dual-branch YOLOv8m detector whose visible branch, neck and head are frozen and COCO-pretrained, and passes the same score to a ByteTrack-based tracker that feeds a rule-based behaviour analyser. On the M3FD benchmark (2,940/630/630 split) the detector reaches 87.6% precision, 79.4% recall, 70.0% mAP@50 and 47.8% mAP@50:95 on the test split, with 17.0 M of its 42.9 M parameters trained. TRC stays near 0.78 on clean frames (standard deviation 0.012), decreases with the severity of thermal saturation and blur, reaching 0.53 and 0.69 at full severity, and is insensitive to RGB-thermal misalignment. With identical weights, the adaptive gate matches a fixed gate on clean data (70.0% versus 69.8% mAP@50) but keeps 43.6% mAP@50 when the thermal stream is corrupted, against 38.4%, 29.2% and 25.0% for a fixed gate, full thermal admission and a closed gate; the 5.1-point gain over the fixed gate has a 95% bootstrap confidence interval of +3.6 to +6.5 points. In tracking, the TRC-gated association performs on par with ByteTrack on a synthetic benchmark with exact ground truth and on 14 M3FD clips, and is slightly less stable under thermal saturation, so its measured benefit is confined to the detector. The limitations of the current evaluation are reported together with the results.

0.23