CS Paper Radar

RAIDAL: Redundancy-Aware Information Density Active Learning for CTC-Based Continuous Sign Language Recognition

Rafael A. Diniz Augusto, Gabriel L. Oliveira, Erickson R. Nascimento

cs.CV ·submitted 2026-09-06 21:39 UTC · v2
publishedcode/data

Abstract

Continuous sign language recognition (CSLR) is a key technology for accessibility, yet its development remains limited by the high cost of annotating continuous video streams. Active learning offers a path toward mitigating this cost, but standard acquisition functions are not designed for weakly aligned sign language videos, where sign executions are interleaved with rest poses, irregular pauses, sign-like motion, and temporally redundant frames. This temporal redundancy can undermine sample selection, as acquisition scores may be influenced by timesteps from regions that are not associated with the decoded gloss sequence, distorting the video's estimated informativeness. In this work, we show that modern CSLR models already contain a mechanism for identifying gloss-level temporal evidence: the CTC decoder. Although typically used only during inference, its alignment peaks indicate where the model localizes each predicted gloss in the feature sequence, providing a source of temporal structure for active learning acquisition functions at zero additional labeling cost. Thus, we introduce RAIDAL (Redundancy-Aware Information Density Active Learning), which repurposes the CTC decoder to restrict representation-based scoring to decoder-aligned gloss regions, rather than exposing the acquisition function to the entire unfiltered video. Across three datasets and two architectures, RAIDAL achieves its strongest data-efficiency gains over competing baselines in large-vocabulary, budget-limited settings, while remaining competitive in the smaller-vocabulary, large-budget setting. The code used in this work is publicly available at github.com/verlab/RAIDAL.

Comment
Accepted to BMVC 2026. This preprint includes one additional analysis not present in the BMVC version, in which we apply RAIDAL's filter to CTC-BADGE

Closest papers

Nearest neighbours by meaning across the whole index.

SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spotting

cs.CV ·2026-08-26 ·Eunjee Choi, JungHoon Sung, Seongwhan Cho, Chu Xin, Younggeun Choi
published similarity 0.78
Abstract · 19 pages, Accepted 37th British Machine Vision Conference, BMVC 2026

Continuous sign language recognition (CSLR) aims to recognize gloss sequences from unsegmented sign videos under weak sequence-level supervision. However, existing methods rely on sentence-level gloss annotations, providing limited temporal and semantic guidance for fine-grained representation learning. Conventional video-text alignment also requires large batch sizes, making it inefficient for memory-intensive sign language video training. In this work, we propose SMART, an MLLM-guided temporal alignment framework for joint sign recognition and spotting. SMART uses MLLMgenerated motion descriptions as auxiliary semantic cues and performs stable videotext alignment under small-batch training. To improve temporal representation learning, we introduce a Multi-Scale Temporal Adapter that models temporal interactions during transformer encoding. For dense temporal localization, SMART incorporates CSFormer, a CSLR-guided spotting module that injects recognition-derived gloss evidence into a boundary-aware spotting network. This unified framework enables CSLR features to benefit spotting, while spotting supervision complements weak CTC-based recognition. Experiments on four sign language benchmarks, including PHOENIX14-T, CSL-Daily, Large-scale KSL, and Disaster and Safety KSL datasets, demonstrate the effectiveness of SMART across both recognition and spotting tasks.

0.36

MoGloss: Joint Motion and Pseudo-Gloss Pretraining for American Sign Language Understanding

cs.CV ·2026-11-19 ·Rui Hong, Jana Košecká
code/data similarity 0.70
Abstract · 15 pages, 4 figures. Project page: https://hongrui16.github.io/MoGloss

Sign language translation and recognition have been shown to benefit from various pretraining approaches. We propose a novel two-stage pretraining approach for the motion encoder, which first learns to reconstruct masked motion, followed by joint masked prediction of pseudo-gloss units and motion. Pseudo-glosses serve as an intermediate structured representation of natural language text and are obtained from the captions by a large language model. We curate a corpus of about 1,100 hours of American Sign Language from five public datasets, estimate 2D pose for all clips and define an evaluation protocol. We then reproduce state-of-the-art translation methods and evaluate them using the same train and test splits. We propose a translation model built on our pretraining strategy, and this model outperforms the previous state-of-the-art methods Uni-Sign and SHuBERT, as well as the YouTube-ASL baseline, on the How2Sign, ASL STEM Wiki and OpenASL datasets. Furthermore, the gain from the joint pretraining using motion and pseudo-gloss grows as translation data shrinks. A classifier built on the same encoder also achieves state-of-the-art accuracy on ASL Citizen isolated sign recognition and outperforms our reproduction of Uni-Sign on WLASL2000.

0.19

From Clips to Streams: A Unified Framework for Streaming Sign Language Translation

cs.CV ·2026-10-12 ·Hoang Quan Dang, Xin Shen, Yanbin Liu, Xin Yu, Ling Chen
top venuebenchmarkcode/data similarity 0.70
Abstract · 27 pages, 7 figures, the paper has been accepted to NeurIPS 2026

Sign Language Translation (SLT) research has typically focused on pre-segmented clips, a constraint that disconnects models from the continuous reality of real-world communication. To bridge the gap from clips to streams, we formally define Streaming SLT, a realistic task demanding the joint discovery and translation of linguistic events from continuous, untrimmed video inputs. To address this challenge, we propose StreamSLST, the first end-to-end framework for Streaming Sign Language Segmentation and Translation via Deformable Transformer architecture. Unlike cascaded approaches that suffer from error propagation, StreamSLST employs a parallel decoding strategy to jointly optimize temporal segmentation and translation in a single pass. To enable scalable training on long-form videos, we incorporate a Sentence-aware Sliding Window Sampler that crops continuous streams into trainable units. We further design a Tri-modal Visual-Language Pretraining stage that aligns pose dynamics with textual semantics before joint optimization. To support comprehensive evaluation, we conduct experiments on two real-world streaming datasets, BOBSL and How2Sign, together with two newly constructed synthetic streaming benchmarks, Streaming-Phoenix-2014T and Streaming-CSL-Daily. The results demonstrate that StreamSLST consistently outperforms a matched cascaded segmentation-then-translation pipeline. Interestingly, our analysis reveals that joint optimization learns wider, translation-effective temporal support that simple expansion or rescaling of the annotated spans fails to reproduce. Our work establishes the first robust benchmark for continuous sign language understanding and paves the way for accessible real-world communication interfaces. The source code and data are available at https://github.com/streamsl/StreamSLST.

0.42

Maswao: An automatically collected, prepared, and benchmarked dataset for South African Sign Language

cs.CL cs.AI ·2026-11-02 ·Thapelo Sindane, Unarine Netshifhefhe, Vukosi Marivate, Abiodun Modupe, Herkulaas Combrink +1 more
benchmark similarity 0.69
Abstract · 16 Pages including references, and 15 Figures

Recent advancements in sign language processing can be attributed to developments in both large-scale dataset collection and algorithmic complexity in areas such as pose estimation techniques, signal processing through Transformer models, and model optimization. However, only a handful of sign languages, such as the American, British, and German, from over 200 globally recognized sign languages, reap the benefits of such advancements, while the rest lag behind. In this study, we aim to address this technological disparity by proposing a framework for the automatic collection and preparation of sign language datasets, with very minimal human-labour intervention, using the South African Sign Language (SASL) as a testbed. Specifically, we automatically collect SASL video content using public resources such as news broadcasts. Additionally, we convert their accompanying audio streams into text using available Automatic Speech Recognition (ASR) tools. Furthermore, we crop the sign language area in the videos and use gradient descent to locate valleys of non-motion to determine end-of-sentence points within long videos to create our weakly sentence-aligned parallel corpora -- the Maswao (Signs) dataset. Finally, we benchmark the weakly aligned dataset on the task of Sign-to-text Translation using Recurrent Neural Networks and Transformer models to compare their individual intricacies. Our results show that Transformer models proposed in the literature outperform recurrent baselines, with the best-performing Transformer achieving BLEU-1 to BLEU-4 scores of 13.78, 5.21, 2.52, and 1.54, respectively, as well as ChrF = 18.09 and ROUGE-L = 10.51. These results indicate that the task remains promising and warrants further investigations.

0.20

Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos

cs.CV ·2026-09-25 ·Bowen Guo, Shiwei Gan, Yafeng Yin, Xiao Liu, Kuizhuang Liu +2 more
top venuebenchmark similarity 0.69
Abstract

Recent advances in sign language understanding have achieved impressive success on short, single-sentence videos, yet their performance drops sharply when applied to long, continuous sign language videos. To bridge this gap, we focus on a challenging and realistic setting: Visual-only Sentence-level Sign Language Segmentation (Vis-SSLS), which aims to partition continuous sign language videos into non-overlapping sentence-level segments without any caption assistance, serving as a crucial prerequisite for downstream recognition and translation tasks. However, sentence transitions in sign language are often smooth and visually ambiguous, lacking explicit pauses or posture resets. As a result, static frame representations may fail to capture the subtle temporal changes that indicate sentence boundaries. To address this challenge, we propose \textbf{SignShift}, a difference-aware segmentation framework that explicitly models frame-to-frame feature variation as semantic cues for sentence boundary detection. First, to model the feature variation, we design a Temporal Difference Module, which incorporates full-frame, facial, and hand cues, and employs inter-frame differencing to learn multi-scale temporal variations that capture both fine-grained local kinematics and global semantic transitions. Second, to mitigate over- and under-segmentation issues, we design a Segment Count Prediction module, which predicts the number of sentences to guide boundary selection. Extensive experiments on benchmark datasets demonstrate that SignShift substantially outperforms existing methods, validating its effectiveness.

0.25

Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production

cs.CL cs.AIcs.CVcs.MM ·2026-08-10 ·Xiao Liu, Shiwei Gan, Yafeng Yin, Jiaxin Yin, Bowen Guo +3 more
top venue similarity 0.68
Abstract

Recent advances in sign language (SL) research have shown a trend toward unifying multiple sign language understanding (SLU) subtasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), within a single framework, leading to substantial progress. Meanwhile, sign language production (SLP), which generates sign sequences from text, has also attracted growing attention. This naturally raises an important question: can sign language understanding and production be unified within a single framework? Compared with unifying SLU subtasks, this problem is substantially more challenging. Existing SLU tasks largely share the same direction of mapping, namely from sign inputs to linguistic outputs, whereas SLT and SLP lie in opposite directions of sign-text mapping. A unified framework must therefore address two key challenges: (1) bridging the modality gap between continuous sign motions and discrete text tokens through a shared sign tokenizer that supports both linguistic abstraction and motion reconstruction; and (2) learning a single conditional autoregressive model that can take either sign or text as input and generate the corresponding target sequence in the opposite modality. To this end, we propose Uni-SLTP, a unified framework for SLT and SLP with two key components: (1) a shared sign tokenizer that converts sign sequences into discrete tokens and latent representations, capturing both semantic and reconstructive information; and (2) a unified autoregressive generation model that formulates both tasks as conditional sequence generation. Experiments on widely used public datasets show that Uni-SLTP achieves superior motion accuracy for SLP while maintaining competitive SLT performance.

0.34

SignMatch: Matching Dictionary Signs to Continuous Sign Language Video

cs.CV ·2026-09-01 ·Ryan Wong, Youngjoon Jang, Liliane Momeni, Gül Varol, Andrew Zisserman
benchmarkcode/data similarity 0.68
Abstract · 24 pages, 8 figures, Project page: https://www.robots.ox.ac.uk/~vgg/research/signmatch/

The objective of this paper is to match dictionary sign videos to corresponding signs in continuous signing videos, where a match is defined by the visual similarity alone - the handshape and motion relative to the body. To achieve this, we learn a prototype-structured sign embedding space from continuous video annotated with signs, where each learnable prototype corresponds to a sign class. Isolated dictionary videos are then mapped into this sign space, enabling the matching between dictionary exemplars and continuous sign instances. This design supports direct dictionary-guided sign matching through embedding similarity and naturally extends to unseen signs using only dictionary exemplars. Experiments on ASL-Citizen dictionary retrieval, ChaLearn OSLWL dictionary-to-continuous sign matching, and using BOBSL's CSLR2 evaluation for automatic sign annotation demonstrate strong generalisation across datasets, tasks and sign languages. Without benchmark-specific supervision, the learned representation transfers effectively across American, British, and Spanish Sign Languages, outperforming prior methods on all three benchmarks. Project page: https://www.robots.ox.ac.uk/~vgg/research/signmatch/

0.54

SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs

cs.CV cs.AI ·2026-08-10 ·Shiwei Gan, Xiao Liu, Yafeng Yin, Zhiwei Jiang, Bowen Guo +3 more
top venue similarity 0.67
Abstract

Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks. However, fine-tuning LLMs for Gloss-Free Sign Language Translation (GFSLT) remains a challenge. In this paper, we investigate how to effectively adapt LLMs to the GFSLT task. We show that there are two key issues that need to be solved: (1) the inherent distributional gap between visual feature inputs and text feature inputs makes it difficult for LLMs to interpret visual inputs; and (2) existing approaches typically concatenate visual and textual features in an autoregressive framework, which leads to the model overemphasizing textual inputs and deprioritizing visual cues, as LLMs are pretrained predominantly on text-centric data. To address the first challenge, we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining, which leverages filtered pseudo-gloss sequences generated from text sequences to supervise the training of the visual backbone. To tackle the second issue, we introduce a Visual-Prioritized Distillation training strategy. Specifically, we define a visual-only prediction path in which text inputs are masked, and the model is required to generate the target sequence relying solely on visual inputs. To guide this path, the outputs from the standard visual-textual prediction are then distilled into the visual-only prediction path, encouraging the model to prioritize visual features. Comprehensive experiments and qualitative analyses demonstrate the effectiveness of the proposed model. The proposed SignLlama achieves very competitive performance on multiple datasets for GFSLT tasks, without using any extra modalities or external sign language datasets for pretraining.

0.64