CS Paper Radar

Low-Cost Deep Learning for Automated Keyframe Interpolation in Sprite Animation

Kenneth Alonso, Gustavo Patow

cs.GR ·submitted 2026-10-12 08:53 UTC · v1

Abstract

Sprite-based animation remains prevalent in video game development, but manual in-betweening---generating intermediate frames between keyframes---is labor-intensive. Existing video frame interpolation methods, optimized for photorealistic content, introduce smoothing artifacts when applied to the sharp edges, limited color palettes, and stylized motion characteristic of hand-drawn sprites. We present a deep learning approach for sprite in-betweening that adapts a U-Net encoder-decoder architecture with domain-specific modifications, including RGBA-aware processing, multi-scale flow refinement, and alpha-gated blending to preserve pixel-perfect sprite aesthetics. Following data-centric AI principles, we demonstrate that a few carefully curated sprite animation triplets---selected for diversity, quality, and representative edge cases---enable effective training without requiring massive generic datasets. Experimental evaluation using the FLIP perceptual metric shows that our method achieves a mean error of 0.258 on 42 test triplets, outperforming state-of-the-art video interpolation RIFE (0.313, 17.5\% worse), traditional optical flow Lucas-Kanade (0.449, 42.6\% worse), and general frame synthesis CyclicGen (0.603, 57.2\% worse). This study shows that quality-over-quantity data curation and domain-specific architectural adaptations yield better interpolation quality than general-purpose methods trained on orders of magnitude more data, validating the data-centric AI paradigm for specialized visual domains. This is particularly important for small-to-medium-sized game development studios, where budget and time constraints are critical.

Comment
19 pages, 7 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Neural Controlled Directional Frame Generation for Sprite Animation

cs.GR ·2026-10-19 ·Kenneth Alonso, Gustavo Patow
similarity 0.88
Abstract

Sprite-based animation remains prevalent in video game development, but production workflows require capabilities beyond traditional in-betweening. While existing neural interpolation methods can generate intermediate frames, animators also need forward extrapolation to predict motion continuations and backward extrapolation for editing and reverse-engineering existing sprite sheets. Current video generation models either operate on natural video content with stochastic diffusion sampling incompatible with production requirements, or focus exclusively on interpolation without addressing extrapolation tasks. We present a deep learning approach for controlled directional frame generation in sprite animation using three specialized expert models that share identical U-Net encoder-decoder architectures and several domain-specific modifications, including RGBA-aware processing, multi-scale flow refinement, pixel-shuffle upsampling, and alpha-gated blending. Each model is trained on different temporal orderings of the same curated dataset: previous (frames 2,3>1), middle (frames 1,3>2), and next (frames 1,2>3). We demonstrate that only 280 carefully curated sprite triplets enable effective training for all three directional tasks without requiring massive generic datasets. Evaluation on 42 test triplets using the FLIP metric shows that all three models achieve comparable quality--previous (0.26), middle (0.26), and next (0.27)--with no significant degradation for extrapolation versus interpolation, outperforming general-purpose baselines by 17-57%. With deterministic single-pass generation and compact model, our method provides the first practical solution for controllable any-position frame generation specifically tailored for pixel art sprite animation workflows, validating that quality-over-quantity data curation enables specialized deep learning for niche visual domains.

0.16

One Basis to Animate Them All: Gaussian Blendshape Distillation for Real-Time Avatars

cs.CV cs.AIcs.HCcs.LG ·2026-10-01 ·Ramazan Fazylov, Stamatis Lefkimmiatis, Ivan Laptev
code/data similarity 0.65
Abstract

3D Gaussian avatars support fast rendering, however, their real-time animation is often challenged by the costly neural inference. We address this bottleneck and show that the animation of pretrained avatar models can be closely approximated by a linear combination of identity-independent blendshapes. Building on this finding, we introduce GALA (Gaussian Animation via Linear Approximation), a distillation method that replaces per-frame heavy neural decoding with a shallow coefficient predictor and a linear blend. To improve fidelity and reduce memory requirements, we propose to construct the basis using block-local PCA under a rendering-aware metric and a memory budget. Our method learns a shallow MLP network to predict blendshape coefficients and applies to various animation architectures without retraining original models. We validate GALA by accelerating the inference of three distinct avatar models for 3D animation of facial expressions and full-bodies with clothing dynamics. Across these models, our distillation generalizes to held-out identities and reduces CPU animation cost by up to three orders of magnitude while preserving most of the rendering quality. Excellent results of our method confirm the shared linear structure of learned avatar representations and enable highly efficient and accurate animation at frame rates reaching up to 60fps on mobile devices. Project page: https://ramazan793.github.io/gala/

0.29

Alpha as an Efficiency Signal: Visibility-Routed RGBA Image-to-Video Generation

cs.CV ·2026-08-10 ·Zhe Li, Honghao Qiao, Zhixin Xu, Qijie Wang, Bo Peng +1 more
similarity 0.64
Abstract

RGBA videos combine RGB appearance with an alpha channel, enabling animated assets to be applied across arbitrary backgrounds, which are heavily used in gaming industry. However, generating high-quality RGBA animations for games remains challenging for two reasons. First, most existing RGBA video datasets are dominated by photorealistic content, with limited coverage of game assets. Second, the traditional generate-then-matte pipelines estimate alpha only after RGB synthesis, so semi-transparent regions are often blurred by background, resulting in unstable matting outputs. More recently, many methods have begun to model RGB and alpha jointly, but existing approaches are mostly text-conditioned, and still have unresolved issues in efficiency and quality. To address these challenges, we introduce GameAlpha-2.4K, a 2.4K-clip game-style RGBA video dataset built with matte-friendly synthesis, multi-hypothesis alpha recovery, and compositing-based quality gates. Using this dataset, we train a reference-conditioned RGBA video generator that jointly produces RGB frames and alpha mattes in a single pass. To improve efficiency, we propose a visibility router that identifies transparent tokens in an early stage and bypasses their later DiT updates, while x_0-lock guides them along the original flow-matching schedule toward self-predicted endpoints. Our model obtains lower FVD than traditional two-stage pipelines, and the visibility router skips 35% of token evaluations in the final two DiT denoising steps, providing a 1.2x backbone speedup with negligible quality degradation compared to dense inference.

0.04

UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation

cs.CV ·2026-10-02 ·Daikun Liu, Xin Zhan, Teng Wang, Xiaoping Wang, Changyin Sun
top venuecode/data similarity 0.63
Abstract · 19 pages, 6 figures, conference, code: https://github.com/KK-xi/Unidynamics

We propose UniDynamics, a diffusion-based framework for future 4D dynamic scenes (RGB, depth, and optical flow) generation from a single event-RGB pair, without requiring long histories or control priors as in existing methods, while explicitly modeling future motion fields. The core idea is to leverage event streams to offer an alternative motion prior for single-RGB extrapolation, and to enforce geometric and motion constraints throughout generation via multimodal modeling. Specifically, we design an Event Latent Enhancement (ELE) module to align and enhance event latents into diffusion-injectable conditioning features, providing robust initial motion priors and reliable texture/structure cues. We further introduce a Perceptual Dynamics Space (PDS) embedded in the multi-scale U-Net, which decouples and adaptively interacts depth and flow while continuously feeding back constraints to appearance features, improving geometric-motion consistency for physically plausible and spatiotemporally coherent prediction. Experiments on VKitti2 and DSEC demonstrate state-of-the-art performance, producing high-quality, temporally coherent, and 4D-consistent future predictions, especially under challenging high-speed motion blur.

0.53

SNM-VFI: Symmetric Nonlinear Motion-Guided Generative Video Frame Interpolation

cs.CV ·2026-08-13 ·Jisoo Jeong, Hong Cai, Jamie Menjay Lin, Hanno Ackermann, Hyeonjun Sim +3 more
similarity 0.63
Abstract · ECCVW 2026

We propose Symmetric Nonlinear Motion-guided Generative Video Frame Interpolation (SNM-VFI), a training-free framework for motion-controllable generative video frame interpolation with pre-trained optical flow and video diffusion models. Unlike conventional diffusion-based VFI methods that synthesize intermediate frames from random noise, SNM-VFI guides the generative process with correspondence-aware frames produced by a symmetric nonlinear motion model. Specifically, we first utilize a pre-trained optical flow model to construct multi-frame nonlinear flow-based intermediate frames and confidence maps. These flow-guided frames are then encoded as latent priors to initialize and iteratively guide a pre-trained Video Diffusion model, enabling the diffusion model to preserve dense motion correspondence while improving perceptual realism. To further enhance output quality, we employ confidence maps to fuse structurally reliable flow-based predictions with diffusion-generated details in uncertain regions such as occlusions and object boundaries. Extensive evaluations on challenging benchmarks, including DAVIS, Sintel, and KITTI, demonstrate that SNM-VFI achieves strong perceptual quality, competitive reconstruction accuracy, and robust temporal coherence across diverse motion scenarios.

0.04

SenseNova-U1.5: Towards Native Unified Visual Intelligence

cs.CV ·2026-09-10 ·Haiwen Diao, Jiahao Wang, Chenjing Ding, Hanming Deng, Jiangnan Chen +60 more
code/datahighly cited similarity 0.62
Abstract · Project page: https://github.com/OpenSenseNova/SenseNova-U1

We launch SenseNova-U1.5, an 8B-MoT native unified multimodal model that understands, reasons about, and generates visual content within an encoder-free and VAE-free architecture. We strengthen its visual interface through spatially coherent patch reconstruction and scale its training with carefully curated generation and editing data, improved task formulation, structural prompt enhancement, and native resolutions of up to 4K. For post-training, we optimize specialized experts for visual aesthetics, bilingual text rendering, infographic generation, and image editing, and consolidate their capabilities through multi-expert on-policy distillation. Across extensive evaluations, SenseNova-U1.5 largely advances image fidelity, text rendering, complex composition, multi-reference editing, and interleaved generation, while improving instruction following and preserving subject identity, geometry, and unmodified regions. Despite limited exposure to structured formats in its generation data, SenseNova-U1.5 generalizes effectively to long, complex, and structured visual instructions, further proving that multimodal understanding can transfer to visual planning and creation. Together, these findings position native unified modelling as a promising path towards systems that perceive, reason and create within a fully end-to-end framework. We will open-source training code, including supervised fine-tuning, reinforcement learning, and on-policy distillation.

0.44

Exploring the Performance Frontier of Compact Unified Image Generation Models

cs.CV ·2026-08-20 ·Taihang Hu, Zhao Wang, Zuan Gao, Tao Liu, Hao Yan +18 more
code/datahighly cited similarity 0.62
Abstract · 28 pages, 11 figures

We present Swift-Image, a compact unified model for text-to-image generation, single-image editing, and multi-image editing. Our goal is to explore how far a relatively small visual generator can be pushed through systematic training engineering under a constrained computational budget. Swift-Image adopts an efficient 6B single-stream DiT and a progressive training pipeline that evolves from broad semantic coverage to higher resolution, stronger visual quality, and unified generation-editing supervision. For post-training, we employ parallel expert reinforcement learning followed by multi-teacher on-policy distillation to alleviate interference among heterogeneous objectives. We further decouple high-level reasoning from pixel-level rendering with a Prompt Enhancer that translates user requests into generator-aligned visual specifications. For efficient deployment, structural pruning and few-step distillation produce 3B and accelerated variants. Swift-Image achieves leading aggregate performance among evaluated open-source models with only 6B parameters and 243K GPU training hours; the compressed 3B model incurs nearly no loss, while few-step distillation further improves aggregate editing performance with substantially fewer sampling steps. Our study also summarizes practical lessons for architecture, data curriculum, post-training, prompt enhancement, and model compression.

0.48

World2Motion: Turning Video World Models into 3D Human Motion Generators

cs.CV ·2026-09-29 ·Fangyuan Tu, Xiangyue Zhang, Yiyi Cai, Yichen Peng, Kunhang Li +6 more
benchmarkdatasetcode/data similarity 0.62
Abstract · 15 pages, 6 figures

We present World2Motion, a framework that generates scene-aware 3D human motion and corresponding video from a single image and a text prompt. While existing 3D motion generators learn from motion datasets, their generalization is constrained by limited coverage of environments. In contrast, video world models such as Cosmos 3 offer broader environmental priors but are not designed for full-body motion generation; recovering motion from their generated videos requires costly two-stage inference. To address these, we turn Cosmos 3 into a single-stage 3D motion generator. This adaptation has two challenges: the scarcity of paired video--motion data and temporal instability in the generated motion. First, we construct a training dataset combining synthetic video--motion pairs with real videos paired with estimated 3D motion. Second, we propose a shift-decoupled noise schedule that assigns different noise levels to video and motion through shared denoising progress. This design accommodates the different denoising requirements of the two modalities, reducing motion jitter. Experiments on a multi-source interaction benchmark show that World2Motion has better motion--text alignment and scene interaction compared with the evaluated 3D motion generators. It also matches the interaction success rate of the two-stage baseline while achieving approximately 3.3$\times$ faster inference. Our project page is available at https://fyantu.github.io/World2Motion/.

0.37