CS Paper Radar

StreamStudio: Exploring Video Authoring with Streaming Autoregressive Generation

Boyu Li, Zhe Lin, Eli Shechtman, Ding Li, Hongbo Fu, Yuqian Zhou

cs.HC ·submitted 2026-10-28 08:43 UTC · v1

Abstract

We present StreamStudio, a prototype system that explores a new authoring paradigm for streaming video generation. Instead of entering a prompt and waiting for a complete video before making changes, creators can interact with the generation as it unfolds: steering what comes next, returning to earlier moments, and exploring different continuations. In this way, StreamStudio shifts creative control from post-generation revision to continuous interaction with the generative process. Our contribution centers on this authoring paradigm and the interaction mechanisms that make streaming generation directly manipulable. StreamStudio is designed as a plug-and-play interface that can work with compatible generation and editing models, allowing its interaction model to evolve alongside advances in underlying video models. As streaming models become faster and easier to control in the future, such an interface may support more flexible and controllable video creation, including richer edits and multimodal interventions during generation.

Closest papers

Nearest neighbours by meaning across the whole index.

EditStream: A Unified Autoregressive Framework for Interactive Video Generation and Editing

cs.CV cs.GRcs.HCcs.LG+1 ·2026-08-16 ·Yuqian Zhou, Zhenghong Zhou, Zongze Wu, Cameron Smith, Richard Zhang +3 more
code/datahighly cited similarity 0.75
Abstract · 25 pages, 12 figures, Project page: https://real-time-video-research.github.io/editstream/

Interactive video generation and editing are becoming increasingly important for creative design. In this report, we introduce EditStream: a unified framework for interactive video generation and editing. EditStream unifies multiple video creation and manipulation tasks within a single DiT-based model through flexible task-specific conditioning, and further transforms it into a fast, few-step autoregressive model for efficient streaming. It supports Text-to-Video, Image-to-Video, Video-to-Video, Editing Propagation, Reference-guided Video Editing, and Camera Pose Change, enabling flexible control over video generation, transformation, and editing within one system. To make the unified model practical for interactive use, we develop a two-stage distillation approach that combines Velocity Moment Matching (VMM) with autoregressive unrolling. VMM matches conditional velocity moments at student-reached intermediate states to preserve generation quality and motion, while unrolling exposes the student to its own autoregressive predictions to improve temporal stability. Together, they alleviate common challenges in few-step autoregressive video generation, including over-saturation, degraded motion, temporal instability, and complex training. EditStream provides a practical and scalable solution that bridges high-quality diffusion-based video models with interactive creative workflows.

0.64

CraftTrace: Unflattening Videos into Malleable, Creation-Inspired Structures for Generative Editing

cs.HC ·2026-09-24 ·Boyu Li, Yuqian Zhou, Duotun Wang, Ding Li, Zhe Lin +4 more
similarity 0.75
Abstract

Recent generative video editing models enable video content modification (e.g., changing a character) but target short clips. Extending them to full multi-shot videos requires tedious work to locate relevant content across shots, segment it into clips, craft context-aware editing prompts for each clip, and repeatedly articulate complex editing intent. To address this, we explore an interaction paradigm for editing through underlying video structures (e.g., scripts, scenes, characters, shots, and their relationships). We present CraftTrace, an interactive prototype that transforms a video into a malleable, multilevel structure for generative editing. Users work in task-centric workspaces to modify elements or reshape relationships, while an AI agent translates and propagates changes across the video. A user study and expert review show that this structure helps users understand videos, formulate and refine editing intent, and explore alternatives, supporting rapid prototyping during early-stage exploration and full video post-production.

0.30

Code World Model 2.0: Towards Infinite Controllable Video Generation

cs.CV ·2026-11-17 ·Ruoyu Wang, Hao Zhong, Yanming Yang, Jiahao Tian, Lin Liu +6 more
code/data similarity 0.69
Abstract · Project Page: https://www.inclusion-ai.org/CodeWorldModel/

Visual proxies decouple explicit world-state simulation from photorealistic video synthesis, offering a controllable interface for both offline clip creation and real-time interactive streaming. Yet a practical renderer must address two challenges. Upstream proxies vary from coarse layouts to fine articulated geometry depending on the interaction, and streaming inference must operate in a few causal steps without accumulating drift across self-generated history. We present Invoker-Render, pairing a bidirectional creation model with a causal four-step streaming model under a shared conditioning interface over structural proxies, an appearance reference, and language. Trained on an aligned approximately 200-hour multi-granularity corpus, Invoker-Render honors fine geometric constraints when supplied while synthesizing plausible local motion under coarse proxies, enabling runtime selection of input detail. To bridge bidirectional creation and real-time streaming without compounding step compression and rollout drift, we introduce Progressive Forcing (ProForcing). ProForcing first trains a causal predictor and distills it into a four-step renderer using histories derived from training videos. Training then shifts to continuous self-generated rollouts. A pipelined serving stack supports efficient streaming inference. Experiments show versatile control across proxy families, granularities, and camera paths, consistent gains over recent streaming baselines, and stable 30-minute racing rollouts that preserve scene and subject appearance.

0.34

Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation

cs.CV cs.LG ·2026-09-10 ·Jintao Zhang, Kai Jiang, Jintao Chen, Xu Wang, Deyuan Liu +30 more
highly cited similarity 0.68
Abstract

We present Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model. Moreover, we explore the feasibility of real-time spatial video generation for both Vidu S2-Avatar and Vidu S2-Editing. Compared with Vidu S1, Vidu S2-Avatar supports real-time 720p video generation, generation with dynamic references that can be updated at any moment, and stronger instruction following, such as dancing. Vidu S2-Editing supports editing a video stream in real time, including style rendering, clothing replacement, character replacement, and background replacement. Experiments show that Vidu S2 outperforms all baselines. A playable online demo is available at https://vidu.com/vidu-stream.

0.43

Live Artifacts: Authoring Dynamic Media via Live Layers Encapsulating Generative Specifications

cs.HC ·2026-08-21 ·Leixian Shen, Haotian Li, Hugo Romat, Fanny Chevalier, Nicolai Marquardt +1 more
similarity 0.68
Abstract

We frame Live Artifacts as a class of persistent generative media between static assets and interactive software. Unlike conventional generative outputs that collapse into static files, Live Artifacts retain their generative logic as a persistent media property, enabling continuous context-dependent regeneration. Time, location, or live data become part of their generative specifications, initiating coordinated updates across modalities (e.g., adapting text, visuals, and audio together) while preserving composition, semantics, identity, and cross-modal coherence. To facilitate experimentation with this medium, we present LiveCanvas, an authoring system that reconceptualizes visual layers as live generative specifications with explicit mutability and constrained dependencies. Creators orchestrate dynamic behaviors and manage generative persistence within a visual canvas rather than through programming, defining what remains stable, what can change, and how changes propagate. We evaluate Live Artifacts through a gallery of responsive examples and a qualitative study with six professionals, finding that LiveCanvas facilitates a shift from composing static outputs to crafting responsive generative artifacts while remaining aligned with familiar authoring practices.

0.07

Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomy

cs.CV cs.MMcs.SDeess.AS ·2026-09-28 ·Abhinav Sharma, Sai Karthik Navuluru, Wang Wei, Daksh Dangi, Xiangbo Gao +19 more
similarity 0.68
Abstract · 36 pages, 3 figures, 15 tables

Video and audio are perceived together, yet most generative models treat them in isolation. We examine methods that model the two modalities jointly, generate one from the other, or edit them in a coupled manner, organized around a single question: how is the output kept coherent across modalities in time and semantics? A unified formulation casts joint generation, cross-modal generation, and joint editing as three problems defined on a single distribution over audio-visual pairs, and a taxonomy compares methods along five design axes. To our knowledge, this is the first overview to systematically taxonomize joint audio-visual editing, which we map as nine edit categories spanning 28 edit types. We describe methods, datasets, and metrics for each setting and close with the open problems we view as most consequential.

0.14

StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generation

cs.SD cs.CVcs.MM ·2026-08-26 ·Kaiqi Liu, Haoxuan Zeng, Jingqi Liu, Jiacong Fang, Ziqi Cai +5 more
benchmarkhighly cited similarity 0.67
Abstract

Recent advancements in generative models are pushing video generation toward unbounded streaming audio-video generation for real-time interactive worlds. However, existing benchmarks primarily evaluate completed sequences and struggle to capture streaming properties. To bridge this gap, we introduce StreamAV-Bench, the first comprehensive benchmark tailored for streaming audio-video generation. StreamAV-Bench establishes a unified evaluation framework, including the progressive track for instruction adherence and long-horizon stability, and the interactive track for interactive response and state retention and reuse. With expert-verified evaluation cases across 32 fine-grained dimensions, we conduct an extensive evaluation of 13 representative systems. Our analysis reveals that current models suffer from temporal drift in progressive generation and responsiveness bottlenecks during interactive control. Based on a comprehensive failure analysis, we share insights to advance the development of native joint audio-video streaming models.

0.42

WanPE: Towards Cinematic Prompt Enhancement for Modern Text-to-Video Generation

cs.CV ·2026-09-24 ·Yubo Zhu, Yawen Shao, Ziyun Dai, Zixun Fang, Kai Zhu +25 more
benchmarkhighly cited similarity 0.67
Abstract

Video generation begins in text space by authoring a cinematic screenplay, then materializes into pixels. As contemporary video generators scale to 30 seconds and faithfully follow complex conditions, the textual prompt largely directs the production, planning how actions, camera trajectories, lighting, and sound unfold across multi-shot sequences. In this paper, we present WanPE, a 397B-parameter prompt enhancement model trained on 1.05M real-world videos to master director-level cinematic planning. WanPE formulates shot-level cinematic plans via video-grounded reverse construction and employs Semantic-Consistency GRPO (SC-GRPO) to faithfully preserve user requirements across shots and over time. To benchmark this capability, we curate WanPEval, a human-annotated testbed covering durations from 5 to 30 seconds across varying intent granularities, supported by approximately 11K blind pairwise assessments. When powering Wan3.0's video generator, WanPE-397B boosts human preference over raw user prompts by 10.66-18.84 points at 5-15 seconds and by a dramatic 50.86 points in the 30-second arena. Ablation studies show that reverse construction demonstrates clear superiority over forward rewriting, while SC-GRPO robustly preserves semantic fidelity across model scales. Ultimately, WanPE leads all evaluated commercial offerings at 5-15 seconds and remains competitive with Seedance 2.5 at 30 seconds.

0.36