CS Paper Radar

Omni2Web: Benchmarking Audiovisual Website Development

Minghao Han, Zhenghao Xing, Xize Cheng, Yuxuan Wang, Junming Lin, Ling Wang, Yinsong Yan, Yunfei Chu, Qize Yang, Jin Xu

cs.SE cs.CV ·submitted 2026-09-20 07:18 UTC · v1
benchmark

Abstract

Screen-recorded web editing requests contain weak deictic expressions such as ``this'' and ``there,'' whose referents depend on speech, cursor trajectories, page state, and edit history. Such requests require intent recovery beyond the explicit specifications assumed by many existing web-editing benchmarks. We introduce Omni2Web, a bilingual benchmark of 918 instances spanning 13,907 edit steps. It defines three complementary tracks: Direct Editing evaluates webpage editing from recordings, Instruction Recovery measures explicit intent recovery, and Instruction Utility tests whether recovered instructions can drive a fixed code executor. We evaluate 17 open- and closed-source models. The best models attain 51.17 on the Edit Fidelity Score (EFS) for Direct Editing and 49.14 on the Instruction Recovery Score (IRS); under the fixed executor, the strongest recovered instructions reach 51.08 EFS, still far below the 89.69 EFS obtained with oracle instructions. Step-level analyses show that correct grounding does not guarantee successful edits, while some Omni models recover instructions that the fixed coding model executes substantially better than their direct edits. Controlled ablations further demonstrate the value of temporally aligned audiovisual evidence, while alternative judges preserve the leader and broad ordering. Together, these findings reveal substantial headroom in multimodal intent recovery and code execution and highlight the promise of pairing Omni rewriters with coding models.

Comment
32 pages, 6 figures, 18 tables

Closest papers

Nearest neighbours by meaning across the whole index.

VoxCanvas-41M: A Large-Scale Bilingual Dataset for Instruction-Based Speech Editing and Its Learning Dynamics

cs.SD eess.AS ·2026-11-05 ·Zhikang Niu, Wenming Tu, Yalin Li, Yanru Huo, Junxi Liu +6 more
datasetcode/data similarity 0.66
Abstract · 22 pages. Project page: https://zhikangniu.github.io/voxcanvas/

Recent advances in speech generation have spurred the development of speech editing techniques. However, achieving high-quality speech editing remains challenging due to two key limitations. First, the major hurdle is the lack of large-scale, high-quality speech editing datasets. Constructing such datasets is costly, while open-source, scalable data pipelines remain limited. Second, the learning dynamics of speech editing models remain poorly understood. In particular, how models acquire editing capabilities and the relationship between generation and editing remain unclear. To address these challenges, we introduce VoxCanvas-41M, a large-scale bilingual dataset comprising approximately 41 million speech editing pairs. VoxCanvas-41M covers two major categories: content editing, including single-span, multi-span, and location-reference editing, and paralinguistic editing, covering the insertion and deletion of 39 classes of nonverbal events such as laughter, sighs, and coughs. Building on VoxCanvas-41M, we train VoxCanvas-Edit, an instruction-based speech editing model, demonstrating the effectiveness of our dataset in improving editing performance. Furthermore, we systematically investigate the learning dynamics of speech editing, examining how models learn different editing operations and how data scale, task interactions, and training strategies affect editing performance. The dataset, pipeline, and demos are publicly available.

0.35

IndicQE-APE: A Consolidated Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages

cs.CL ·2026-08-17 ·Diptesh Kanojia, Archchana Sindhujan, Sourabh Deoghare, Daria Sokova, Shenbin Qian +12 more
publishedbenchmarkcode/data similarity 0.65
Abstract · Accepted to Eleventh Conference on Machine Translation (WMT) @ EMNLP 2026; 10 pages body and 27 pages including appendix

Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and language pairs on one footing. We consolidate the WMT 2020-2024 shared-task lineage with an extended English-Malayalam resource into IndicQE-APE: $126{,}754$ instances over nine directional pairs, with up to four label types aligned on the same segment, a direct assessment, a human post-edit, word-level tags and an error explanation, and a test set stratified over four difficulty axes. We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE. Two of the axes are defined partly on direct assessment and select a compressed slice of it. Segments whose segment-level and token-level signals disagree are ranked below equally scored segments of the same language. Four-shot prompting costs every model at or below $3.4$B both correlation and output-format compliance. Unedited MT beats every APE system we run on three of the four pairs. The benchmark (https://huggingface.co/datasets/surrey-nlp/IndicQE-APE) and code (https://github.com/surrey-nlp/IndicQE-APE) are released.

0.65

ProCraft: Instruction-Driven Music Production from Free-Form Text

cs.SD eess.AS ·2026-11-05 ·Yuan-Chiao Cheng, Yi-Hsuan Yang, Vincent K. M. Cheung
similarity 0.65
Abstract · 27 pages, 6 figures, 11 tables

Music production typically involves multiple editing operations that each require specialized tools and expertise. Existing text-guided music editing systems support few operations or instruments, operate on short clips, or handle only one operation per instruction, unlike real workflows where several changes are requested at once. Here, we present ProCraft, a lightweight 0.5B-parameter autoregressive codec language model that edits 30s music excerpts end-to-end from free-form instructions, including compound requests. ProCraft supports seven operations across eight instruments, from tasks with a unique target (e.g., track removal) to creative ones (e.g., accompaniment generation). We align intermediate representations with pretrained music embeddings and pitch and instrument targets to preserve content outside the edit, and gated cross-attention in every decoder layer to keep generation tied to the instruction. To train on compound requests, we introduce On-the-fly Editing-pair Synthesis, which generates paired instructions and audio combining track-level operations with effect changes. ProCraft performs competitively against task-specific systems on single-task editing, and outperforms four free-text editors on compound instructions for track removal, extraction and denoising on real (MoisesDB) and synthetic (Slakh) recordings. In a user study (16 listeners), ProCraft was rated highest on removal and denoising and on-par with a 16-times larger model on extraction and instrument transfer. Ablations further show that representation alignment and cross-attention reduce leakage from unwanted tracks and raise editing success rates, especially on out-of-domain data.

0.11

LEDGE: Separating Critic and Editor Failures in Text-to-Image Self-Correction

cs.CV ·2026-11-16 ·Vyom Agarwal, Mokshda Gangrade, Samet Ayhan, Giacinto Paolo Saggese
similarity 0.64
Abstract

Text-to-image (T2I) self-correction combines a vision-language model (VLM) critic that diagnoses errors and proposes instructions with an image editor that executes them. End-to-end evaluation obscures whether failures arise from incorrect diagnoses or an editor unable to execute valid instructions. We introduce LEDGE (Learning Error Diagnosis in Generation and Editing), a framework that separates these bottlenecks. LEDGE compiles oracle edit instructions from GenEval while holding the generated images and the editor fixed, isolating the editor's correction capability from the critic's diagnostic accuracy. On Stable Diffusion 3.5 Large, oracle instructions raise GenEval from 72.0% to 81.2%, versus 74.1% for the baseline VLM critic. Yet 435 of 644 failed images remain unrepaired, exposing a substantial editor gap. We also use oracle labels from disjoint training prompts to supervise the critic. Oracle supervision raises exact diagnosis from 33.3% (the same VLM with our structured output, untrained) to 70.8%. With the same editor, the resulting critic improves GenEval from 42.2% (uncorrected) to 59.4% on Stable Diffusion 1.5, against 50.9% for the baseline critic, and from 72.0% to 76.0% on Stable Diffusion 3.5 Large, despite training only on Stable Diffusion 1.5. Improvements transfer to unseen object classes, a second editor, and independent T2I-CompBench judges and prompts. A small human evaluation confirms the ranking in estimated net corrections but suggests a smaller gain than detector-based evaluation. LEDGE thus distinguishes two limits of self-correction: knowing what to fix and being able to fix it.

0.32

Confident but Wrong: A Constrained Decoding Diagnostic for Low-Resource Automatic Post-Editing

cs.CL ·2026-08-30 ·Isuru Wijesiri, Nisansa de Silva, Kavindu Warnakulasuriya, Aloka Fernando, Surangika Ranathunga
top venue similarity 0.64
Abstract

Automatic Post-Editing (APE) for low-resource languages (LRLs) often fails to improve Machine Translation (MT), and the score alone cannot say why: whether more training would help, or whether the training data is too inconsistent to learn from. We introduce a black-box, inference-time diagnostic that tells these two cases apart without retraining or annotation. It varies an edit-distance penalty $λ$ that drives the model from free editing towards copying the MT, and reads two signals: (1) the shape of the Translation Edit Rate (TER)-vs-$λ$ curve, U-shaped if edits from the model reduce error and monotonically decreasing if none does; and (2) the ordering of constraint variants that trust model confidence to increasing degrees, which shows whether confidence tracks edit quality. Across decoder-only and encoder-decoder models on English-Sinhala, the diagnostic exposes two failure modes consistent with a heterogeneous post-edit signal as the underlying cause: Binary Collapse, where the model copies the MT or makes off-target edits, and Confident Miscalibration, where the confidence signals we test do not separate useful edits from unnecessary ones. The pattern holds on English-Marathi and English-Tamil, with the failure modes tracking the post-edit distribution rather than MT quality or language family. Beyond diagnosis, the curve shape prescribes a concrete next step for practitioners; in the favorable case, a static constraint yields a free inference-time accuracy gain. We release the first English-Sinhala (~66k) and a new English-Tamil (~39k) APE datasets with all code.

0.37

WeVisCut: The Whole Editing Room in One Agent From Narration to Long-Form Video

cs.CV cs.AI ·2026-10-28 ·Chenyu Mu, Wenming Weng, Qinrui Wang, Xingsong Ye, Hao Yu +6 more
benchmarkcommunity pick similarity 0.63
Abstract

Producing long-form videos from narration requires interdependent decisions that remain coherent across shots. Existing agentic editors often use multi-stage workflows driven by general models, leaving the quality-cost trade-off of compact, task-adapted policies unclear. We address this gap with WeVisCut, a long-form script-to-video system that connects tool execution with policy learning. Its agent harness maintains shared production state so that tool feedback informs subsequent decisions and repairs. Because finished videos do not expose these decisions, our four-stage data pipeline combines editing inversion with verified replay to turn recovered editing intent into executable supervision. Applied to 3,750 finished videos, this pipeline yields over 130K training examples. We use this supervision to train a single 9B policy that coordinates production and recovers from failures within the same harness. Moreover, we introduce a 100-task benchmark and five metrics assessing editing quality and delivery reliability. Under a shared tool environment, WeVisCut substantially outperforms its base model and achieves comparable performance to Claude Opus 5 while using nearly 33x fewer generated tokens per successful output minute. It also outperforms task-matched native editors on our proposed metrics.

0.40

SMEBENCH: Benchmarking Neural Music Editing with Synthetized MIDI Edits

cs.SD cs.LGeess.AS ·2026-10-27 ·Matthieu Cervera, Francesco Paissan, Micael Antunes, Mirco Ravanelli, Ricard Marxer +1 more
benchmarkcode/data similarity 0.63
Abstract

Neural music editing requires a system to perform a requested change while preserving the musical content that should remain unchanged. Existing evaluations often lack a paired target edit, making these requirements difficult to measure separately. We introduce SMEBench, a large-scale benchmark of controlled music edits constructed by modifying symbolic MIDI performances and rendering matched source and target audio. Each example contains a source, an instruction, a paired target, edit metadata, and complementary descriptions. The benchmark covers twelve instrumentation, articulation, dynamics, and pitch operations and includes a balanced 3,600-example test set together with a smaller real-audio complement derived from multitrack recordings. The paired targets allow source preservation, target agreement, and textual compatibility to be evaluated independently. We use these references to diagnose common audio-editing metrics and benchmark six representative editing systems, showing that the metric used to rank a system can materially change the conclusion. The dataset is available at our repository.

0.24

CoinVE-200K: A Large-Scale High-Quality Dataset for Compositional Instruction-Guided Video Editing

cs.CV ·2026-08-18 ·Fuchen Long, Cong Wang, Zitao Gao, Wenhao Zhong, Yu Cheng +6 more
benchmarkcode/datahighly cited similarity 0.62
Abstract · Project page: https://coinve200k.github.io and Dataset is available at https://huggingface.co/datasets/FireCRT/CoinVE-200K and see source co…

The quality and diversity of instruction-based video editing datasets are steadily improving, yet existing datasets mainly focus on single editing operations and fall short in supporting compositional instruction-guided video editing. In particular, multiple editing intents must be jointly understood and faithfully executed within the same video. To address this issue, we introduce CoinVE-200K, a large-scale, high-quality dataset for Compositional Instruction-Guided Video Editing. CoinVE-200K contains 1080p video-editing pairs of up to 201 frames, covering diverse compositional scenarios where each sample involves 2 to 5 atomic editing operations. The instructions target humans, objects, and backgrounds, and cover edit types such as addition, removal, modification, and stylization. All samples are built through a carefully designed generation and filtering pipeline to ensure instruction faithfulness, visual quality, temporal consistency, and compositional diversity. We also introduce CoinVE-Bench, a benchmark for compositional-instruction video editing across diverse subjects, operation types, and instruction complexities. Furthermore, we present CoinVE-Edit, a 22B compositional video editing model built upon Wan2.1-T2V-14B and Qwen3-VL-8B-Instruct. CoinVE-Edit disentangles region-aware attention for different editing instructions, enabling precise multi-region editing while preserving irrelevant content and temporal coherence. Experiments on CoinVE-Bench show that CoinVE-Edit achieves strong performance in instruction following, compositional editing accuracy, visual quality, and temporal consistency.

0.54