CS Paper Radar

CAPQ-FAST: Content-Adaptive Perceived Quality Assessment for Faster Audiovisual Playback

Jiarun Song, Yuxin Song, Fuzheng Yang, Weisi Lin

cs.MM ·submitted 2026-09-03 08:01 UTC · v1
published

Abstract

Faster playback has become a common feature in modern online audiovisual services, allowing users to consume content in less time while still maintaining a coherent viewing experience. However, different modalities of media content, such as video, audio (including speech and music), and audiovisual, exhibit varying requirements for understandability and information integrity under faster playback. These differences lead to noticeable variations in perceived quality depending on the content type. Nevertheless, users' perceived quality at different playback speeds remains insufficiently investigated. To address this gap, this paper conducts a series of subjective experiments to analyze the relationship between playback speed and perceived quality for video, audio, and audiovisual content. Content-specific intrinsic features are extracted to capture temporal dynamics, including temporal information (TI) for video, words per minute (WPM) for speech, and beats per minute (BPM) for music. Predictive models of perceived quality under faster playback are then developed separately for video, speech, and music. By integrating these models, a unified content-adaptive perceived quality assessment model (CAPQ-FAST) is proposed for faster audiovisual playback. Experimental results demonstrate that the proposed model effectively predicts perceived quality under different playback speeds. This model can help service providers better understand users' viewing intentions and perceptual experiences under faster playback, thereby enabling more adaptive personalized recommendations and playback control to enhance user experience adaptability.

Comment
IEEE Transactions on Circuits and Systems for Video Technology, doi: 10.1109/TCSVT.2026.3716481
DOI
10.1109/TCSVT.2026.3716481

Closest papers

Nearest neighbours by meaning across the whole index.

PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation

cs.MM cs.AIcs.SD ·2026-09-04 ·Yuchen Sun, Qian Yang, Jun Wang, Detai Xin, Guoqiao Yu +2 more
top venuebenchmarkcode/data similarity 0.62
Abstract · 19 pages, 10 figures, 4 tables. Accepted at ACM Multimedia 2026 (MM '26). This arXiv version includes supplementary appendices not included …

Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing benchmarks either treat audio as an auxiliary component of video quality or assess it in isolation from audiovisual grounding, making it difficult to diagnose where current systems truly succeed or fail in audio generation. We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation. Built from a rigorously curated dataset of 900 human-verified samples, PRISM-Bench factorizes audio evaluation along two orthogonal axes: audio type (Speech, Music, and Sound) and sound-source visibility (On-screen vs. Off-screen). It evaluates generated content across four perceptual dimensions (Audio-Visual Coherence, Audio Quality, Audio Expressiveness, and Prompt Following) with 35 fine-grained criteria. To ensure reliable assessment, we adopt an enhanced MLLM-as-a-Judge protocol based on blind, side-by-side comparison against ground-truth references, demonstrating strong alignment (over 70% mean agreement) with human raters. Our evaluation of recent T2AV systems highlights a significant performance gap between frontier and open-source models. Furthermore, we demonstrate that current generation paradigms overfit to perceptual fidelity while struggling with complex grounding and control tasks, particularly in generating music and synchronized On-screen audio.

0.46

SemABR: Measuring Video Semantic Fidelity with Multimodal LLMs for Adaptive Bitrate Streaming

cs.MM cs.NI ·2026-09-16 ·Shiqi Xu, Soung Chang Liew, Yuyang Du
similarity 0.59
Abstract

Conventional video metrics such as PSNR, SSIM, and VMAF measure visual distortion or perceptual quality, but they do not directly capture semantic preservation: whether compression retains a video's objects, actions, and temporal narrative. Existing Quality-of-Experience (QoE)-driven bitrate-selection and resource-allocation methods primarily aim to minimize rebuffering and bitrate switching while maximizing perceptual video quality, without explicitly considering semantic preservation. To address this gap, we introduce video semantic fidelity (SF), a metric that quantifies how well a compressed video preserves the semantic content of its source. An offline multimodal large language model (MLLM) generates structured descriptions of the reference and compressed versions of the video, and a separate text-only large language model (LLM) evaluates their semantic correspondence. The resulting content-dependent SF--bitrate profiles are cached and queried by the online bitrate selector without invoking MLLMs at runtime. Evaluations on three subjective QoE benchmarks show a consistent positive association between SF and mean opinion scores (MOS). A separate human semantic-rating study evaluates semantic preservation and shows that SF correlates more strongly with human judgments than conventional video metrics. We then embed these profiles into a 5G MEC-assisted video-on-demand (VoD) resource-allocation framework at the base station. When wireless resources cannot support high bitrate levels for all users, the framework uses the SF--bitrate profiles to select bitrate levels jointly across users and reduce the semantic loss caused by the required bitrate reductions. NS-3 simulations with the 5G NR module show that the proposed framework achieves higher average and worst-user SF than the evaluated baselines, with a widening advantage as the wireless resources available to each user decrease.

0.32

Are Caption Metrics Broken? Latency, Deaf and Hard of Hearing User Ratings, and Bias across Technologies

cs.HC ·2026-09-10 ·Bernard Thompson, James Waller, Luz Fanny Calderon Torres, Lu Ming, Mariana Arroyo Chavez +3 more
published similarity 0.59
Abstract · Accepted for publication at ACM ASSETS 2026

Live captions on TV often contain errors and timing issues, making it hard for deaf and hard-of-hearing (DHH) viewers to follow dialog. It is essential that caption quality metrics reflect the lived DHH TV viewing experience. To this end, we describe a U.S.-based large-scale online survey with 216 validated participants, who provided 302 responses containing a cumulative 4,832 data points. Participants viewed videos drawn from a pool of 70 clips recorded from live TV, and were asked to rate the caption quality and subjective understanding of the content across four conditions: TV captions as originally recorded with up to 7-12 seconds delay, TV captions synchronized with audio, Automatic Speech Recognition (ASR)-generated captions synchronized with audio, and ASR captions with an average two-second delay. All captions were evaluated against the Word Error Rate (WER), Automated Caption Evaluation (ACE2) and Number, Edition and Recognition (NER) metrics. Results show that TV and ASR captions were rated similarly. For TV captions, all three metrics were moderately-to-highly correlated with viewer ratings, but far less so for ASR captions, making them far from technology-neutral. Additionally, caption latencies significantly impact the viewer experience, especially typical 7-12-second TV delays. We discuss the implications for the adoption of caption quality metrics.

0.39

PADP: Perceptual Audio Data Perturbation for Probing Perception Awareness in Audio Quality Models

eess.AS ·2026-10-01 ·Guanxin Jiang, Andreas Brendel, Pablo M. Delgado, Jürgen Herre
similarity 0.59
Abstract · 5 pages, 3 figures

This paper presents a collection of audio transforms that introduce perceptually irrelevant distortions and demonstrates their use as perceptual stress tests for audio quality models. We refer to these methods as Perceptual Audio Data Perturbation (PADP). PADP exploits the insensitivity of the human auditory system to certain fine-grained signal variations to substantially alter the waveform while preserving perceived audio quality and content. The audibility of PADP and the selection of its parameters are evaluated through controlled listening tests, ensuring that the transformations achieve transparent or near-transparent quality for both non-critical and critical items. We further probe the robustness of state-of-the-art (SOTA) perception-motivated objective audio quality models and foundation models. The results reveal a misalignment between model responses and human auditory perception, highlighting the limited perceptual awareness of these models for certain proposed transforms.

0.25

StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generation

cs.SD cs.CVcs.MM ·2026-08-26 ·Kaiqi Liu, Haoxuan Zeng, Jingqi Liu, Jiacong Fang, Ziqi Cai +5 more
benchmarkhighly cited similarity 0.58
Abstract

Recent advancements in generative models are pushing video generation toward unbounded streaming audio-video generation for real-time interactive worlds. However, existing benchmarks primarily evaluate completed sequences and struggle to capture streaming properties. To bridge this gap, we introduce StreamAV-Bench, the first comprehensive benchmark tailored for streaming audio-video generation. StreamAV-Bench establishes a unified evaluation framework, including the progressive track for instruction adherence and long-horizon stability, and the interactive track for interactive response and state retention and reuse. With expert-verified evaluation cases across 32 fine-grained dimensions, we conduct an extensive evaluation of 13 representative systems. Our analysis reveals that current models suffer from temporal drift in progressive generation and responsiveness bottlenecks during interactive control. Based on a comprehensive failure analysis, we share insights to advance the development of native joint audio-video streaming models.

0.42

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

cs.CV cs.AI ·2026-10-15 ·Yuruo Wu, Kaihang Pan, Kai Shen, Jiacheng Li, Shuyuan Tu +4 more
benchmarkcode/data similarity 0.58
Abstract

As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/

0.34

AVTR-1: Open Stack for Real-Time Interactive Avatars

cs.CV cs.AIcs.LG ·2026-09-19 ·Artem Kravtsov, Dmitrii Ziganshin, Vsevolod Poletaev, Gleb Balitskiy, Anastasia Tikhonova +2 more
similarity 0.57
Abstract

Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live system must synchronize the model's output with speech from an external voice agent, schedule video frames for playback, and handle interruptions. We introduce AVTR-1, an open stack for real-time interactive avatar conversations, built around a compact 153M-parameter autoregressive flow-matching motion generator conditioned on both participants' audio. We adapt its audio encoder for streaming through self-distillation. The stack turns the model's chunk-based generation into a continuous, synchronized audio-video stream driven by an external voice agent, and we analytically derive its contribution to the user-facing latencies and validate the resulting bounds with two commercial voice agents. Further experiments demonstrate that AVTR-1 leads the compared dyadic systems on all reported visual-quality metrics and most conventional listening-motion metrics while remaining competitive in lip synchronization. Its inference runtime operates in real time on data-center and consumer GPUs. However, conventional listening metrics do not establish whether the paired speaker's speech contributes to generated motion. We therefore introduce the Reference-Based Directed Granger Gain (R-DGG), which measures the additional predictive information carried by speaker speech after accounting for listener history and speaker motion. R-DGG finds statistically supported predictive dependence for recorded listeners and all evaluated dyadic systems, but not for talking-head generators without paired audio or mismatched speaker-listener pairs. We release the model weights, renderer, and serving backend under component-specific licenses.

0.41

Evaluating perceptual transparency of lossy music audio compression with Opus codec

cs.SD ·2026-10-14 ·Benjamin Martin, Romain Hennequin
code/data similarity 0.57
Abstract · This work has been submitted to the IEEE ICASSP 2027 for possible publication. Copyright may be transferred without notice, after which this…

Although the open-source Opus audio codec is widely used for digital audio streaming, its perceptual transparency threshold for high-bitrate music compression remains insufficiently evaluated. To establish this threshold, this study evaluates Opus across a two-phase ITU-R BS.1116-3 double-blind framework, analyzing Difference Grades using a Linear Mixed-Effects Model (LMM) and Two One-Sided Tests (TOST) for statistical equivalence. In a first phase with 21 qualified listeners and 23 music excerpts, 128 kbps achieves statistical in-distinguishability ($p = 0.456$) but narrowly fails strict equivalence to the uncompressed reference ($p = 0.064$). In Phase 2, Best Linear Unbiased Predictors (BLUPs) isolate the 10 most critical listeners and 8 most challenging samples for a stress-test. Under these worst-case conditions, 128 kbps is conclusively detected as degraded ($p = 0.030$), whereas 160 kbps successfully reaches perceptual transparency ($p = 0.747$) and strict statistical equivalence ($p = 0.024$). The study concludes that 160 kbps demonstrates perceptual transparency to uncompressed CD quality in critical listening environments. For reproducibility, we publicly release results for both phases, anonymized subject background info and sample characteristics.

0.31