CS Paper Radar

RESTORE: REal-time Steerable Music resTORation and bandwidth Extension via stem disentanglement

Meiying Chen, Benjamin R. Thompson, Michael C. Heilemann

eess.AS ·submitted 2026-09-23 18:23 UTC · v1
code/data

Abstract

Neural methods for audio restoration are typically framed as rigid mappings from degraded inputs to single clean outputs, enforcing decisions about what audio content is removed, and potentially adding unwanted content to the restored signal. Because what constitutes a restored audio signal is subjective, we introduce RESTORE, a framework that formulates audio restoration as a six-source semantic decomposition to allow for real-time interactive user control over the process. By expanding a pretrained HTDemucs backbone, a single forward pass disentangles a degraded mixture into vocals, music, broadband hiss, impulsive transients, and an unmodeled residual, while jointly synthesizing a high-frequency extension. Users may steer the restoration by adjusting stem gains, ensuring generative content remains isolated and auditable. RESTORE improves audio quality on diverse historical recordings compared to baselines,lowering Frechet Audio Distance (FAD) (12.13 VGGish; 0.92 CLAP) and delivering aesthetic steerability (Spearman rho greather than 0.91) at 50x real-time on a single GPU. Code and audio samples are available at https://melissachen15.github.io/restore-audio-demo.

Comment
Code and audio samples: https://melissachen15.github.io/restore-audio-demo

Closest papers

Nearest neighbours by meaning across the whole index.

Restore, Separate, Restore: A Modular Framework for Music Source Restoration

cs.SD ·2026-10-06 ·Tobias Morocutti, Emmanouil Karystinaios, Gerhard Widmer
code/data similarity 0.72
Abstract · Code and models: https://github.com/theMoro/music_source_restoration

Music Source Restoration (MSR) seeks to recover original, unprocessed instrument stems from mixed, mastered, and possibly degraded recordings. Unlike conventional source separation, which treats the mixture as a linear sum of clean sources, MSR must additionally invert nonlinear production effects, such as equalization and compression, and transmission-related degradations, such as codec artifacts. We propose a three-stage framework built around this distinction: (1) a mixture restoration model that addresses degradation before separation, (2) a single model that separates the restored mixture into eight target stems (vocals, guitars, keyboards, synthesizers, bass, drums, percussion, and orchestra), and (3) stem-specific restoration experts fine-tuned on the separator's own residual artifacts. Each stage improves restoration quality over the previous one on the MSR Challenge test set. We release code and models to support future research in MSR at https://github.com/theMoro/music_source_restoration.

0.24

Steer Unified Audio Editing Toward Reliable Source Selection

cs.SD ·2026-11-07 ·Youquan Fu, Dongnuan Cai, Yu Pan, Henghui Du, Yujia Zhang +6 more
benchmarkcode/data similarity 0.67
Abstract

Unified audio editors support diverse operations, yet often struggle to modify the intended source while preserving non-target content. We show that source selection in audio editing is shaped by early denoising states, while incorrect early selections can hardly be corrected during subsequent denoising. Motivated by these findings, we propose Source-selection Training via Early-trajectory Emphasis and Recovery (STEER), a two-stage post-training algorithm for reliable selective audio editing. STEER-Emphasis prioritizes training on task-critical early states to reduce source-selection errors, while STEER-Recovery learns corrective denoising from self-generated rollout states to recover from early deviations. To systematically assess source-selection reliability, we introduce the Selective Audio Editing Benchmark, enabling large-scale automatic evaluation of target manipulation and non-target preservation. Applying STEER to our pretrained multitask base editor gives SteerSound, a unified audio editor with leading selective editing performance among open-source audio editors. Experiments on our base editor, UNISON, and SAM Audio demonstrate that STEER consistently improves both selective editing capability and overall editing performance. Early-trajectory emphasis further brings substantial improvements in transcript-conditioned target speaker removal and separation, highlighting the broader applicability of this training principle.

0.24

Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing

cs.SD cs.AI ·2026-09-24 ·Ilpo Viertola, Giulio Cengarle, Gouthaman KV, Daniel Arteaga, Lie Lu
benchmarkdatasetcode/data similarity 0.67
Abstract · Submitted to ICASSP 2027. Project page https://sse-ai.notion.site

We introduce Spot, Separate, and Enhance (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement. SSE enhances video content by rebalancing the audio, removing unwanted audio sources, and reducing reverberation, guided by both video and textual descriptions. To support its training and evaluation, we propose DegradedMix, a new dataset built on the audio remixing benchmark MuddyMix. We also adopt evaluation metrics from generative modeling, which better capture the creative nature of remixing than standard reconstruction-based metrics. SSE outperforms existing baselines in both controllability and remixing quality, as shown by extensive experiments. Project page: https://sse-ai.notion.site

0.41

How Much Audio Is Left In An Embedding? An Inversion Audit Of Audio Encoders

cs.SD cs.AI ·2026-10-08 ·Marios Glytsos, Brian McFee
similarity 0.65
Abstract · ICASSP 2027 submission; 5 pages, 2 figures, 2 tables

Pretrained audio encoders are reused for downstream tasks that are often unknown when the encoder is trained, so their usefulness depends partly on which signal properties survive the pretext objective. We study this retained information through paired source reconstruction. Using a shared Stable Audio Open latent diffusion decoder, we reconstruct five-second, 44.1-kHz stereo music from frozen representations produced by supervised classifiers (VGGish, ConvNeXt), an audio-text contrastive model (CLAP), and a waveform-reconstruction model (EnCodec). These objectives impose different pressures to preserve source detail, while their exposed interfaces vary substantially in temporal and spectral resolution. Evaluating on the Million Song Dataset (MSD), we find clear differences in reconstructability across encoder families, while within encoder comparisons show improved recovery when finer temporal or spectral structure is exposed. Even compressed task oriented embeddings support reconstructions that preserve measurable source specificity and high level musical content.

0.22

End-to-End Historical Music Restoration in Latent Space

eess.AS cs.LGcs.SDeess.SP ·2026-09-30 ·Steven Cho, Junghyun Koo, Raphael Lafargue, Tushar Dhyani, Eloi Moliner +1 more
benchmarkcode/data similarity 0.65
Abstract · 5 pages, 2 figures, 3 tables; submitted to ICASSP 2027. Code and audio demos available at the project repository

Historical music restoration (HMR) has almost exclusively focused on constrained problems such as Super-Resolution or the restoration of solo pieces, under-exploring the general task of restoring orchestral historical music, which has multiple instruments. This under-exploration is largely because the HMR domain, early-20th-century recordings, has no pre-degradation ground-truth pairs, making the restoration task unsupervised and more challenging. This paper presents a supervised end-to-end orchestral HMR benchmark by exploring both the synthetic degradation functions and the end-to-end generative deep-learning restoration methods. We simulate the historical recording degradation chain more faithfully than prior work, which makes orchestral restoration into a tractable supervised problem. A latent flow-matching model trained on the resulting synthetic pairs outperforms existing HMR baselines on intrusive, non-intrusive, and subjective evaluations. We also curate and release a 9.3-hour license-free, unpaired, historical classical-music test set, along with code and audio demos.

0.37

MiDiff: MIDI-Conditioned Diffusion Models for Drum Audio Enhancement

eess.AS eess.SP ·2026-10-28 ·Elad Aspis, Sharon Gannot
similarity 0.64
Abstract · 13 pages. Submitted to IEEE/ACM Transactions on Audio, Speech, and Language Processing. Audio demos and code: https://midiff-paper.github.io…

Bridging the semantic gap between perfect symbolic musical structure and severely degraded acoustic signals offers a powerful paradigm for high-fidelity audio restoration. We introduce MiDiff, a diffusion-based generative model for drum audio enhancement that conditions the reverse process on temporally aligned Musical Instrument Digital Interface (MIDI) piano rolls. We extend the Score-Based Generative Models for Speech Enhancement (SGMSE) framework in the complex Short-Time Fourier Transform (STFT) domain by integrating a pretrained MIDI encoder, multi-scale temporal alignment, and three complementary conditioning strategies: early injection, Feature-wise Linear Modulation (FiLM), and cross-attention. We further show that providing the same piano roll only through classifier guidance, driven by an Automatic Drum Transcription (ADT) model on the audio-only score, yields no measurable gain: the symbolic prior is effective when it conditions the score network. To relax the requirement for ground-truth MIDI at inference, we further propose a hybrid reverse schedule that transcribes the diffusion state online with the same noise-aware ADT network and hands over from the audio-only score to MiDiff at a tunable switch time, cutting test FAD by 47% relative to the audio-only baseline without any symbolic input (switch time selected on the test set). We train our model on the StemGMD dataset using a non-additive degradation pipeline that simulates room reverberation with echo, electret capsule frequency response, and soft-clipping saturation with noise. MiDiff substantially outperforms the SGMSE baseline in terms of Fréchet Audio Distance (FAD) on both the validation and held-out test sets. Ablation studies demonstrate that combining all three conditioning strategies yields superior performance and that authentic MIDI velocity dynamics are critical for effective symbolic conditioning.

0.24

AudioCodex: Streaming Autoencoders for Live Audio

cs.SD cs.LGeess.AS ·2026-11-09 ·Marco Pasini, Javier Nistal, Cyran Aouameur, Stefan Lattner, George Fazekas
published similarity 0.64
Abstract · Accepted to ISMIR 2026

We introduce AudioCodex, a fully causal autoencoder for live generative audio applications that compresses stereo audio into compact continuous latent sequences at ~10 Hz. It combines a Mean Flow magnitude autoencoder with latent hypersphere projection, bounded-noise augmentation, representation alignment, and adversarial phase recovery to learn representations that support both efficient downstream generation and high-quality reconstruction. Through extensive ablations, we show that AudioCodex outperforms the state-of-the-art in terms of generation-reconstruction trade-off, whilst remaining strictly causal in both encoding and decoding. This work paves the way towards truly real-time interactive and streaming generative audio applications.

0.17

Rethinking Music Tokenization: A Semantic Codec toward High-Fidelity LLM Music Generation

eess.AS ·2026-09-18 ·Huakang Chen, Guobin Ma, Yuepeng Jiang, Dake Guo, Jingbin Hu +8 more
similarity 0.64
Abstract

Discrete audio tokenization has become the critical interface between raw waveforms and autoregressive modeling in recent music generation. As a result, music tokenizers must simultaneously support high-fidelity reconstruction and produce discrete sequences that remain amenable to language modeling. Existing reconstruction-oriented tokenizers often mix musical structure with fine acoustic details, producing high-entropy tokens that are hard to model. In contrast, semantics-guided alternatives are designed for speech and do not fit music well, often hurting reconstruction quality. We address these trade-offs by rethinking music tokenization around a measurable notion of music semantic content grounded in downstream Music Information Retrieval tasks. Guided by this definition, we propose MuSeC, a music semantic codec that factorizes semantic and acoustic content directly from mixed signals without source separation. MuSeC preserves information required for high-fidelity reconstruction while producing more LM-friendly discrete units. Empirically, it improves reconstruction quality and yields more predictable token sequences, providing a practical foundation toward high-fidelity LLM music generation. Demos are available at https://longwaytog0.github.io/MuSeC/.

0.22