CS Paper Radar

SAGE: Semantic Audio Generative Encoder

Francesco Brigante, Luca Cerovaz, Davide Marincione, Giorgio Strano, Luca Zhou, Emanuele Rodolà, Michele Mancusi

cs.SD cs.LGeess.AS ·submitted 2026-09-26 16:21 UTC · v3
code/data

Abstract

Audio autoencoders compress waveforms into compact latent representations that serve as the interface between raw audio and downstream models. Current systems navigate a three-way trade-off between reconstruction quality, semantic structure of the latent space, and inference speed, typically favoring one or two of these at the expense of the others. This paper introduces SAGE, Semantic Audio Generative Encoder: a compact variational autoencoder, trained solely on publicly available music, that shapes its latent by distilling embeddings from a pretrained audio-text model. This 105M-parameter model runs about 3x faster than Stable Audio Open and reaches the listening-test quality of SAME-L, an autoencoder 8x larger and over 2x slower, while surpassing both on objective perceptual and distributional metrics of reconstruction. Furthermore, it sets the state of the art on all nineteen probing tasks of latent semantics, in domain and out of domain. These results establish SAGE as a lightweight audio autoencoder that strikes the best balance of the three-way trade-off among those we evaluate, combining high reconstruction fidelity, state-of-the-art semantic structure, and fast inference.

Comment
18 pages, 6 figures, 11 tables. Code and weights: https://github.com/francescobrigante/SAGE. Project page: https://sage-music.pages.dev/ v3: inference times re-measured on one NVIDIA RTX PRO 6000 Blackwell GPU in TF32, with a revised protocol and all numerical precisions in Appendix C; speed claims updated accordingly. Training, reconstruction, listening-test and probing results are unchanged

Closest papers

Nearest neighbours by meaning across the whole index.

AudioGAR: Bridging Reconstruction and Generation in Latent Audio Generative Models

cs.SD ·2026-10-05 ·Xianghong Fang, Geeyang Tay, Wentao Ma, Tim G. J. Rudner, Dehan Kong
similarity 0.76
Abstract · 16 pages, 9 figures and 5 tables

Latent audio generative models are typically trained in two stages: an audio codec is learned first, followed by a latent generative model. This decomposition leads to a decoder train-generation mismatch: the codec decoder is trained on encoder-induced latents but deployed on generator-produced latents at inference time. Across diverse datasets and latent generative models, we observe clear reconstruction-generation gaps under both FD and FAD, showing that strong reconstruction quality does not necessarily translate into strong end-to-end generation quality. A natural remedy is to adapt the decoder on generation-produced latents, but generated latents lack correspondence with source audio and therefore cannot directly provide the paired supervision used for decoder fine-tuning. We introduce \textbf{AudioGAR}, which constructs intermediate latents by perturbing encoder latents and denoising them through the frozen latent diffusion model. These latents form a trajectory from reconstruction toward generation, with lower-noise latents retaining source correspondence and supporting paired decoder fine-tuning. We fine-tune only the codec decoder on these latents, while keeping the codec encoder and latent generative model frozen. When applied to AudioX, AudioGAR substantially improves generative performance. It requires only 1.5\% of the original training audio hours and 0.26\% of the original training cost.

0.23

MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces

eess.AS cs.SD ·2026-10-12 ·Guobin Ma, Yuepeng Jiang, Dake Guo, Huakang Chen, Yuan Zhou +13 more
similarity 0.75
Abstract

Continuous latent representations underlie recent diffusion-based lyrics-to-song and text-to-music systems, yet aligning lyric conditions with these latents remains difficult: existing systems resort to generator-side machinery such as sentence-level timestamps, auxiliary alignment objectives, or semi-autoregressive decoding, while the latents they predict are still produced by autoencoders trained purely for reconstruction. We address this difficulty from the representation side. We propose MSVAE (Music Semantic Variational Autoencoder), which structures its continuous latent around two complementary semantic streams: work-level compositional content and lyric-level linguistic-vocal content. Partitioned dual-stream alignment makes both streams explicit by aligning disjoint latent subspaces with frozen lead-sheet and ASR references, while a semantic-first gated residual posterior derives its base from a frozen music-SSL encoder and admits acoustic information only through gated, bounded corrections. MSVAE achieves competitive stereo reconstruction and stronger MARBLE probing than continuous-autoencoder baselines, and improves downstream lyrics-to-song and text-to-music generation. Notably, with only an untimed phoneme prefix and no lyric timing, MSVAE yields the lowest phoneme error rate, whereas all baseline representations require sentence-level alignment to produce intelligible vocals.

0.06

Exploring Representation Autoencoders for Text-to-Audio Generation

cs.SD eess.ASeess.SP ·2026-10-09 ·Bernardo V. Miranda, Hugo Malard, Gaël Richard, Mathieu Fontaine, Slim Essid
similarity 0.74
Abstract

Representation autoencoders (RAE), created by pairing a frozen pretrained self-supervised learning (SSL) encoder with a trained decoder, have recently demonstrated state-of-the-art performance on image generation. Their use in the audio domain, however, remains limited: text-to-audio (TTA) models are typically trained in the latent space of dedicated autoencoders where the encoder and the decoder are trained jointly with reconstruction objectives. This paper presents a study on audio representation autoencoders (AuRA) and their applicability to general audio TTA. We train decoders for two pretrained SSL audio encoders and evaluate their reconstruction quality against autoencoders used by recent TTA systems. After obtaining decoders for the SSL representations, we also train rectified flow models in the SSL and baseline latent spaces to isolate the autoencoder's effect on generation. We find that generation in SSL spaces achieves comparable or superior performance to generation in the baseline spaces, despite differences in reconstruction performance. These results highlight that reconstruction quality alone is not a sufficient proxy for generative performance.

0.14

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation

eess.AS cs.SD ·2026-09-23 ·Runwu Shi, Kai Li, Yujin Wang, Dong Yang, Jiahui Li +6 more
similarity 0.74
Abstract

Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. However, reconstruction-oriented representations may be suboptimal for generation, motivating joint representation and generative learning. To this end, we introduce Unite-Audio, to our knowledge, is the first to jointly learn continuous audio representations and latent flow matching for TTA. By coupling reconstruction with self-supervised generative prediction, Unite-Audio allows the generative objective to directly shape the latent space rather than treating it as a fixed intermediate representation. We further employ Flow-GRPO post-training to improve text-conditioned generation. Experiments show competitive TTA performance with a compact latent flow model, while ablation studies confirm the benefit of jointly learning the audio representation and generative model. Audio samples are available at https://runwushi.github.io/Unite-Audio.

0.22

AudioCodex: Streaming Autoencoders for Live Audio

cs.SD cs.LGeess.AS ·2026-11-09 ·Marco Pasini, Javier Nistal, Cyran Aouameur, Stefan Lattner, George Fazekas
published similarity 0.74
Abstract · Accepted to ISMIR 2026

We introduce AudioCodex, a fully causal autoencoder for live generative audio applications that compresses stereo audio into compact continuous latent sequences at ~10 Hz. It combines a Mean Flow magnitude autoencoder with latent hypersphere projection, bounded-noise augmentation, representation alignment, and adversarial phase recovery to learn representations that support both efficient downstream generation and high-quality reconstruction. Through extensive ablations, we show that AudioCodex outperforms the state-of-the-art in terms of generation-reconstruction trade-off, whilst remaining strictly causal in both encoding and decoding. This work paves the way towards truly real-time interactive and streaming generative audio applications.

0.17

How Much Audio Is Left In An Embedding? An Inversion Audit Of Audio Encoders

cs.SD cs.AI ·2026-10-08 ·Marios Glytsos, Brian McFee
similarity 0.74
Abstract · ICASSP 2027 submission; 5 pages, 2 figures, 2 tables

Pretrained audio encoders are reused for downstream tasks that are often unknown when the encoder is trained, so their usefulness depends partly on which signal properties survive the pretext objective. We study this retained information through paired source reconstruction. Using a shared Stable Audio Open latent diffusion decoder, we reconstruct five-second, 44.1-kHz stereo music from frozen representations produced by supervised classifiers (VGGish, ConvNeXt), an audio-text contrastive model (CLAP), and a waveform-reconstruction model (EnCodec). These objectives impose different pressures to preserve source detail, while their exposed interfaces vary substantially in temporal and spectral resolution. Evaluating on the Million Song Dataset (MSD), we find clear differences in reconstructability across encoder families, while within encoder comparisons show improved recovery when finer temporal or spectral structure is exposed. Even compressed task oriented embeddings support reconstructions that preserve measurable source specificity and high level musical content.

0.22

Open-Qwen-Music: An Auditable Framework for LLM-Based Music Composition and Diffusion Rendering

cs.SD cs.AIeess.AS ·2026-09-14 ·Yangbin Yu, Mingyu Yang
code/data similarity 0.72
Abstract

We present Open-Qwen-Music, an open reconstruction of Qwen-Music and a fully specified research system for text-to-music generation that couples LLM-based semantic composition with diffusion-based acoustic rendering. The system comprises a 25 Hz single-codebook music tokenizer, a 3B-parameter autoregressive Music LLM, and a diffusion renderer producing 48 kHz stereo audio, following the cross-module interfaces reported by Qwen-Music. The strongest systems of this design remain closed, and prominent open music-generation projects release weights and inference code without their training corpora or end-to-end training implementations. This limits independent and controlled study of how information loss and prediction errors propagate from semantic representation through autoregressive planning to acoustic rendering. To our knowledge, Open-Qwen-Music is the first fully open release of an LLM-composition-plus-diffusion-rendering text-to-music system. Beyond model weights and inference code, the release includes the training datasets and provenance manifests, complete data-processing, annotation, training, inference, and evaluation pipelines, configurations, and pretrained weights for every learned module. Artifact manifests bind the identities of these artifacts across the complete workflow. Together, these artifacts establish a reproducible implementation of the modular architecture and provide an empirical basis for component-level analysis and future evaluation. We present the system as a transparent, executable research baseline and a starting point for the community, not as evidence of quality parity with Qwen-Music. Open-Qwen-Music is an ongoing effort, and we will continue to improve its generation quality, controllability, and robustness. All release artifacts are available at https://github.com/biang15343100-source/Open-Qwen-Music.

0.23

StreamSE: Neural Audio Codec-based Streaming Speech Enhancement for On-device Inference

eess.AS ·2026-10-27 ·Hyunjin Choi, Zinu Kim, Junwon Moon, Jeongyun Chae, Seungbeom Kim +1 more
published similarity 0.72
Abstract · NeurIPS 2026 Workshop on On-Device Intelligence

Recent generative speech enhancement (SE) models achieve high perceptual quality, but their non-causal processing and expensive computation make them difficult to use for streaming, on-device inference. We propose StreamSE, a 20M-parameter streaming SE model that operates in the latent space of a frozen pretrained neural audio codec. At each frame, StreamSE directly regresses a clean continuous latent from the noisy continuous latent while conditioning on its own predictions from previous frames. Enhancement therefore requires a single forward pass per frame and no iterative sampling or look-ahead. We further truncate the codec's residual vector quantizer (RVQ) to its coarse levels, halving the number of tokens per frame while improving robustness to reverberation. StreamSE is the only streaming generative model in our comparison, yet it matches or exceeds the perceptual quality of the offline generative baselines. Furthermore, the complete StreamSE pipeline runs in real time on a single smartphone CPU core. Our results show that continuous latent regression provides a practical route to generative SE under fully causal, on-device streaming constraints.

0.14