CS Paper Radar

Exploring Representation Autoencoders for Text-to-Audio Generation

Bernardo V. Miranda, Hugo Malard, Gaël Richard, Mathieu Fontaine, Slim Essid

cs.SD eess.ASeess.SP ·submitted 2026-10-09 12:42 UTC · v1

Abstract

Representation autoencoders (RAE), created by pairing a frozen pretrained self-supervised learning (SSL) encoder with a trained decoder, have recently demonstrated state-of-the-art performance on image generation. Their use in the audio domain, however, remains limited: text-to-audio (TTA) models are typically trained in the latent space of dedicated autoencoders where the encoder and the decoder are trained jointly with reconstruction objectives. This paper presents a study on audio representation autoencoders (AuRA) and their applicability to general audio TTA. We train decoders for two pretrained SSL audio encoders and evaluate their reconstruction quality against autoencoders used by recent TTA systems. After obtaining decoders for the SSL representations, we also train rectified flow models in the SSL and baseline latent spaces to isolate the autoencoder's effect on generation. We find that generation in SSL spaces achieves comparable or superior performance to generation in the baseline spaces, despite differences in reconstruction performance. These results highlight that reconstruction quality alone is not a sufficient proxy for generative performance.

Closest papers

Nearest neighbours by meaning across the whole index.

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation

eess.AS cs.SD ·2026-09-23 ·Runwu Shi, Kai Li, Yujin Wang, Dong Yang, Jiahui Li +6 more
similarity 0.78
Abstract

Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. However, reconstruction-oriented representations may be suboptimal for generation, motivating joint representation and generative learning. To this end, we introduce Unite-Audio, to our knowledge, is the first to jointly learn continuous audio representations and latent flow matching for TTA. By coupling reconstruction with self-supervised generative prediction, Unite-Audio allows the generative objective to directly shape the latent space rather than treating it as a fixed intermediate representation. We further employ Flow-GRPO post-training to improve text-conditioned generation. Experiments show competitive TTA performance with a compact latent flow model, while ablation studies confirm the benefit of jointly learning the audio representation and generative model. Audio samples are available at https://runwushi.github.io/Unite-Audio.

0.22

AudioGAR: Bridging Reconstruction and Generation in Latent Audio Generative Models

cs.SD ·2026-10-05 ·Xianghong Fang, Geeyang Tay, Wentao Ma, Tim G. J. Rudner, Dehan Kong
similarity 0.75
Abstract · 16 pages, 9 figures and 5 tables

Latent audio generative models are typically trained in two stages: an audio codec is learned first, followed by a latent generative model. This decomposition leads to a decoder train-generation mismatch: the codec decoder is trained on encoder-induced latents but deployed on generator-produced latents at inference time. Across diverse datasets and latent generative models, we observe clear reconstruction-generation gaps under both FD and FAD, showing that strong reconstruction quality does not necessarily translate into strong end-to-end generation quality. A natural remedy is to adapt the decoder on generation-produced latents, but generated latents lack correspondence with source audio and therefore cannot directly provide the paired supervision used for decoder fine-tuning. We introduce \textbf{AudioGAR}, which constructs intermediate latents by perturbing encoder latents and denoising them through the frozen latent diffusion model. These latents form a trajectory from reconstruction toward generation, with lower-noise latents retaining source correspondence and supporting paired decoder fine-tuning. We fine-tune only the codec decoder on these latents, while keeping the codec encoder and latent generative model frozen. When applied to AudioX, AudioGAR substantially improves generative performance. It requires only 1.5\% of the original training audio hours and 0.26\% of the original training cost.

0.23

SAGE: Semantic Audio Generative Encoder

cs.SD cs.LGeess.AS ·2026-09-26 ·Francesco Brigante, Luca Cerovaz, Davide Marincione, Giorgio Strano, Luca Zhou +2 more
code/data similarity 0.74
Abstract · 18 pages, 6 figures, 11 tables. Code and weights: https://github.com/francescobrigante/SAGE. Project page: https://sage-music.pages.dev/ v3:…

Audio autoencoders compress waveforms into compact latent representations that serve as the interface between raw audio and downstream models. Current systems navigate a three-way trade-off between reconstruction quality, semantic structure of the latent space, and inference speed, typically favoring one or two of these at the expense of the others. This paper introduces SAGE, Semantic Audio Generative Encoder: a compact variational autoencoder, trained solely on publicly available music, that shapes its latent by distilling embeddings from a pretrained audio-text model. This 105M-parameter model runs about 3x faster than Stable Audio Open and reaches the listening-test quality of SAME-L, an autoencoder 8x larger and over 2x slower, while surpassing both on objective perceptual and distributional metrics of reconstruction. Furthermore, it sets the state of the art on all nineteen probing tasks of latent semantics, in domain and out of domain. These results establish SAGE as a lightweight audio autoencoder that strikes the best balance of the three-way trade-off among those we evaluate, combining high reconstruction fidelity, state-of-the-art semantic structure, and fast inference.

0.31

AudioCodex: Streaming Autoencoders for Live Audio

cs.SD cs.LGeess.AS ·2026-11-09 ·Marco Pasini, Javier Nistal, Cyran Aouameur, Stefan Lattner, George Fazekas
published similarity 0.71
Abstract · Accepted to ISMIR 2026

We introduce AudioCodex, a fully causal autoencoder for live generative audio applications that compresses stereo audio into compact continuous latent sequences at ~10 Hz. It combines a Mean Flow magnitude autoencoder with latent hypersphere projection, bounded-noise augmentation, representation alignment, and adversarial phase recovery to learn representations that support both efficient downstream generation and high-quality reconstruction. Through extensive ablations, we show that AudioCodex outperforms the state-of-the-art in terms of generation-reconstruction trade-off, whilst remaining strictly causal in both encoding and decoding. This work paves the way towards truly real-time interactive and streaming generative audio applications.

0.17

SemiCodec: Bridging Speech Tokenization and High-fidelity Reconstruction with Semi-discrete Representations

eess.AS cs.SD ·2026-10-10 ·Kecan Mao, Nan Xu, Zhaoyang Meng, Zhengyao Ma, Yingming Gao +2 more
similarity 0.69
Abstract · 5 pages, 2 figures, 3 tables. Submitted to ICASSP 2027

Speech representation learning requires a discrete modeling interface while preserving high waveform fidelity. Existing approaches rarely achieve both: discrete tokens ease modeling but lose fine-grained acoustic information, while continuous latents preserve fidelity but require specialized generative objectives. We introduce SemiCodec, a novel Semi-discrete neural speech Codec with synchronized discrete tokens and continuous latents. Direct joint optimization may create a reconstruction shortcut through the continuous pathway, leaving the discrete tokens insufficiently grounded. To address this issue, we propose a Preserve-then-Ground training paradigm that first establishes a high-fidelity synthesis space and then grounds SSL-guided discrete tokens within it. Experiments show that SemiCodec outperforms fully discrete codecs in reconstruction quality and intelligibility while remaining competitive with continuous representations in acoustic fidelity and speaker preservation. Audio samples are available at https://pennyw1-se.github.io/SemiCodec-Demo-Page/

0.29

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

eess.AS cs.AIcs.SD ·2026-08-20 ·Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic
similarity 0.68
Abstract · Project website: https://umbertocappellazzo.github.io/nape

Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.

0.14

Empirical Variational Autoencoder

cs.LG ·2026-10-05 ·Kaede Shiohara
theorycode/datahot direction similarity 0.68
Abstract · Project page: https://mapooon.github.io/EVAPage

We present Empirical Variational Autoencoder, a general generative framework for continuous-valued (i.e., non-vector-quantized) sequences. EVA is based on the evidence lower bound of the Variational Autoencoder (VAE) but learns autoregressive latent priors empirically from training data, which can be implemented only by an additional single linear layer on top of VAEs. By replacing the conventional standard-Gaussian constraint with the self-predicted priors, EVA significantly alleviates the latent distribution gap between prior and posterior which is typically observed in conventional VAEs, and leads to high-fidelity ancestral sampling for sequential data generation. Extensive experiments on image and sound synthesis demonstrate that EVA achieves competitive generation quality with autoregressive diffusion baselines despite its much faster inference time.

0.38

Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization

cs.SD ·2026-08-12 ·Peijie Chen, Zhuanling Zha, Zhipeng Nie, Weijie Wu, Yiming Liu +6 more
theory similarity 0.68
Abstract

In current zero-shot text-to-speech systems, conventional semantic tokenizers are typically optimized using supervised automatic speech recognition or self-supervised learning objectives. However, due to the inherent nature of speech, semantic and acoustic information cannot be completely decoupled, and ASR-based tokenizers discard acoustic details to focus on linguistic content; models relying on them usually struggle to achieve optimal speaker similarity. Furthermore, these tokenizers are optimized independently and lack direct supervision from downstream acoustic generation tasks. This isolated training creates a feature gap between the extracted discrete tokens and the continuous space required by acoustic models, fundamentally bottlenecking the upper bound of synthesis quality. To bridge this gap, we propose Phoenix TTS, a unified framework that tightly couples representation learning with generative acoustic modeling. Specifically, our speech tokenizer is optimized to reconstruct self-supervised features to maintain semantic richness, while simultaneously receiving direct supervision from a Flow Matching training loss. Through this joint training paradigm, the extracted discrete tokens successfully preserve essential semantic information and natively align with the feature space of the downstream Flow Matching model. Comprehensive evaluations highlight the efficiency and effectiveness of Phoenix TTS. Trained on 110K hours of data, the system achieves excellent speech intelligibility, yielding WER that consistently falls below that of ground-truth recordings. Simultaneously, it maintains robust zero-shot speaker similarity, rivaling or outperforming several prominent large-scale baselines. Furthermore, as an advantageous byproduct of this unified training, the learned tokenizer can be seamlessly adapted to zero-shot voice conversion tasks without requiring task-specific fine-tuning.

0.04