CS Paper Radar

Beyond Dry References: Learning Relative Audio Effects Representations via Contrastive Distance Learning

Xinlu Liu, Huibin Lin, Weixing Wei, Zhenhai Yan

cs.SD ·submitted 2026-08-11 07:00 UTC · v1
publishedpositioncode/data

Abstract

Audio effects (Fx) representation learning plays a key role in intelligent music production, including automatic mixing and Fx style transfer. Existing methods typically rely on dry or nearly dry references for effect modeling, yet truly unprocessed audio is rarely available in practice, as real recordings inevitably reflect the microphone, room acoustics, and preceding signal processing. Instead of pursuing absolute effect encodings, we argue that the relative effect distance between audio signals is more meaningful for real-world music production. Motivated by this, we propose RelFx, a contrastive learning framework that learns relative effect transformations from general audio collections without requiring dry references during representation training. Our approach uses a dual-branch Siamese encoder equipped with cross-attention and differential gating fusion to infer the shared effect transformation from a reference clip and an effect-processed, content-related clip. We further propose an antisymmetric fusion variant for bidirectional effect encoding, such that swapping the input order directly produces a nearly sign-reversed embedding, a property not explored in earlier work. Moreover, our dry-reference-free formulation eliminates the reliance on dry multitrack datasets and enables training on effect-bearing audio. Experiments on Fx style transfer demonstrate state-of-the-art performance under the standard Fx-Encoder++ MUSDB18 evaluation protocol, consistently outperforming existing approaches across all four instrument categories.

Comment
8 pages (6 pages of main text), 3 figures, 3 tables. Accepted at the 27th International Society for Music Information Retrieval Conference (ISMIR 2026). Project page: https://relative-fx.github.io

Closest papers

Nearest neighbours by meaning across the whole index.

Exploring the Design Space of Representation Learning for Audio Transformations

cs.SD eess.AS ·2026-08-28 ·Sungho Lee, Marco Martínez-Ramírez, Junghyun Koo, Wei-Hsiang Liao, Kyogu Lee +1 more
published similarity 0.72
Abstract · Accepted to ISMIR 2026

Neural audio representation learning has enabled a range of content-oriented applications, but the resulting features remain limited for tasks involving audio processing. Furthermore, it is not obvious what processing-aware representations should capture: the processing itself, abstracted away from source content, or the processed audio that retains it. Existing approaches implicitly commit to one or the other and also differ in their models, data, and evaluation, obscuring which design choices drive their behavior. We address both questions within a unified framework of three objectives: processing consistency, description alignment, and equivariance via forward prediction. We compare all combinations of the objectives under a controlled setup and reveal their relative strengths and interactions. Our framework produces both a transformation embedding and a processed-audio embedding, and we find that the two play complementary roles: distance-based tasks favor the former, while probe-based tasks favor the latter. Combined with improvements in network architecture and training pipeline, our representations outperform prior baselines across retrieval, probe-based evaluation, and style transfer.

0.44

Exploring Post-Hoc Disentanglement of Audio Foundation Models for Music Similarity

cs.SD cs.LG ·2026-11-10 ·Lucas de Souza Brandão, Gabriel da Costa Merlin, Diego Furtado Silva
similarity 0.69
Abstract · 5 pages

Audio foundation models like CLAP, MuQ, and MERT excel at capturing general musical similarity. On the other hand, their entangled latent spaces limit interpretability and fine-grained attribute control. Conditional Similarity Networks (CSN) address this by pairing a backbone representation model with learned masks that gate the embedding space into attribute-specific subspaces. However, existing methods train the backbone from scratch, rather than leveraging pretrained foundation models. We instead present a systematic study of post-hoc disentanglement, adapting CSN to gate frozen pretrained embeddings. We evaluate adding trainable parameters to adapt the embedding under different masking and pooling regimes across two datasets. Our findings show that minimal adaptation capacity is sufficient for post-hoc gating to improve both attribute-specific disentanglement and overall retrieval performance, while pooling, masking, and depth choices yield limited additional gains.

0.20

X-AVDD: Cross-Attentive Audio-Visual Dataset Distillation

cs.CV cs.LGcs.SD ·2026-10-09 ·Saumyaranjan Mohanty, Aravind Reddy, Konda Reddy Mopuri
top venue similarity 0.67
Abstract · Accepted at NeurIPS 2026. Camera-ready version forthcoming. 17 pages, 6 figures, 16 tables

Audio-visual dataset distillation aims to synthesize a small dataset from a large paired audio-visual dataset, while nearly preserving the training utility for neural networks. Existing approaches such as DM and AVDD face two critical limitations: 1) they are memory-intensive, making them difficult to scale to complex architectures and higher samples-per-class (SPC) settings, and 2) they often fail to preserve strong cross-modal semantic correspondence. To tackle these issues, we propose X-AVDD, a cross-attention-based distillation method that explicitly captures inter-modal relationships during synthesis. By decoupling cross-modal alignment from joint optimization, X-AVDD is substantially more memory-efficient, reducing end-to-end synthesis cost by $\sim 8\times$ GPU-hours, while producing highly aligned synthetic data. Empirically, X-AVDD significantly outperforms other audio-visual distillation methods. On VGGSound, an audio-visual dataset of short YouTube clips, X-AVDD improves top-1 test accuracy from $8.2\%\rightarrow24.0\%$ at SPC=$10$ and $9.8\%\rightarrow26.5\%$ at SPC=$20$. Furthermore, X-AVDD improves cross-architecture generalization (e.g., $25.5\%\rightarrow35.3\%$ for a ViT target on AVE dataset at SPC $=50$) and successfully scales to complex architectures such as AudioCLIP, achieving $62.3\%$ accuracy using only $5.65\%$ of the full dataset, against the full dataset accuracy of $72.4\%$ while completing in $9\times$ lower training time compared to the full dataset.

0.40

SAIL: Spatial Audio Intelligence with Large Language Models via Disentangled Acoustic-Spatial Encoding and Dual-Stream Q-Former

cs.SD cs.AIeess.AS ·2026-09-28 ·Zhengding Luo, Jinyang Wu, Haozhe Ma, Yanghao Zhou, Woon-Seng Gan +1 more
similarity 0.67
Abstract

Spatial audio large language models (LLMs) enable embodied agents, wearable assistants, and immersive systems to recognize sound events, localize sources, and reason about their spatial relationships. However, existing spatial audio LLMs often rely on early fusion of acoustic and spatial features and source-agnostic token representations. These designs make it difficult to preserve the correspondence between individual sound events and their spatial attributes, particularly in multi-source scenes. To address this limitation, we propose SAIL, a Spatial Audio Intelligence framework with LLMs that preserves acoustic-spatial structure and source-level correspondence from audio encoding to LLM alignment. SAIL introduces a Disentangled Spatial Audio Transformer that represents Mel-spectrogram and interaural phase difference features as separate acoustic and spatial streams. Source-discriminative task queries further learn event, direction, and distance information for each source. A Dual-Stream Q-Former then aligns the two streams with the LLM using acoustic and spatial queries organized by source slots. Compared with the early-fusion baseline, SAIL achieves consistent improvements in dual-source sound event detection, direction and distance estimation, and spatial reasoning. These results demonstrate the importance of structured, source-discriminative audio representations for multi-source spatial understanding and reasoning.

0.45

Rel2Traj: Inter-Source Relation-Aware Trajectory Generation for Text-To-Spatial-Audio

cs.SD eess.AS ·2026-10-31 ·Yihang Bao, Lingfeng Yao, Shengyao Wang, Junpei Xun, Miao Pan +1 more
dataset similarity 0.66
Abstract · 5 pages, 1 figure, 3 tables. Submitted to ICASSP 2027

"A car drives from front left to front right while a dog follows it." Such a description specifies no particular trajectory, yet imposes a constraint on the joint motion of both sources that must hold at every frame. Existing text-to-spatial-audio methods can position and move each sound source according to its description, but none of them ensures or verifies that such inter-source relations hold. We therefore propose Rel2Traj, which parses the description into sources, relations, and anchors, jointly generates the trajectories of all sources using a relation-conditioned diffusion model, and finally renders sounds from a pretrained text-to-audio model along these trajectories to produce binaural audio. For training and evaluation, we construct a synthetic dataset from a controlled language of source motions and inter-source relations. A single frame-wise relational geometry defines both our evaluation metric, the Relation Satisfaction Rate (RSR), and an auxiliary training loss. On motion-relation combinations held out from training, Rel2Traj maintains all relation types, whereas every baseline fails on at least one. A listening test further confirms that listeners recognize the described relations. To our knowledge, this is the first text-to-spatial-audio work to model relations among moving sources as explicit geometric constraints.

0.10

Do Audio Representations Compose Additively?

cs.SD cs.MM ·2026-09-23 ·Chenhao Xue, Zhijin Guo, Joyraj Chakraborty, Martin Reed, Nikolaos Thomos
similarity 0.66
Abstract · Submitted to IEEE Signal Processing Letter

Compositionality, the ability to represent complex acoustic scenes as combinations of simpler sound sources, is central to auditory perception and classical additive signal models. Still, it remains unclear whether modern pre-trained audio representations internalize additive structure without compositional supervision. Existing evaluation frameworks of audio compositional reasoning largely focus on cross-modal audio-text alignment, leaving open whether audio representations themselves exhibit additive compositional structure independent of text grounding, analogous to vector arithmetic in word representations. To investigate this, we adopt a two-step diagnostic for frozen audio representations. First, we quantify linear alignment between representations and sound source labels using canonical correlation analysis. Second, we test additive compositional generalization via leave-one-combination-out reconstruction, grouping clips by exact source-label set, averaging their representations, and predicting held-out means from per-source contributions fitted only on training combinations. With larger combination holdouts, CLAP outperforms the permuted and label-overlap baselines on FSD50K, while the speech models do not outperform the label-overlap baseline. We examine representations generated by Wav2Vec2, HuBERT, and CLAP on FSD50K and CHiME-Home datasets. All three models show consistently higher linear correlation and more accurate leave-one-combination-out reconstructions than the permuted baselines. However, only CLAP shows large cosine similarity gains, which could be associated with its training on many kinds of audio and text. Finally, we note that all three models exhibit reconstruction residuals, revealing limits of additive compositionality such as nonlinear or non-compositional audio structure.

0.21

How Much Audio Is Left In An Embedding? An Inversion Audit Of Audio Encoders

cs.SD cs.AI ·2026-10-08 ·Marios Glytsos, Brian McFee
similarity 0.66
Abstract · ICASSP 2027 submission; 5 pages, 2 figures, 2 tables

Pretrained audio encoders are reused for downstream tasks that are often unknown when the encoder is trained, so their usefulness depends partly on which signal properties survive the pretext objective. We study this retained information through paired source reconstruction. Using a shared Stable Audio Open latent diffusion decoder, we reconstruct five-second, 44.1-kHz stereo music from frozen representations produced by supervised classifiers (VGGish, ConvNeXt), an audio-text contrastive model (CLAP), and a waveform-reconstruction model (EnCodec). These objectives impose different pressures to preserve source detail, while their exposed interfaces vary substantially in temporal and spectral resolution. Evaluating on the Million Song Dataset (MSD), we find clear differences in reconstructability across encoder families, while within encoder comparisons show improved recovery when finer temporal or spectral structure is exposed. Even compressed task oriented embeddings support reconstructions that preserve measurable source specificity and high level musical content.

0.22

Semantic Refinement of Universal Audio Representations through Audio-Description Alignment

eess.AS cs.SD ·2026-09-08 ·Lejun Min, Junyu Dai, Ruichen Zheng, Xinyue Fan, Yang Xiang +5 more
hot direction similarity 0.66
Abstract

Universal audio representations must preserve acoustic detail while making high-level concepts accessible across speech, music, environmental sound, and downstream models of different capacities. We study semantic refinement of an acoustically pretrained encoder by adding audio-description alignment to a foundation of BEST-RQ, reconstruction, and CTC. We compare matched control, shuffled-description, and correctly paired trajectories to distinguish correct correspondence from an extra contrastive objective. Each endpoint is frozen and evaluated with a temporal-mean linear probe and a sequence-aware LLM readout, testing whether the refined information is directly accessible and remains useful to a stronger model. Across three paired seeds, correct alignment improves domain-balanced classification by 4.66 points with the linear probe and 2.59 points with the sequence-aware LLM, with positive changes in every domain. Correct pairing accounts for 87% of the linear-probe gain, while the LLM shows its clearest correspondence-specific benefit in captioning. Dense acoustic objectives provide complementary gains under both readouts. A separate 24-layer continuation remains competitive with leading public encoders under the shared evaluator, supporting the recipe beyond the controlled study.

0.47