CS Paper Radar

Convolutional VocalTrax: Replacing Backpropagation Through Time with Convolution in Differentiable Articulatory Synthesis

Yongyi Zang, Debasish Ray Mohapatra, Randal Leistikow, Perry R. Cook

cs.SD ·submitted 2026-11-12 17:49 UTC · v1

Abstract

VocalTrax recovers interpretable articulatory controls from a recording by gradient descent through a differentiable Pink Trombone synthesizer. Its vocal tract is a Kelly-Lochbaum waveguide updated twice per audio sample, so every gradient step backpropagates through a long recurrence, which is slow on GPUs. We propose Convolutional VocalTrax, which computes the exact frequency response of the same waveguide and applies it as a causal convolution. We assume that the tract is stationary within 5.8 ms blocks and that its lossy impulse response can be truncated after 93 ms. Gradients then pass through FFTs and a short recursion along the tract instead of through time. On 600 AudioMNIST recordings, the tract forward and backward pass runs 47 to 63 times faster on an A100 GPU, and a classifier trained only on real speech recognizes our reconstructions more often than those of VocalTrax (14.5% vs. 10.0%), although the log-mel distance to the target is 11% higher. Parameters fitted with our model keep this gain when rendered by the original VocalTrax (13.8%).

Comment
9 pages, 3 figures, 5 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Articulatory Source-Filter TTS: Physically Grounded Control through Vocal Tract Kinematics

eess.AS ·2026-09-30 ·Jesuraj Bandekar, Shinji Watanabe, Prasanta Kumar Ghosh
similarity 0.63
Abstract

Modern neural text-to-speech (TTS) systems achieve remarkable acoustic fidelity but act as black boxes, offering little interpretable control over the vocal tract filter. We propose a controllable source-filter TTS architecture grounded in articulatory kinematics. An Acoustic-to-Articulatory Inversion (AAI) model, enhanced by large-scale pretrained representations, generates kinematic pseudo-trajectories for a large TTS corpus. These trajectories condition the filter response, while predicted pitch and energy contours parameterise the glottal source. Source and filter are predicted independently, and the source is refined by an Optimal Transport Conditional Flow Matching (OT-CFM) module before recombination into the final spectrogram. Our model achieves intelligibility and naturalness competitive with similarly sized baselines, with only a modest spectral fidelity cost in exchange for explicit control. Evaluations reveal clear source-filter disentanglement, enabling stable prosodic scaling and cross-speaker source/filter recombination, where F0 remains tied to the source speaker while vocal-tract characteristics follow the filter speaker. Finally, direct manipulation of articulatory trajectories enables fine-grained control, such as accent modification, offering a new direction for interpretable speech synthesis. Audio samples: https://coding-phoenix-12.github.io/ArticulatorySFTTS/

0.26

Reconstructing the Vocal Tract with Differentiable Acoustic Simulation

cs.SD cs.CLeess.AS ·2026-09-29 ·Eric Ming Chen, Jin Woo Lee, Vincent Sitzmann
top venue similarity 0.63
Abstract · Accepted as NeurIPS 2026 spotlight paper. Supplementary material at https://people.csail.mit.edu/echen/vocal_recon/

The vocal tract is the region of the human body responsible for filtering one's voice to create speech. In this paper, we present a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it is readily integrated with other deep learning pipelines to enable novel linguistics and medical imaging applications. (1) We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and (2) we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's moving vocal tract from only their speech without paired data.

0.50

TAPE: Learning General Audio Effect Representations

cs.SD cs.AIcs.LGeess.AS ·2026-10-29 ·Phillip Long, Yongyi Zang, Qihui Yang, Christodoulos Benetatos, Junfei Liu +3 more
similarity 0.62
Abstract · Submitted to Transactions of Machine Learning Research; 25 pages, 5 figures, 18 tables

Musicians and mixing engineers often can decipher the signal processing chain they hear on a reference recording, then adapt it to new material. We present TAPE (Transformations of Audio Parameter Embeddings), which estimates this processing from the processed reference audio alone. An encoder trained to identify processors in eight major effect families maps the reference to an embedding. A differentiable digital signal processing (DSP) chain decodes the embedding into editable parameters in one forward pass and applies them to a different unprocessed recording. Its modules are equalization, compression, waveshaping, a small residual network, and reverberation. TAPE achieves lower spectral error than DeepAFx-ST, the previous state of the art for single-step regression, and lower spectral error than per-example search over a chain approximating ST-ITO's, while running 275$\times$ faster than that search. Because the decoder outputs DSP parameters, its chain can also run causally: a C++ implementation with the trained weights processed 128-sample buffers 1.8$\times$ faster than real time, with no missed deadlines.

0.22

Unsupervised Instantaneous Phase and Frequency Tracking by Inverse Voice Synthesis

eess.AS cs.SD ·2026-10-02 ·Chin-Yun Yu, György Fazekas
similarity 0.61
Abstract · Submitted to ICASSP 2027

Knowledge-driven neural vocoders struggle to learn reliable fundamental frequency end-to-end, because spectral objectives provide weak supervision of periodic structure and lack phase information. We address this with a source-filter model whose alias-free additive source makes the instantaneous phase of the glottal cycle explicit; differentiating it yields the instantaneous frequency, and thus $F_0$, without an external tracker. Waveform error supervises only the deterministic harmonic path, while a spectral loss covers the full signal. On M4Singer and LM-SSD, the reconstruction is phase-aligned, reaching a signal-to-reconstruction-error ratio of 8.1 dB, while neural baselines remain negative. However, GOLF, given an external $F_0$, still reaches lower spectral distortion. On LM-SSD, the recovered $F_0$ attains the highest overall accuracy of any method tested, including supervised neural pitch trackers applied off the shelf, and the glottal closure instants come within 0.53 points of REAPER's identification rate, without any $F_0$ label.

0.51

Knowledge Distillation for Efficient Acoustic Echo Control

eess.AS ·2026-08-26 ·Ernst Seidel, Pejman Mowlaee, Tim Fingscheidt
published similarity 0.60
Abstract · 5 pages, accepted to EUSIPCO 2026

In recent years, many efforts have been made to supersede classical acoustic echo control (AEC) algorithms with more powerful machine-learned approaches. While surpassing the performance of well-established adaptive filters is very much possible, a remaining challenge is computational complexity. Popular architectures, such as convolutional recurrent networks (CRNs), are by multiple orders of magnitude computationally more expensive than classical signal processing solutions. Scaling down such models is usually straight-forward, but it comes at the cost of a notably reduced performance. We show - to the author's knowledge for the first time in AEC - how these performance drops can be successfully alleviated to a large degree by employing an effective knowledge distillation (KD) process, enabling more potent efficient AEC. Our proposed CGGN16 student AEC models show significantly less near-end speech distortion at only 2% of its teacher's computational complexity, surpass the overall performance of a six times more complex model trained on ground-truth labels, and outperform other AEC-focused architectures from recent literature.

0.22

Validating Acoustic-to-Articulatory Inversion with Real-Time MRI: From Vocal-Tract Fidelity to Phonetic Consequences

eess.AS ·2026-11-16 ·Tharinda Piyadasa, Michael Proctor, Amelia Gully, Tuende Szalay, Emily Kiff +2 more
similarity 0.60
Abstract · 5 pages, 3 figures. Submitted to ICASSP 2027

Acoustic-to-articulatory inversion provides a promising way to estimate vocal-tract behaviour directly from speech, but inferred articulatory variables need to be validated against independent physical measurements before they can be interpreted as faithful estimates of articulation. We investigate this using paired real-time MRI (rtMRI) and speech. A WavLM-based inversion model is trained to predict eight vocal tract variables (TVs) extracted from rtMRI, achieving a mean correlation of 0.643 on held-out speakers. Prediction accuracy is substantially higher for constriction degree than for constriction location, and the inferred trajectories systematically underestimate the dynamic range of rtMRI-derived articulation. We then use articulatory-to-acoustic synthesis as a functional probe of these errors, comparing speech generated from inversion-derived and rtMRI-derived TVs while holding the source features fixed. The discrepancy is small for vowels, approximants, stops, and nasals, but increases markedly for fricatives and affricates, where speech production depends on maintaining a narrow critical constriction and small geometric errors can strongly affect the acoustics. These results show that current acoustic-to-articulatory models recover broad vocal-tract behaviour well, but further work is required before inferred TVs can be regarded as faithful measurements of fine articulatory geometry.

0.19

Lyric: Wave-Domain Computing for Efficient Spoken-Digit Recognition

eess.AS ·2026-10-05 ·Jeeven Balasubramaniam, Nakul Garg
similarity 0.59
Abstract · 5 pages, 5 figures, 2 tables, Submitted to IEEE ICASSP 2027

Low-power speech recognition requires reducing audio acquisition and feature-processing costs as well as neural inference. We present Lyric, a speech-recognition front end that uses wave-domain computing to extract features before digitization. Our proposed system uses passive acoustic resonators to separate speech by frequency and supplies their slowly varying envelopes to a compact temporal neural network. This moves spectral filtering into the acoustic structure, reducing the data and processing needed for recognition. We build a prototype and evaluate spoken-digit recognition on a speakerdisjoint AudioMNIST split across nine classifier families. The front end acquires 32 times fewer scalar samples than a 16-kHz waveform. In the EdgeSpeechNet-A comparison on a Raspberry Pi 4, we reduce preprocessing-plus-inference latency and estimated processor energy per inference by 98.6%, with a 3.58-percentage-point decrease in accuracy to 95.70%.

0.18

Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training

eess.AS cs.LGcs.SD ·2026-10-09 ·Quentin Le Tellier, Albert Rilliard, Olivier Perrotin, Marc Evrard
code/data similarity 0.59
Abstract · 5 pages, 2 figures, 1 table. Submitted to IEEE ICASSP 2027. Code: https://github.com/vers-project/vocal-intensity-conversion. This work has …

Vocal intensity conversion renders a recorded utterance at a different vocal intensity, rather than simply scaling its amplitude, while preserving its content and speaker identity. Existing systems control intensity through a reference utterance or a few discrete categories. Instead, we control the conversion with any target sound pressure level (SPL) on a continuous dB scale. A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder resynthesizes the speech. We compare three representations, log-mel spectrograms, WavLM, and SpeechTokenizer, and introduce an objective protocol that measures the achieved intensity change against the target one on a calibrated corpus. With WavLM, the converter achieves a measured 83% of the target change in intensity and also shifts the fundamental frequency. A perceptual evaluation confirms that perceived vocal intensity increases with the target level on amplitude-normalized stimuli.

0.30