CS Paper Radar

Unsupervised Instantaneous Phase and Frequency Tracking by Inverse Voice Synthesis

Chin-Yun Yu, György Fazekas

eess.AS cs.SD ·submitted 2026-10-02 09:42 UTC · v1

Abstract

Knowledge-driven neural vocoders struggle to learn reliable fundamental frequency end-to-end, because spectral objectives provide weak supervision of periodic structure and lack phase information. We address this with a source-filter model whose alias-free additive source makes the instantaneous phase of the glottal cycle explicit; differentiating it yields the instantaneous frequency, and thus $F_0$, without an external tracker. Waveform error supervises only the deterministic harmonic path, while a spectral loss covers the full signal. On M4Singer and LM-SSD, the reconstruction is phase-aligned, reaching a signal-to-reconstruction-error ratio of 8.1 dB, while neural baselines remain negative. However, GOLF, given an external $F_0$, still reaches lower spectral distortion. On LM-SSD, the recovered $F_0$ attains the highest overall accuracy of any method tested, including supervised neural pitch trackers applied off the shelf, and the glottal closure instants come within 0.53 points of REAPER's identification rate, without any $F_0$ label.

Comment
Submitted to ICASSP 2027

Closest papers

Nearest neighbours by meaning across the whole index.

Self-Supervised Source-Filter Model of Speech Production

eess.AS ·2026-10-13 ·Nani Meka, Raghavan Ramesh, Sri Rama Murty Kodukula
similarity 0.67
Abstract

Deep learning models of speech seldom represent the source-filter process by which speech is produced, relying instead on latent features with no physical correspondence to articulation or phonation. We propose SoFilNet, a self-supervised, differentiable Source-Filter Network that estimates the parameters of speech production directly from the raw waveform: a source, specified by fundamental frequency (F0), voicing and the aperiodicity of five frequency bands, and a filter, specified by ordered formant poles, additional poles that shape the rest of the spectral envelope, and antiresonance zeros. The model is trained only to reconstruct the input speech; no formant or pitch annotation is used. Since per-bin spectral losses favor blurred spectra wherever the model is uncertain, we train F0 and voicing against autocorrelation pseudo-labels refined with the model's own filter, aperiodicity against the periodic fraction of each band, and formant sharpness against spectral contrast. On VTR-FORMANTS, the estimated formants attain the lowest mean error among the reported methods, including supervised trackers, and on the unseen PTDB-TUG corpus the F0 frame error matches the best classical tracker and is lower than that of all learned trackers. The parameters are also consistent with speech production theory: aperiodicity rises with frequency and orders phone classes by the turbulence of their excitation, and the zeros deepen on nasal consonants, the lower one rising with place of articulation.

0.15

Tracing Decoder Artifacts for Compact Synthetic Speech Screening

cs.SD cs.AIeess.AS ·2026-09-25 ·Yi Chen Liu, Jian Liu
similarity 0.63
Abstract

Recent advances in speech synthesis and voice cloning have increased the need for reliable synthetic-speech detection, yet high-accuracy detectors increasingly rely on large pretrained models that are costly to invoke on every recording. Rather than replacing such detectors, we investigate a compact front-end screen that processes all inputs cheaply and forwards only suspicious recordings for more expensive analysis. To enable lightweight screening without a large learned encoder, we exploit spectral traces introduced by speech-generation operations. We analyze how learned upsampling and inverse short-time Fourier transform synthesis can produce predictable spectral artifacts and measure their presence directly in generated waveforms. Because the strength of these artifacts varies across generators, we combine decoder-guided spectral measurements with complementary descriptors of short-time spectral shape and temporal variation in a compact gradient-boosted tree. Across seven speech generators and two human-speech sources, the proposed screen achieves an equal error rate of 0.021\% with an estimated model storage of 151 KiB. When used as the first stage of a simulated cascade with a 1.15-billion-parameter detector, it reduces estimated detection energy by 84.4\% while operating at a 0.050\% synthetic-speech miss rate, demonstrating the potential of decoder-guided acoustic evidence for low-cost front-end screening.

0.30

Interpretable Destination-Aware Synthesizer Modulation Recovery

eess.AS ·2026-09-30 ·David Liu, Giulio Cengarle, David Cooper, Mark Vinton, Haici Yang
similarity 0.62
Abstract · Submitted to ICASSP2027

Synthesizer programming is a challenging task, particularly in configuring modulation: the time-varying control of parameters such as pitch, filter cutoff, and oscillator level by modulators such as Low-Frequency Oscillators (LFOs). Prior work has shown that modulation can be reconstructed from the clean audio through curve matching, yet the recovered parameters are often untransferable to modern synthesizers. We present an interpretable destination-aware modulation and waveform recovery pipeline that mirrors how musicians often recreate sounds on a synthesizer. First, the system identifies the modulated destinations; then it recovers the LFO shape associated with each destination, and estimates the oscillator waveform to match the source timbre. The training is supported by a differentiable synthesizer that incorporates a noise oscillator, and more modulation options than previous work. We carefully train the models for optimal perceptual quality, with extensive exploration of perceptual losses and adversarial training. Through objective and subjective evaluation, we show that predicting the destinations correctly is essential, our model excels in the modulation focused inverse synthesis tasks, and our Gammatone loss and CQT discriminator significantly outperform the traditionally-used MSS loss in improving perceptual quality. We provide audio samples from the subjective test.

0.19

Convolutional VocalTrax: Replacing Backpropagation Through Time with Convolution in Differentiable Articulatory Synthesis

cs.SD ·2026-11-12 ·Yongyi Zang, Debasish Ray Mohapatra, Randal Leistikow, Perry R. Cook
similarity 0.61
Abstract · 9 pages, 3 figures, 5 tables

VocalTrax recovers interpretable articulatory controls from a recording by gradient descent through a differentiable Pink Trombone synthesizer. Its vocal tract is a Kelly-Lochbaum waveguide updated twice per audio sample, so every gradient step backpropagates through a long recurrence, which is slow on GPUs. We propose Convolutional VocalTrax, which computes the exact frequency response of the same waveguide and applies it as a causal convolution. We assume that the tract is stationary within 5.8 ms blocks and that its lossy impulse response can be truncated after 93 ms. Gradients then pass through FFTs and a short recursion along the tract instead of through time. On 600 AudioMNIST recordings, the tract forward and backward pass runs 47 to 63 times faster on an A100 GPU, and a classifier trained only on real speech recognizes our reconstructions more often than those of VocalTrax (14.5% vs. 10.0%), although the log-mel distance to the target is 11% higher. Parameters fitted with our model keep this gain when rendered by the original VocalTrax (13.8%).

0.22

HC-AINet: An Auditory-Inspired Network with Harmonic Compensation for Low-Complexity Speech Enhancement

eess.AS ·2026-10-22 ·Nan Li, Xiaoxiong Wu, Huawei Liu, Ziping Zhao
similarity 0.61
Abstract · 12 pages, 5 figures, 9 tables

Speech enhancement on resource-constrained devices requires noise suppression with limited computation and minimal loss of spectral detail. Auditory-inspired subband processing reduces computation, but its smooth gains have limited ability to distinguish harmonic peaks from neighboring noise. We propose a harmonic-compensated auditory-inspired network (HC-AINet) for low-complexity speech enhancement. A triangular equivalent rectangular bandwidth (ERB) filterbank supplies compact features to a lightweight network comprising inverted-residual units, temporal Conformer layers, and parallel frequency-gated convolutions. The network predicts global and harmonic gains together with fusion weights. A harmonic-location map derived from the noisy spectrum confines the harmonic contribution to selected bins on the original frequency grid, while the global branch controls the broader spectral envelope. The compensated magnitude, noisy phase, and a magnitude-derived minimum-phase cue jointly condition a compact phase estimator for waveform reconstruction. On VoiceBank-DEMAND, the 64-band model achieves a PESQ of 3.20 and a STOI of 0.95 with approximately 0.27 million parameters.

0.20

PhaseGAN: High-Fidelity Vocoder via Decoupled Amplitude and GAN-Driven Phase Reconstruction

cs.SD ·2026-09-11 ·Wenzheng Zhang, Xueliang Zhang, Shulin He, Fei Zhao, Xin Liu +5 more
code/data similarity 0.61
Abstract · 15 pages, 1 figure

A vocoder is a pivotal component of modern text-to-speech (TTS) systems. Despite the significant progress of neural network-based vocoders, accurate phase reconstruction remains the main challenge limiting both audio quality and modeling efficiency. We introduce PhaseGAN, a lightweight vocoder that addresses this limitation through a "mel $\rightarrow$ Amplitude $\rightarrow$ Phase" reconstruction pipeline. By reconstructing amplitude and phase spectra via distinct methodologies, the proposed PhaseGAN outperforms state-of-the-art baselines while utilizing fewer model parameters and reduced computational requirements. The compact version generates high-fidelity audio with approximately 500K parameters and 1 GMAC computational load, making it highly suitable for real-time applications on edge devices. In addition, our approach exhibits exceptional musical audio synthesis capabilities despite no training on musical data, illustrating unprecedented cross-domain generalization. See https://github.com/phasegan/phasegan-audio-demo for demos of our work.

0.35

MPASCNet: Magnitude-Phase-Aware Adaptive Spectral Compression for Compressed Music Restoration

cs.SD ·2026-10-12 ·Xinlong Deng, Yu Xia, Qiuqiang Kong, Yin Cao, Jie Jiang
code/data similarity 0.61
Abstract · Submitted to ICASSP 2027

Compressed music restoration aims to recover high-fidelity audio from lossy codecs such as MP3 by restoring missing spectral content and reducing coding artifacts. Recently, generative adversarial networks and band-split models have made progress in compressed music restoration. However, existing methods either operate without explicitly modeling phase or rely on predefined frequency partitions, which may limit restoration performance and the use of input-dependent frequency information. To address these issues, we propose MPASCNet, a Magnitude-Phase-Aware Adaptive Spectral Compression network. Instead of using fixed bands, MPASCNet combines magnitude and phase features to guide cross-attention, adaptively compressing full-resolution spectral features into compact tokens. These tokens are then processed by a spectro-temporal backbone to model spectral-temporal dependencies. Finally, separate decoder branches reconstruct the full-resolution magnitude and wrapped phase spectra. Experiments on MP3 restoration show that MPASCNet outperforms the Apollo and AEROMamba baselines across multiple restoration metrics while using only 6.49 M parameters. The code is available at https://github.com/xinghour/MPASCNet.

0.26

Articulatory Source-Filter TTS: Physically Grounded Control through Vocal Tract Kinematics

eess.AS ·2026-09-30 ·Jesuraj Bandekar, Shinji Watanabe, Prasanta Kumar Ghosh
similarity 0.61
Abstract

Modern neural text-to-speech (TTS) systems achieve remarkable acoustic fidelity but act as black boxes, offering little interpretable control over the vocal tract filter. We propose a controllable source-filter TTS architecture grounded in articulatory kinematics. An Acoustic-to-Articulatory Inversion (AAI) model, enhanced by large-scale pretrained representations, generates kinematic pseudo-trajectories for a large TTS corpus. These trajectories condition the filter response, while predicted pitch and energy contours parameterise the glottal source. Source and filter are predicted independently, and the source is refined by an Optimal Transport Conditional Flow Matching (OT-CFM) module before recombination into the final spectrogram. Our model achieves intelligibility and naturalness competitive with similarly sized baselines, with only a modest spectral fidelity cost in exchange for explicit control. Evaluations reveal clear source-filter disentanglement, enabling stable prosodic scaling and cross-speaker source/filter recombination, where F0 remains tied to the source speaker while vocal-tract characteristics follow the filter speaker. Finally, direct manipulation of articulatory trajectories enables fine-grained control, such as accent modification, offering a new direction for interpretable speech synthesis. Audio samples: https://coding-phoenix-12.github.io/ArticulatorySFTTS/

0.26