CS Paper Radar

Articulatory Source-Filter TTS: Physically Grounded Control through Vocal Tract Kinematics

Jesuraj Bandekar, Shinji Watanabe, Prasanta Kumar Ghosh

eess.AS ·submitted 2026-09-30 21:27 UTC · v1

Abstract

Modern neural text-to-speech (TTS) systems achieve remarkable acoustic fidelity but act as black boxes, offering little interpretable control over the vocal tract filter. We propose a controllable source-filter TTS architecture grounded in articulatory kinematics. An Acoustic-to-Articulatory Inversion (AAI) model, enhanced by large-scale pretrained representations, generates kinematic pseudo-trajectories for a large TTS corpus. These trajectories condition the filter response, while predicted pitch and energy contours parameterise the glottal source. Source and filter are predicted independently, and the source is refined by an Optimal Transport Conditional Flow Matching (OT-CFM) module before recombination into the final spectrogram. Our model achieves intelligibility and naturalness competitive with similarly sized baselines, with only a modest spectral fidelity cost in exchange for explicit control. Evaluations reveal clear source-filter disentanglement, enabling stable prosodic scaling and cross-speaker source/filter recombination, where F0 remains tied to the source speaker while vocal-tract characteristics follow the filter speaker. Finally, direct manipulation of articulatory trajectories enables fine-grained control, such as accent modification, offering a new direction for interpretable speech synthesis. Audio samples: https://coding-phoenix-12.github.io/ArticulatorySFTTS/

Closest papers

Nearest neighbours by meaning across the whole index.

ECHO-G: Embodied Co-speech Humanoid mOtion Generation

cs.RO cs.AI ·2026-09-30 ·Yizhao Li, Pusen Gao, Ming Wang, Shaojie Shen, Shuo Yang +1 more
benchmarkdatasetcode/data similarity 0.70
Abstract · 8 pages, 5 figures, 3 tables. Project page: https://echo-g-project.github.io/

Generating full-body co-speech motion for humanoid robots requires coordinating speech prosody, linguistic content, and embodiment-specific motion. To this end, we present ECHO-G, a framework jointly conditioned on speech audio and timed transcripts. Its Speech-Grounded Diffusion Transformer (SGDiT) combines frame-aligned acoustic features with token-level linguistic context, preserving their distinct granularities. Trained with rectified flow matching, it models one-to-many utterance-motion relationships directly in robot space. To support training and evaluation, we introduce a BEAT2-derived audio-text-robot dataset and a benchmark covering co-speech characteristics, robot-motion quality, and runtime efficiency. Comparative evaluation supports direct robot-space generation over the evaluated human-motion generation and retargeting pipelines, while modality ablations highlight the benefits of joint audio-text conditioning. We further demonstrate deployment on a physical humanoid robot. A complementary video-rating study also favors joint conditioning over the alternatives. The dataset and training, inference, and evaluation code are available through our project page.

0.28

Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization

cs.SD ·2026-08-12 ·Peijie Chen, Zhuanling Zha, Zhipeng Nie, Weijie Wu, Yiming Liu +6 more
theory similarity 0.69
Abstract

In current zero-shot text-to-speech systems, conventional semantic tokenizers are typically optimized using supervised automatic speech recognition or self-supervised learning objectives. However, due to the inherent nature of speech, semantic and acoustic information cannot be completely decoupled, and ASR-based tokenizers discard acoustic details to focus on linguistic content; models relying on them usually struggle to achieve optimal speaker similarity. Furthermore, these tokenizers are optimized independently and lack direct supervision from downstream acoustic generation tasks. This isolated training creates a feature gap between the extracted discrete tokens and the continuous space required by acoustic models, fundamentally bottlenecking the upper bound of synthesis quality. To bridge this gap, we propose Phoenix TTS, a unified framework that tightly couples representation learning with generative acoustic modeling. Specifically, our speech tokenizer is optimized to reconstruct self-supervised features to maintain semantic richness, while simultaneously receiving direct supervision from a Flow Matching training loss. Through this joint training paradigm, the extracted discrete tokens successfully preserve essential semantic information and natively align with the feature space of the downstream Flow Matching model. Comprehensive evaluations highlight the efficiency and effectiveness of Phoenix TTS. Trained on 110K hours of data, the system achieves excellent speech intelligibility, yielding WER that consistently falls below that of ground-truth recordings. Simultaneously, it maintains robust zero-shot speaker similarity, rivaling or outperforming several prominent large-scale baselines. Furthermore, as an advantageous byproduct of this unified training, the learned tokenizer can be seamlessly adapted to zero-shot voice conversion tasks without requiring task-specific fine-tuning.

0.04

TTSonic: A Unified Framework for Composable and Controllable Speech Generation

cs.SD ·2026-11-03 ·Zhedong Zhang, Kaiwen Zhou, Junyan Wu, Shengqiang Li, Xuran Chi +3 more
similarity 0.69
Abstract · Demo Page: https://bytedance.github.io/TTSonic/

Modern text-to-speech (TTS) systems can synthesize natural and expressive speech, yet a practical speech foundation model must go beyond naturalness to offer stable quality, precise controllability, and a broad set of composable capabilities within a unified framework. We present TTSonic, a unified speech generation model that supports zero-shot TTS, voice cloning, prosody cloning, instruction following, voice conversion, and cloning from noisy reference speech. Rather than relying on separate task-specific models, TTSonic accommodates diverse conditioning signals, including text, natural language instructions, reference speech, and text-described timbre, within the same model framework. Specifically, we develop a semantic--acoustic disentangled speech tokenizer through ASR-supervised semantic learning and reconstruction-guided generative refinement. Serving as the representational foundation of our unified framework, the tokenizer enables the Text-to-Semantic autoregressive model to focus on semantic and prosodic planning, while the acoustic flow-matching model specializes in high-quality speech rendering under diverse acoustic and timbre conditions. Building on this framework, we incorporate CoT-guided instruction-following planning and Text2Timbre voice design to support controllable speech generation. Experiments across zero-shot synthesis, multilingual and cross-lingual TTS, instruction following, voice design, voice conversion, and noisy-reference cloning demonstrate the competitive performance and broad applicability of TTSonic.

0.10

Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models

cs.SD ·2026-10-03 ·Vasily Zadorozhnyy, Can Goksen, Kazuhito Koishida, Dung Tran
similarity 0.69
Abstract · 5 pages, 2 figures, 1 table, 1 algorithm, 11 equations

In recent years, flow-matching models have produced significant improvements in zero-shot text-to-speech synthesis. Conditioned on an audio prompt and text, these models learn a velocity field and generate speech by iteratively solving an ODE. During inference, the solver evolves a single state spanning both the prompt and the region to be generated, although only the generated region is ultimately retained. The discarded prompt state, however, still matters; its intermediate values influence generation through the velocity field that couples the two regions. As sampling continues, this state can drift away from the prescribed conditional path, introducing a discrepancy into subsequent generation updates. Unlike the unknown generated trajectory, the prompt path is available in closed form from the reference audio and the initial noise. We exploit this observation with Prompt-Consistency Inference (PCI), a training-free rule that restores the prompt block to its analytic value before each velocity evaluation, while leaving the generated block unchanged. PCI improves speaker similarity and intelligibility across the evaluated flow-matching TTS backbones without additional network evaluations. Our ablation studies further show that PCI keeps post-step prompt discrepancies smaller and that corrections covering the later sampling stages recover much of the observed similarity gain.

0.14

Validating Acoustic-to-Articulatory Inversion with Real-Time MRI: From Vocal-Tract Fidelity to Phonetic Consequences

eess.AS ·2026-11-16 ·Tharinda Piyadasa, Michael Proctor, Amelia Gully, Tuende Szalay, Emily Kiff +2 more
similarity 0.68
Abstract · 5 pages, 3 figures. Submitted to ICASSP 2027

Acoustic-to-articulatory inversion provides a promising way to estimate vocal-tract behaviour directly from speech, but inferred articulatory variables need to be validated against independent physical measurements before they can be interpreted as faithful estimates of articulation. We investigate this using paired real-time MRI (rtMRI) and speech. A WavLM-based inversion model is trained to predict eight vocal tract variables (TVs) extracted from rtMRI, achieving a mean correlation of 0.643 on held-out speakers. Prediction accuracy is substantially higher for constriction degree than for constriction location, and the inferred trajectories systematically underestimate the dynamic range of rtMRI-derived articulation. We then use articulatory-to-acoustic synthesis as a functional probe of these errors, comparing speech generated from inversion-derived and rtMRI-derived TVs while holding the source features fixed. The discrepancy is small for vowels, approximants, stops, and nasals, but increases markedly for fricatives and affricates, where speech production depends on maintaining a narrow critical constriction and small geometric errors can strongly affect the acoustics. These results show that current acoustic-to-articulatory models recover broad vocal-tract behaviour well, but further work is required before inferred TVs can be regarded as faithful measurements of fine articulatory geometry.

0.19

DuoFlow: Towards Synchronized Speech and Motion Generation with Fine-Grained Control

cs.CV ·2026-11-16 ·Jun Dong, Ronghui Li, Zhongyuan Hu, Yunqi Zhao, Lijie Guo +4 more
benchmarkcode/data similarity 0.68
Abstract

An embodied speaker's expressiveness depends not only on what is said, but also on how the voice sounds and how the body moves. However, existing joint speech-motion generation models are largely driven implicitly by transcript semantics or intermediate speech representations, offering little explicit control over motion hierarchy, body parts, or vocal style. To bridge this gap, we construct MotionSpeech, a structured benchmark for controllable joint speech-motion generation that includes transcripts, voice-style descriptions, and motion conditions at two hierarchical levels. Building on this benchmark, we propose DuoFlow, a dual-stream latent flow transformer that jointly denoises speech and motion under a shared noise level and a unified second-based temporal coordinate. Modality-specific residual connectors inject structured conditions into the corresponding streams, layer-wise bidirectional cross-modal attention coordinates their generation, and a monotonic phoneme pathway combines normalized progress anchors with CTC supervision to strengthen speech temporal modeling. DuoFlow provides a unified interface for fine-grained motion and vocal-style control and demonstrates the ability to synthesize high-quality, synchronized, and instruction-consistent speech-motion sequences. Project page, https://jundong-dev.github.io/DuoFlow-project-page.

0.13

ARIS: Low-Resource Glass-Box Neural Source-Filter Synthesis for Phonetic Stimulus Manipulation

eess.AS ·2026-09-24 ·Yiran Ding, Wenwei Xu
hot direction similarity 0.68
Abstract · 5 pages, 2 figures, 3 tables. Submitted to ICASSP 2027. Audio samples: https://n1r.github.io/ARIS_nsf/

Phoneticians often need to construct stimuli in which specific acoustic cues are precisely manipulated while preserving decent speech quality. Classical synthesis and modern neural methods sit along a trade-off between precise parametric control and high fidelity, and neural synthesis typically demands more data than phoneticians can easily obtain. We present ARIS (Analytic Resonant Interpretable Synthesis), a neural source-filter model that pairs neural parameter estimation with deterministic DSP synthesis. Every control is a coefficient of the synthesizer, so F0, formants and the glottal source can be edited directly. On five small single-speaker corpora in three languages, ARIS resynthesizes speech with quality comparable to WORLD and edits single parameters more accurately than Praat KlattGrid, with negligible crosstalk between cues. Compared with HiFi-Glot, pre-trained on a large corpus and fine-tuned on the same data, ARIS scores slightly lower on predicted naturalness but reproduces the recordings more faithfully and manipulates them more precisely. Audio samples: https://n1r.github.io/ARIS_nsf/.

0.46

Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning

eess.AS cs.CLcs.SD ·2026-09-29 ·Jian Chen, You Zhang, Mark Vinton
similarity 0.68
Abstract · Under Review

TTS systems with autoregressive semantic modeling have demonstrated strong zero-shot voice cloning performance and rich expressive variation, but their sequential decoding incurs substantial latency. Non-autoregressive alternatives offer much faster generation, yet often rely on more restrictive reference conditioning, such as requiring transcripts of the reference speech during inference. We present Tacit-TTS, an efficient transcript-free zero-shot voice cloning system distilled from IndexTTS2. Our model replaces autoregressive text-to-semantic decoding with masked non-autoregressive generation, introduces training-free acoustic length estimation, and accelerates the flow-matching renderer through ReFlow distillation. Across two English and two Mandarin datasets, Tacit-TTS achieves competitive zero-shot quality while generating speech over 10x faster than IndexTTS2 for utterances longer than 5 seconds. Its transcript-free conditioning further supports cross-lingual and non-lexical references. We validate this capability using references from eight other languages, infant babble, and synthetic gibberish, where transcript-dependent systems often degrade or fail due to unreliable ASR transcripts.

0.07