CS Paper Radar

A low-dimensional dynamical basis for tongue shape control

Sam Kirkham, Patrycja Strycharczuk

cs.CL ·submitted 2026-10-16 13:39 UTC · v1

Abstract

Speech production involves the control of a high-dimensional articulatory system with many degrees-of-freedom, but a widespread hypothesis is that speakers navigate this complexity using a set of lower-dimensional control variables. Common candidates for these control variables include individual articulators (e.g. tongue body) and abstract tract variables (e.g. tongue body constriction location), but identifying the most appropriate coordinate system for articulatory control remains an open question. In this study, we advance a model in which holistic tongue shape modes represent a potential low-dimensional control basis for a simple dynamical law. We first discover empirical tongue shape modes in a cross-speaker coordinate space and then fit the model to ultrasound data on monophthongal vowels from 36 speakers of British English. A three-dimensional linear point attractor fitted to these time-varying modes recovers the dynamics of tongue shape with a high degree of accuracy. This suggests that complex tongue movement can arise from independent modes that linearly combine under simple dynamical laws, thereby relocating some complexity from the control signal to the coordinate system. We show that this framing explains why one-dimensional articulatory measures can exhibit observable nonlinearities, and hypothesize that shape modes are a plausible kinematic expression of underlying muscle synergies.

Comment
Pre-print currently under review

Closest papers

Nearest neighbours by meaning across the whole index.

From syllable representation via gesture generation to articulation: A spiking neural model

cs.CL ·2026-11-17 ·Bernd J. Kröger
similarity 0.63
Abstract · 47 pages, 9 figures. Submitted to the Journal of Neurolinguistics on 8 November 2026

Speech production requires a transformation from phonological representations into coordinated vocal-tract movements. This paper presents a spiking neural sensorimotor model within the Neural Engineering Framework and Semantic Pointer Architecture. Each syllable occurrence has a neural oscillator whose evolving phase coordinates its gestures and initiates the next syllable. Phase-dependent commands recruit separate neural populations for vocalic, oral-constriction, velopharyngeal, and glottal actions (gestures). Neural ramp integrators shape the rise and fall of gesture activation. Together with articulatory targets, these trajectories determine the extent and time course of articulatory movements. Opponent neural populations provide control signals to a geometric articulatory model. Thirty reference recordings cover an input corpus of eight lexical words, an alternative realization of one word, a syllable sequence, and isolated syllables. Two additional paired simulation subcorpora of sixty runs each, covering two words and one syllable sequence, assess activation reliability across neural realizations. Adjusting gesture command amplitudes increased successful realizations from 57/60 to 60/60 under the specified criteria, with ramp rate factors held constant. The model demonstrates how neural oscillator and ramp dynamics generate articulatory timing under specified phasing rules. It complements a preceding mental-lexicon model through an ordered syllable interface. Continuous online execution across all components and connection to the acoustic extension remain further integration steps. Accessible code and synchronized visualization support inspection of the neural activation trajectories and resulting vocal-tract movements.

0.30

Vocal-Tract Acoustics, Aerodynamics, and Overlap-and-Add Synthesis in DYNARTmo

cs.CL ·2026-10-17 ·Bernd J. Kröger
similarity 0.61
Abstract · 6 figures; ancillary Python reference implementation and word-transcription guide. No audio recordings or saved timelines

This paper presents the acoustic and aerodynamic extension of DYNARTmo, a dynamic articulatory model for visualizing and synthesizing speech production. Gestural control trajectories determine vocal-tract geometry, which is converted into an acoustic tube with nasal branching and effective lateral passages. A lumped aerodynamic network relates prescribed glottal motion and pulmonary pressure to airflow and supraglottal pressure storage. Phonatory excitation, oral friction, and glottal aspiration are synthesized as separate contributions. Frame-specific acoustic responses are combined by overlap-and-add, preserving response tails beyond individual control frames. Three German words, ``Tasse'', ``Muse'', and ``Lade'', illustrate the coordination of articulation, pressure, flow, and acoustic sources across vowels, nasals, laterals, fricatives, and plosives. The implementation has been optimized by the author primarily for sound recognizability; independent perceptual evaluation remains outstanding, and fricative-noise naturalness requires further refinement. The model provides an inspectable, articulatorily controlled production plant for future neural simulations of babbling, imitation, and early word acquisition. Further work will improve acoustic realism, reduce empirical source corrections, and extend the sensory information available to developmental learning models.

0.45

An Interpretable Acoustic Coordinate System for Japanese Consonant Morae in Dysarthric Speech

cs.SD cs.AI ·2026-11-15 ·Yuma Enomoto, Misa Yamaguchi, Hiroki Nishikawa, Ittetsu Taniguchi, Kazuhide Matsunaga +2 more
similarity 0.60
Abstract · 8 pages, 9 figures, IEEE SLT 2026

Dysarthria reduces speech intelligibility, requiring self-directed rehabilitation based on fine-grained feedback to capture gradual articulatory changes. For Japanese speakers, such feedback is naturally provided in mora units due to its distinct moraic timing and consonant-vowel structure. While recent visualization methods have explored distance-based feedback, most are limited to vowels. Extending them to consonants is challenging, as generative embeddings often smooth transient spectral structures. We propose a visualization system for Japanese consonant morae that maps articulatory deviations onto a two-dimensional space defined by temporal and spectral distances from healthy reference speech. Experiments show consistent geometric separation of consonant categories, with Silhouette Coefficients exceeding 0.5 and origin-based ROC-AUC scores above 0.86. Furthermore, evaluations using real dysarthric speech achieved a high ROC-AUC of 0.897 in screening for unclear pronunciations. Perturbation tests confirm orthogonal temporal and spectral shifts, yielding a physically interpretable mapping for rehabilitation.

0.23

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

cs.GR cs.CVcs.MM ·2026-10-02 ·Danzel Serrano, Przemyslaw Musialski
similarity 0.60
Abstract · 11 pages, 9 figures, 3 tables, under review

Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.

0.14

Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

eess.AS cs.GRcs.LG ·2026-09-24 ·Hyung Kyu Kim, Byungchan Hwang, Hak Gu Kim
top venue similarity 0.56
Abstract · Accepted to NeurIPS 2026

Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.

0.36

Validating Acoustic-to-Articulatory Inversion with Real-Time MRI: From Vocal-Tract Fidelity to Phonetic Consequences

eess.AS ·2026-11-16 ·Tharinda Piyadasa, Michael Proctor, Amelia Gully, Tuende Szalay, Emily Kiff +2 more
similarity 0.56
Abstract · 5 pages, 3 figures. Submitted to ICASSP 2027

Acoustic-to-articulatory inversion provides a promising way to estimate vocal-tract behaviour directly from speech, but inferred articulatory variables need to be validated against independent physical measurements before they can be interpreted as faithful estimates of articulation. We investigate this using paired real-time MRI (rtMRI) and speech. A WavLM-based inversion model is trained to predict eight vocal tract variables (TVs) extracted from rtMRI, achieving a mean correlation of 0.643 on held-out speakers. Prediction accuracy is substantially higher for constriction degree than for constriction location, and the inferred trajectories systematically underestimate the dynamic range of rtMRI-derived articulation. We then use articulatory-to-acoustic synthesis as a functional probe of these errors, comparing speech generated from inversion-derived and rtMRI-derived TVs while holding the source features fixed. The discrepancy is small for vowels, approximants, stops, and nasals, but increases markedly for fricatives and affricates, where speech production depends on maintaining a narrow critical constriction and small geometric errors can strongly affect the acoustics. These results show that current acoustic-to-articulatory models recover broad vocal-tract behaviour well, but further work is required before inferred TVs can be regarded as faithful measurements of fine articulatory geometry.

0.19

Articulatory Source-Filter TTS: Physically Grounded Control through Vocal Tract Kinematics

eess.AS ·2026-09-30 ·Jesuraj Bandekar, Shinji Watanabe, Prasanta Kumar Ghosh
similarity 0.56
Abstract

Modern neural text-to-speech (TTS) systems achieve remarkable acoustic fidelity but act as black boxes, offering little interpretable control over the vocal tract filter. We propose a controllable source-filter TTS architecture grounded in articulatory kinematics. An Acoustic-to-Articulatory Inversion (AAI) model, enhanced by large-scale pretrained representations, generates kinematic pseudo-trajectories for a large TTS corpus. These trajectories condition the filter response, while predicted pitch and energy contours parameterise the glottal source. Source and filter are predicted independently, and the source is refined by an Optimal Transport Conditional Flow Matching (OT-CFM) module before recombination into the final spectrogram. Our model achieves intelligibility and naturalness competitive with similarly sized baselines, with only a modest spectral fidelity cost in exchange for explicit control. Evaluations reveal clear source-filter disentanglement, enabling stable prosodic scaling and cross-speaker source/filter recombination, where F0 remains tied to the source speaker while vocal-tract characteristics follow the filter speaker. Finally, direct manipulation of articulatory trajectories enables fine-grained control, such as accent modification, offering a new direction for interpretable speech synthesis. Audio samples: https://coding-phoenix-12.github.io/ArticulatorySFTTS/

0.26

Structured Phonological Representations for Audio-Articulatory rtMRI Speech Classification

cs.CL cs.SDeess.AS ·2026-08-10 ·Abner Hernandez, Tomás Arias Vergara, Daiqi Liu, Andreas Maier, Paula Andrea Pérez-Toro
similarity 0.55
Abstract · Submitted for review at SLT 2026

Real-time MRI makes it possible to observe vocal-tract articulation during speech, but mapping these articulatory patterns to phonetic and phonological categories remains challenging. We investigate whether PhonoQ, an audio-based model trained to recognize structured phonological features, provides useful information for audio--articulatory modeling. Specifically, we extract representations from PhonoQ's Conformer module, whose training is shaped by supervision for manner, place, voicing, and vowel features. Using articulatory contours with synchronized audio-derived features, we compare WavLM-large and HuBERT-large baselines with models that incorporate PhonoQ-derived representations. Across unseen-speech and unseen-subject settings, these features improve macro-F1 for phonological targets including manner, place, voicing, vowel height, and vowel backness, and also improve fine-grained 39-phoneme classification. In a contour-only inference setting, audio-derived teacher supervision yields modest but consistent gains over contour-only training, indicating that phonological information from synchronized audio can be partially transferred to articulatory models. Finally, posterior analyses show interpretable surface-sensitive patterns consistent with flapping-like /t/ realizations, /t/-/r/ retraction or affrication, and nasal place assimilation.

0.04