CS Paper Radar

Optimal Sampling and Noise Mitigation for Vocal Fold Kinematics in High-Speed Videoendoscopy

Harikrishnan Unnikrishnan, Rita Patel, Kevin D. Donohue

eess.IV ·submitted 2026-10-28 18:53 UTC · v1

Abstract

Objectives: High-Speed Videoendoscopy glottal area waveforms conflate lateral fold excursion with longitudinal opening, precluding direct estimation of velocity, acceleration, and left-right asymmetry. Displacement waveforms preserve these quantities but are susceptible to spatial quantization and temporal sampling errors. We characterize these errors, evaluate reduction via Kalman smoothing and temporal interpolation, and determine optimal frame rate bounds relative to pixel excursion, speed quotient (SQ), and open quotient (OQ). Methods: Sampling theory was used to derive analytical frame-rate bounds required for kinematic feature fidelity. Kalman smoothing was compared with FFT-based denoising, paired with sinc or spline interpolation. Interpolation was evaluated on 4000 fps recordings (N=55) downsampled and reconstructed. A 16000 fps dataset (N=12) served as reference for noise reduction across 2000-8000 fps downsampled waveforms evaluated on peak displacement, velocity, OQ, and SQ. Results: Sinc interpolation yielded lower median mean absolute error than spline (0.29 vs 0.52 pixels), with 85% of recordings (47/55) versus 45% (25/55) below 0.5 pixels. For kinematic feature estimation (N=12), Kalman-sinc outperformed FFT-sinc in 75% of feature-by-frame-rate pairings. Conclusion: Computing kinematic features without artifacts requires joint consideration of temporal and spatial resolution. Kalman-sinc preprocessing at 8000 fps is recommended for velocity and SQ measures. Peak lateral displacement can be measured at 2000 fps, while OQ requires 4000 fps for typical spatial resolutions (9-14 pixels maximum excursion). Pixel excursion should be reported alongside kinematic measures to contextualize data reliability.

Comment
This is the preprint Auhors manuscript to be published in Journal of Voice

Closest papers

Nearest neighbours by meaning across the whole index.

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

cs.GR cs.CVcs.MM ·2026-10-02 ·Danzel Serrano, Przemyslaw Musialski
similarity 0.59
Abstract · 11 pages, 9 figures, 3 tables, under review

Speech-driven 3D facial animation can reproduce recognizable mouth poses. However, it can simplify the motion between them, and that motion carries coarticulation, the way the sounds around each sound shape its articulation. We introduce a geometric measure of this trajectory shaping: lip-path length compared with the shortest route through the vowel, consonant and vowel positions of a speech segment. In contrast to the endpoint chord, this consonant-aware route accounts for obligatory transit and avoids degeneracy, while preserving invariance to uniform motion gain. The measure needs only a forced alignment, so it applies where no ground truth exists. We demonstrate it on four state-of-the-art methods, one per architectural family, real-time and offline. All four trace flatter lip trajectories than captured speech. Against frame-rate-matched ground truth, DiffPoseTalk, ARTalk and FaceFormer show clear deficits, equivalent on this measure to removing 15-60% of real speech's fast articulatory component. CodeTalker is marginal on the primary measure and clear on a companion measure. A pre-registered study with 97 viewers and 3,523 judgments underpins the measured direction: controlled damping of real motion lowers the score and is penalized, whereas exaggeration shows no detected penalty over the tested range. Viewers also prefer real speech in 73.4% of sentence comparisons and, in the aggregate, on single words. Together, the measure, its calibration and the study identify a perceptually relevant loss of trajectory shaping and a concrete target for improving synthesized articulation.

0.14

Beyond Region Memory: Local Point Tracking for Vocal Cord Localization in Video Laryngoscopy

cs.CV ·2026-10-28 ·Huayu Wang, Gyungmin Ko, Jung Heon Kim, Jia-Xian Jian, Dae Kon Kim +7 more
published similarity 0.56
Abstract · Accepted by WACV 2027

Reliable vocal-cord localization in emergency video laryngoscopy remains visually challenging due to rapid tissue deformation, specular highlights, and instrument occlusions. We propose Local Point Tracking (LPT), a robust framework that combines a single-frame detector, local point propagation, and stateless mask refinement. LPT follows a localize first, snap second pipeline. First, a single-frame semantic detector generates sparse anchors, within which LPT initializes local query points (SAM-supported by default, with a regular grid as fallback). A point-propagation module tracks these points through detector-missed frames and recovers temporal bounding boxes from their geometry. Second, because point-derived boxes can be loose around the visible boundary, a box-snap step refines the selected box to the target boundary. This decomposition keeps temporal state in local point coordinates rather than a stateful mask or whole-region representation, while the spatial refinement module handles single-frame boundary precision without feeding mask errors back into temporal memory. Extensive evaluations on two distinct datasets demonstrate that our proposed LPT method achieves strong overall performance against existing baselines, with the best mAP_50, success AUC, and missing rate on both datasets. Furthermore, ablation studies confirm that point tracking primarily enhances temporal tracking robustness, while box-snap refinement effectively improves boundary tightness.

0.40

Investigating the trackability of landmarks in NBI and WLI

eess.IV cs.CV ·2026-10-23 ·Jonas Below, Piermarco Gerli, Anna Jötten, Dennis Eggert, Axelle Felicio-Briegel +3 more
similarity 0.55
Abstract · Accecpted at CURAC conference 2026

Reliable visual tracking of anatomical landmarks in endoscopic video is a key prerequisite for computer-assisted interventions in laryngeal surgery. While Narrow Band Imaging (NBI) is clinically established for enhancing vascular contrast, its effect on automated tracking relative to conventional White Light Imaging (WLI) remains unexplored. We present a quantitative comparison of landmark trackability in NBI and WLI using a forward-backward error framework applied to 61 endoscopic videos from 39 patients. Two regions of interest are evaluated: the vocal folds (VF) and mucosa with a prominent vascular pattern (MPVP). We compare a classical feature-based tracker (CSRT) against learning-based point trackers (CoTracker3-Online and TrackOnR) across both modalities. NBI consistently reduced the forward-backward error in both regions, reaching statistical significance for the vocal folds, while failure rates stayed low. These findings indicate that NBI can improve tracking robustness in laryngeal endoscopy, with implications for motion analysis, navigation, and robotic assistance.

0.24

Fréchet Oral Distance: A Coordinate-Free Metric and Benchmark for Speech-Driven Oral Animation

cs.CV cs.GR ·2026-11-01 ·Jaehong Myung, Seungho Eum, Hyunjung Chung, Unsang Park
benchmark similarity 0.54
Abstract

Recently, speech-driven facial and oral animation technologies have been used in various fields such as education, speech therapy, and healthcare. But methods to objectively evaluate the quality of generated results are still limited. Accordingly, we propose the Fréchet Oral Distance (FOD), which uses embeddings from an Oral Motion Encoder (OME) and a TIMIT-Vis reference distribution. In experiments, FOD reproduced model rankings similar to the coordinate-based metric, Lip Vertex Error (LVE), and showed trends consistent with user preference. Our framework provides objective and reproducible benchmarking without relying on expensive equipment or subjective evaluation, and it can be extended to compare a variety of 2D/3D models.

0.06

Articulatory Source-Filter TTS: Physically Grounded Control through Vocal Tract Kinematics

eess.AS ·2026-09-30 ·Jesuraj Bandekar, Shinji Watanabe, Prasanta Kumar Ghosh
similarity 0.54
Abstract

Modern neural text-to-speech (TTS) systems achieve remarkable acoustic fidelity but act as black boxes, offering little interpretable control over the vocal tract filter. We propose a controllable source-filter TTS architecture grounded in articulatory kinematics. An Acoustic-to-Articulatory Inversion (AAI) model, enhanced by large-scale pretrained representations, generates kinematic pseudo-trajectories for a large TTS corpus. These trajectories condition the filter response, while predicted pitch and energy contours parameterise the glottal source. Source and filter are predicted independently, and the source is refined by an Optimal Transport Conditional Flow Matching (OT-CFM) module before recombination into the final spectrogram. Our model achieves intelligibility and naturalness competitive with similarly sized baselines, with only a modest spectral fidelity cost in exchange for explicit control. Evaluations reveal clear source-filter disentanglement, enabling stable prosodic scaling and cross-speaker source/filter recombination, where F0 remains tied to the source speaker while vocal-tract characteristics follow the filter speaker. Finally, direct manipulation of articulatory trajectories enables fine-grained control, such as accent modification, offering a new direction for interpretable speech synthesis. Audio samples: https://coding-phoenix-12.github.io/ArticulatorySFTTS/

0.26

AVTR-1: Open Stack for Real-Time Interactive Avatars

cs.CV cs.AIcs.LG ·2026-09-19 ·Artem Kravtsov, Dmitrii Ziganshin, Vsevolod Poletaev, Gleb Balitskiy, Anastasia Tikhonova +2 more
similarity 0.53
Abstract

Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live system must synchronize the model's output with speech from an external voice agent, schedule video frames for playback, and handle interruptions. We introduce AVTR-1, an open stack for real-time interactive avatar conversations, built around a compact 153M-parameter autoregressive flow-matching motion generator conditioned on both participants' audio. We adapt its audio encoder for streaming through self-distillation. The stack turns the model's chunk-based generation into a continuous, synchronized audio-video stream driven by an external voice agent, and we analytically derive its contribution to the user-facing latencies and validate the resulting bounds with two commercial voice agents. Further experiments demonstrate that AVTR-1 leads the compared dyadic systems on all reported visual-quality metrics and most conventional listening-motion metrics while remaining competitive in lip synchronization. Its inference runtime operates in real time on data-center and consumer GPUs. However, conventional listening metrics do not establish whether the paired speaker's speech contributes to generated motion. We therefore introduce the Reference-Based Directed Granger Gain (R-DGG), which measures the additional predictive information carried by speaker speech after accounting for listener history and speaker motion. R-DGG finds statistically supported predictive dependence for recorded listeners and all evaluated dyadic systems, but not for talking-head generators without paired audio or mismatched speaker-listener pairs. We release the model weights, renderer, and serving backend under component-specific licenses.

0.41

Pitch Smoothing Using Relative Interval Networks

eess.AS cs.SD ·2026-09-30 ·Chin-Yun Yu, Chi-Jen Peng, Li Su, György Fazekas
similarity 0.53
Abstract · Submitted to ICASSP 2027

Pitch tracking systems typically couple a per-frame fundamental frequency ($F_0$) estimator with a temporal smoothing stage to obtain continuous trajectories. Conventional Viterbi smoothers enforce first-order continuity but lack long-term temporal awareness and could lock into octave errors across corrupted frames. We propose Relative Interval Networks (RIN), a trajectory smoothing framework that reconciles per-frame pitch estimates with data-driven multi-hop pitch differences. We extract robust relative pitch intervals across arbitrary frame offsets using Variable-Q Transform cross-correlation. We formulate pitch smoothing as an $L_1$-norm optimization problem and prove its equivalence to a minimum cost circulation problem, solved efficiently via linear programming. Evaluations across speech, singing, and instrumental datasets show that RIN substantially improves weak estimators, matches or outperforms Viterbi decoding at a comparable computational cost, and provides superior robustness under certain acoustic degradation.

0.35

OAA: Three Phases of Vocal Guidance in Human-Drone Teleoperation

cs.RO ·2026-08-11 ·Allan Henry, Christian Graff, Solange Rossato, José-Ernesto Gomez-Balderas, Sylvain Huet
top venue similarity 0.53
Abstract

Voice-guided teleoperation requires systems that adapt to the evolving dynamics of human guidance. Yet most voice-controlled robot systems treat spoken commands as a stationary stream, ignoring how the guide's communicative behavior changes as the task progresses. Using motion capture and speech data from two experimental configurations, humanhuman guidance (finger pointing, N =10 dyads) and humandrone teleoperation (gamepad control, N =29 dyads), we show that spontaneous vocal guidance consistently organizes into three kinematically and linguistically distinct phases: Orientation, Approach, and Adjustment. These phases are identified automatically via change point detection on 3D trajectory signals, and validated statistically (Kruskal-Wallis, p<.001). Three lexical families replicate across configurations: rotation vocabulary marks Orientation, translation vocabulary is scarce there, and attenuators accumulate toward Adjustment. Together with inter-utterance silence, these cues mark the Orientation boundary that speech rate alone leaves unmarked. The same three-phase structure emerges in both configurations despite radically different motor interfaces, suggesting it is an intrinsic property of human spatial guidance rather than an artifact of the experimental setup. We discuss implications for OAA-aware adaptive control in voice-guided teleoperation.

0.40