CS Paper Radar

Self-Distilled Pronunciation and Accent Control for Neural Text-to-Speech

Shuhei Kato

cs.SD eess.AS ·submitted 2026-09-15 14:13 UTC · v3

Abstract

Text-to-speech that reads raw text has no lexicon: a rare word is read as guessed, and a native Japanese listener accepts a word only if its reading and pitch accent are both right. A known remedy installs a reading-and-accent channel into a released model, but it needs many recordings. This paper removes the recordings: the frozen backbone reads a sentence containing a common word it already says correctly, and that output serves as the teacher for the same sentence with the word replaced by an annotated reading with its pitch accent. Screened raters judged the tag right on 0.80 to 0.93 of unseen difficult words on four backbones spanning autoregressive, diffusion, and encoder-decoder synthesis; plain kana, which cannot express an accent, got 0.38 to 0.60. On words needing no edit, naturalness is non-inferior on one backbone; on the other three, listeners prefer the unedited rendition by 0.19 to 0.26.

Comment
5 pages, 1 figure, 2 tables. Submitted to ICASSP 2027

Closest papers

Nearest neighbours by meaning across the whole index.

Reading-Level Verification of Kanji Words in Japanese Financial Speech Synthesis

cs.SD cs.CLeess.AS ·2026-10-29 ·Cheng Zhang
similarity 0.65
Abstract · 6 pages, 4 figures, 7 tables

Few-shot voice cloning makes it cheap to produce daily Japanese financial commentary with several synthetic speakers, but kanji words with more than one reading, such as the word for "market" (shijo vs. ichiba) or "overall trend" (taisei vs. oze), are still misread. A common quality gate transcribes each synthesized sentence with automatic speech recognition (ASR), converts both the input and the transcript to kana, and regenerates on mismatch. We show that this gate is unreliable in both directions: when the audio contains a misreading, the ASR language model frequently writes the expected kanji back, hiding the error, and when the audio is correct, ordinary recognition errors trigger needless regeneration. We propose a reading-level check that scores the expected reading and its confusable alternatives with a connectionist temporal classification (CTC) hiragana recognizer and flags a word when an alternative explains the audio better. On 540 utterances from three voice-cloned production voices covering 30 financial terms, with misreadings injected under controlled conditions, the transcript-based gate detected 78.9% of misreadings at a 44.4% false-alarm rate, whereas the proposed check detected 91.7% at 5.0% (AUC 0.990). On natural kanji input, it caught a front-end error that the transcript-based gate missed for one voice, and reduced regeneration requests from 38.9% to 6.1% of sentences.

0.19

Low Language Transfer Speech-to-Text

cs.CL ·2026-11-12 ·Husein Zolkepli
benchmark similarity 0.62
Abstract

A multilingual speech recogniser has heard much speech in its own languages. We ask how much of that it brings to a language it was never trained on, when it is fine-tuned on the little that language has. We score 28 open multilingual models untouched on Low-Language-TTS, 50 low-resource languages in 11 writing systems, and fine-tune 24 of them on at most 1,000 clips a language, 82 hours in all, with a dev set choosing every epoch. Fine-tuning lowers every model's macro character error rate (CER), by 16.6 to 53.1 points, to 9.5 at best, and a word language model takes three CTC models 1.2 to 1.4 points lower; WER is still 34.4 at best. On the 12 languages that no model lists, every model starts at 46 or worse and the best ends at 11.7. The same architectures from random weights stay 60 points or more behind, so the gain is transfer, and an encoder left untouched already gives a Whisper or Qwen decoder most of what it needs. It is the language, not the corpus: on FLEURS recordings that ours do not share, the median model keeps 79% of its gain. It costs the languages the models knew, 5.7 points of CER on English, Spanish, French and German in the median model. Low-rank adapters on 4% of the weights come within 1.1 points of tuning everything, and on 2 of 3 models forget less. Seq2seq and language-model decoders are done within three to five epochs, while a new CTC head is still improving at twenty or thirty. Data beats epochs: 25 clips a language take Qwen3-ASR 0.6B from 60.9 to 39.3, and at equal clips presented, ten times the data is worth 9.7 points. A model that first learned forty of the languages learns the other ten no faster. On a public Malay benchmark, untouched Whisper large-v3 scores 8.9 when it detects the language and 14.9 when told the clip is Malay, because it then translates the English that speakers mix in; trained on the Scicom telephony dataset it scores 6.0.

0.23

Ruby-OPD: Enhancing Pronunciation Controllability and Accuracy for Flow-Matching Text-to-Speech via On-Policy Distillation

cs.CL ·2026-11-04 ·Rui Hu, Xiaolong Lin, Zhenpeng Zhan
benchmark similarity 0.62
Abstract · 5 pages, 3 tables, work in progress

Achieving both high pronunciation accuracy and fine-grained controllability remains a central challenge in Japanese text-to-speech (TTS), as Japanese is an unsegmented language mixing logographic Kanji and phonographic Kana with extensive polyphony. Traditional cascaded systems convert text into phoneme sequences via a front-end, ensuring pronunciation control but discarding lexical semantics and prosodic cues in surface orthography. Conversely, recent zero-shot TTS models directly consume raw orthographic text to improve naturalness, yet suffer from polyphone errors and lack explicit pronunciation control. To bridge this gap, we propose a two-stage framework for flow-matching TTS. First, we investigate Ruby text representations with orthography perturbation to jointly preserve surface forms and target pronunciations for reliable control. Second, we introduce On-Policy Distillation (OPD) on large-scale text-only corpora to enhance pronunciation accuracy on raw orthographic text without paired speech. On the Joyo-Kanji-Yomi benchmark, our model achieves 97.13% Accuracy, 2.02% Target PER, 0.70% Sentence PER, and 4.21% Sentence CER under raw text while retaining strong controllability.

0.10

AccentCanvas: Activity-Guided Bidirectional Accent Conversion

eess.AS ·2026-11-06 ·Jaehun Kim, Kang Zhang, Sung Woo Kim, Junmo Kim, Joon Son Chung
similarity 0.61
Abstract · 5 pages, 2 figures

Accent conversion for dubbing must change pronunciation while preserving content, voice, and source timing. Normalization alone cannot produce requested non-native accents, and matching total duration does not ensure local speech/pause alignment. We present AccentCanvas, a shared discrete diffusion model for normalization and L1-to-L2 conversion, trained on cross-paired shared-prompt recordings and synthetic pairs. Frame-level speech activity and repetition-preserving units guide local timing through synthesis. Across both tasks, AccentCanvas achieves the highest accent conversion performance (Accent Arrival) and mean listener-rated accent similarity among compared systems, with over 73% lower speech-boundary error than the best baseline for each task. These gains accompany higher L1-to-L2 WER than TokAN and Vevo-Style, highlighting a tradeoff between requested-accent resemblance and ASR-based content recognition. https://smokedindia.github.io/accentcanvas/

0.19

PronCFT: Pronunciation Contextual Fine-Tuning for Contextual Biasing of Out-of-Vocabulary Words in LLM-Based ASR

eess.AS ·2026-10-19 ·Yifan Xu, Kaixun Huang, Mengzhe Geng, Mingyu Cui, Shen Huang +2 more
code/data similarity 0.61
Abstract · 5 pages, 1 figure, 6 tables

LLM-based ASR systems often replace rare and out-of-vocabulary (OOV) words with familiar words that sound similar. Two factors contribute to this error: the model has not learned how an unseen word's sound maps to its spelling, and it may favor a familiar word over the one spoken, a failure we call prior suppression. Providing candidate pronunciations alone may not help: the model can treat a phoneme annotation as a marker that raises a candidate's output probability without checking its pronunciation against the speech. We propose PronCFT, which uses fixed phoneme sequences and trains the model to transcribe speech into phonemes, match them to context candidates, and output the matched spelling. During training, we alter candidate spellings while keeping their pronunciations unchanged, discouraging the model from choosing a familiar spelling without checking how the candidate sounds. A curriculum then gradually removes intermediate outputs to reduce decoding latency and prevent phoneme transcription errors from affecting the final text. The model's retained ability to use pronunciation cues suggests that this ability can be internalized. On an OOV test set filtered from open-source benchmarks, the final model, PronCFT-C, significantly improves recall from 59.89% to 83.57% (+23.68 pp). Wrong-phoneme and spelling-interference tests indicate that the gain depends on phoneme content rather than the presence of an annotation.

0.34

Pronunciation-Oriented Reinforcement Learning for Japanese Text-to-Speech with Kana-Domain ASR Rewards

cs.SD eess.AS ·2026-10-06 ·Shiao Zhu, Lianbo Liu, Kai Washizaki, Koki Nikaido, Yui Sudo
similarity 0.60
Abstract

Character error rate (CER) computed by automatic speech recognition (ASR) is widely used as an intelligibility reward for reinforcement learning (RL) post-training of text-to-speech (TTS) systems. For Japanese, however, orthographic CER introduces a representation mismatch for pronunciation-oriented optimization: distinct kanji readings may collapse to the same orthographic representation, while equivalent pronunciations may admit different orthographic forms. We instead compute CER in the kana domain using a kana-transcribing ASR model and reference readings (Kana-CER). Under matched group relative policy optimization (GRPO) conditions, Kana-CER reduces target-kanji reading error by approximately 26% relative to the orthographic CER reward, while maintaining comparable orthographic CER and similar speaker similarity and objective speech quality. It also reaches its best validation performance in substantially fewer optimization steps (4k vs. 18k). We further observe severe output elongation under unregularized Kana-CER optimization, which is substantially suppressed by KL regularization.

0.25

Error Rate is Not Enough: Costs of Augmentation in Low-Resource ASR

cs.CL ·2026-10-09 ·Katsumi Ibaraki, David Chiang
similarity 0.60
Abstract · 5 pages

Data augmentation is standard practice for low-resource automatic speech recognition (ASR), but its benefits are almost always reported as aggregate error rates, leaving open why and when a method helps. On four low-resource languages and English at five data sizes, we cross two augmentations that modify existing audio (speed perturbation and SpecAugment) with two that modify text transcriptions and synthesize audio from it. We analyze errors at the phoneme level and find that they help for complementary reasons. SpecAugment has the most reliable aggregate improvements, but increases tone and palatalization errors, whereas text-based methods introduce consonant insertions; in combination, each reduces the errors the other introduces. This complementarity is predictable. Tone and palatalization are each signaled by a single acoustic property and occur least frequently in training (lowest training exposure), and text-based methods that flatten word frequency raise their frequency by 37--38%, reversing those errors while further improving aggregate error. The improvement from combining the methods exceeds the sum of their individual improvements; the superadditivity vanishes by 100 hours, limiting the complementarity to the low-resource regime. Where specific error types matter, we recommend selecting augmentation by training exposure and acoustic properties, both measurable before any training.

0.11

How Should We Evaluate Naturalness for Synthetic Speech with a Foreign Accent?

cs.SD eess.AS ·2026-10-09 ·Kentaro Onda, Satoru Fukayama, Daisuke Saito, Nobuaki Minematsu
published similarity 0.60
Abstract · Accepted to SLT2026

Recent advances in speech synthesis technology have enabled the generation of speech with diverse characteristics. One such characteristic is a foreign accent. In our preliminary experiments, we found that UTMOS, a widely used automatic speech quality assessment system, tended to assign lower scores to speech with stronger foreign accents, even though it is produced by a human speaker. This observation raised a question: How do humans evaluate the naturalness of foreign-accented speech? We conducted a large-scale subjective evaluation involving both native and non-native listeners under different evaluation instructions. The results showed that humans also considered accentedness when evaluating naturalness, but that defining naturalness in terms of human-likeness and selecting raters who have greater exposure to the target accent may enable more appropriate evaluation. This study aims to provide a starting point for future discussions on fairer and more appropriate evaluation of synthetic speech with a foreign accent.

0.29