CS Paper Radar

Synthetic Child-Adult Interactions Are Not Developmentally Plausible

Bastian Bunzeck, Francesca Padovani

cs.CL ·submitted 2026-10-26 08:40 UTC · v1

Abstract

Synthetic data has become a common resource for training language models, and this trend has spilled over into research on developmentally plausible language modeling. However, it remains unclear whether LM-generated child-adult interactions actually resemble natural ones. We compare the synthetic TinyDialogues dataset to natural dialogue across a broad range of linguistic measures. While synthetic data matches natural data on the lexical level, it drastically underestimates the messy, fragmented nature of spoken interaction. To mitigate this, we propose instruction-tuning with data created from real dialogues, and show in a pilot study that a small model approximates the statistical tendencies of natural data after moderate parameter-efficient fine-tuning.

Closest papers

Nearest neighbours by meaning across the whole index.

Specialization, Diversity, and Scaling in Child-Scale Language Models

cs.CL cs.AIcs.LG ·2026-10-27 ·Steven Y. Feng, Michael C. Frank
code/data similarity 0.74
Abstract · Code and data at https://github.com/styfeng/child-scale-lm

Modern language models are trained on far more language than children experience, but we know much less about learning from small, naturalistic datasets. We train small GPT-2 models on transcripts of speech with children from dense home recordings (BabyView) and on synthetic child-directed dialogue (TinyDialogues), varying the amount, source, and diversity of training data. Models trained on one child's language environment predict that environment substantially better than other children's environments. Much of this specialization is lexical and distributional: a simple 5-gram model reproduces the cross-child pattern (r=.91). Training on data from multiple children clearly improves prediction of pooled held-out language relative to training on a single child's input without a similarly clear improvement on grammatical benchmarks. Across data sizes, naturalistic and synthetic training inputs also produce different grammatical and lexical-semantic patterns, with important differences between grammatical phenomena. Together, these results show that small-data language learning depends not only on how much language a model sees, but on which environments it comes from and how well that input matches what the model is asked to predict.

0.18

Pretrained Persona Mixture Models and Tandem Models for Human Simulation

cs.CL ·2026-09-18 ·Minwoo Kang, Téa Wright, Seun Eisape, Ayush Raj, Suhong Moon +4 more
position similarity 0.71
Abstract · 11 pages in body, 36 with appendices. 7 figures. 8 tables

We argue here that the current dominant practice in LLM human simulation: prompting instruction-tuned assistant language models to role-play personas, is inaccurate and produces stereotyped predictions (lacking natural diversity). It has previously been shown that LLMs can be bound to personas using naturalistic, freetext dialog avoiding stereotyping. Here we show that binding can also be achieved using short, individual samples of dialog from specific people. Demographics can be added later without negative effects by simply querying the model. We use the term Persona Mixture Models (PMMs) for well-calibrated human models, currently realized as pretrained base models. We show that PMMs produce more accurate predictions than instruction-tuned models and retain more of the lexical, semantic, and pragmatic diversity found in human dialog. We measure realism and diversity of LLMs simulating human interlocutors across a diverse set of corpora spanning open-domain text, human-AI chat, and task-oriented dialogue between human speakers. However, base pretrained models can produce out-of-domain dialog and may lose some of the human's internal state over long contexts. We propose and explore tandem models which combine a pre-trained model with an instruction-tuned supervisor. Tandem models achieve the best overall accuracy and diversity in our experiments.

0.22

How to start a conversation proactively?

cs.HC cs.LGcs.RO ·2026-08-22 ·Mohit Nathani, Aaditya Gupta, Eshwar Dhande, Chayan Sarkar
publisheddatasetcode/data similarity 0.66
Abstract · The article is accepted in IEEE RO-MAN 2026. The dataset is available at https://github.com/mohit-nathani1/conversation-opener-dataset

Proactively initiating a conversation is a fundamental social competence and a critical capability for social robots. However, determining how and when to start a conversation in a given situation remains an open challenge, primarily due to the lack of a principled definition of conversational context. While recent large language models (LLMs) demonstrate impressive conversational fluency, they are predominantly turn-reactive and often produce generic or monotonous openers when tasked with initiating interaction, even when contextual information is available. Such limitations are particularly problematic for embodied social robots, where inappropriate or poorly timed initiations can negatively affect user engagement and rapport. In this work, we propose a schema of contextual variables that jointly characterize situations relevant for proactive conversation initiation. The schema is designed to structure perceptual, situational, and interactional cues such that they can be effectively exploited by language models. Furthermore, motivated by the absence of suitable training data, we introduce a curated dataset of contextually-aligned conversation openers (CCO) covering diverse everyday scenarios. Using this dataset, we fine-tune small language models (SLMs) to generate proactive conversation initiations. Through a comparative evaluation involving zero-shot LLMs, off-the-shelf SLMs, and our fine-tuned models, we demonstrate that explicit contextual structuring and targeted data curation substantially improve the quality and contextual appropriateness of conversation openers. Our results highlight the importance of context modeling for proactive social interaction and provide actionable building blocks for conversational robots.

0.41

Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance

cs.CL eess.AS ·2026-09-25 ·Manato Yaguchi, Yotaro Kubo, Hikaru Asano, So Kuroki
similarity 0.66
Abstract · Submitted to ICASSP 2027. 5 pages, 1 figure, 2 tables

Tandem speech-to-speech architectures couple a responsive speech frontend with an asynchronous text backend. In KAME, a large language model (LLM) serves as the backend, supplying candidate responses as guidance to the speech frontend while the user is still speaking. Ordinary conversation recordings capture the eventual response but not the guidance the backend would supply during the user's utterance. Generating the missing guidance with a simulator LLM adds substantial data-preparation overhead when training on real conversations. We propose randomized intermediate guidance, which derives guidance directly from the conversation corpus rather than simulating backend LLM behavior. During training, target responses provide informative guidance, while randomly sampled responses provide potentially irrelevant updates during the utterance. This combination aims to teach the frontend to use backend information selectively. On synthetic dialogues, KAME trained with this recipe achieves response quality comparable to that of the LLM-generated and similarity-based baselines. Training on 3.8k hours of real conversations improves smooth turn-taking and audio-judge naturalness over synthetic-data KAME while retaining a response-quality advantage over Moshi. These results show that randomized guidance offers a practical route to combining the response-quality benefits of tandem models with natural conversational behavior learned from real speech.

0.24

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

cs.CL ·2026-10-19 ·Heyang Liu, Lixin Zhang, Jiayi Huang, Ziyang Cheng, Yuhao Wang +4 more
similarity 0.65
Abstract

The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.

0.08

Why is AI-generated language so hard to understand?

cs.CL cs.AI ·2026-10-21 ·Thomas Hikaru Clark, Matias B. Martin, Kyle Mahowald
similarity 0.65
Abstract

There is a widespread sense that AI-generated text has recently become harder to understand. In this study, we investigate a set of psycholinguistic features that are known to correlate with human processing difficulty (e.g., object relative clauses, long syntactic dependencies, low-frequency n-grams). Using a novel corpus of AI-generated text across diverse domains, we find that AI models use these difficult features more often than paraphrases generated by round-trip translation. Compared with human writing, all nine frontier and near-frontier models use more inanimate subjects with animate verbs and more low-frequency n-grams, and most also use longer dependencies, more stacked nouns, and more reduced object relatives in conversation; we also demonstrate substantial variation across model families. We propose and test a hypothesis as to why: while human language is optimized for successful communication, AI-generated text is increasingly optimized for other goals like math, code, and reasoning, in which there is a pressure for \textit{information density} rather than successful communication. We provide a proof-of-concept for this hypothesis by fine-tuning a language model under a length penalty, which leads to a divergence from human-like linguistic behavior. Our findings highlight the axes of misalignment between linguistic processing demands in humans and LLMs, and point toward possible remedies.

0.20

Zero-Shot Voice Cloning for Synthetic Data Augmentation in Children's ASR

eess.AS ·2026-11-02 ·Mohammad Mohsen Goodarzi, Stefan Wabnik, Matthias Stennes, Stephanus Volke, Alexander Schwarz
published similarity 0.65
Abstract · Accepted to 2026 IEEE MMSP

Children's automatic speech recognition (ASR) lags behind adult ASR due to theacoustic mismatch between adult-trained acoustic models and children's speech, as well as the scarcity of labelled children's speech corpora. We investigate whether zero-shot text-to-speech (TTS) synthesis with voice cloning, requiring only a short reference clip per speaker, can serve as a practical substitute for real children's speech when fine-tuning large pre-trained ASR models. Using the OmniVoice multilingual TTS system we synthesise up to 16,000 child-like utterances from two English corpora (ENNI, clinical narrative samples, ages 7-12, and MyST, conversational science-tutor speech, grades 3-5) and fine-tune Whisper-small under different experimental conditions covering real data, synthetic data, cross-corpus data, and all-data combination. A systematic sensitivity analysis over speaker count, text-pool size, and utterances per speaker reveals that synthetic data alone reduces word error rate (WER) from a 19.8 % zero-shot baseline to 16.4 % on ENNI, recovering approximately 30 % of the gap to real-data fine-tuning (8.8 %). Combining synthetic with real data is essentially indistinguishable from real-data-only fine-tuning (9.2 % vs. 8.8 %). Speaker count plateaus at ~40 speakers for cross-domain transfer, while multi-speaker acoustic diversity per transcript, rather than raw text coverage, emerges as the primary determinant of cross-domain generalisation; out-of-vocabulary (OOV) rate proves to be a poor proxy for WER given Whisper's pre-trained language modelling capacity.

0.11

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

cs.CL eess.AS ·2026-10-26 ·Yizhou Peng, Chen Chen, Donghang Wu, Hexin Liu, Eng Siong Chng
benchmarkcode/data similarity 0.65
Abstract · Submitted to ICASSP 2027

Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete delays execution even when the request is already actionable. However, neither existing spoken agents nor spoken dialogue corpora define the earliest point at which a request becomes actionable. We introduce a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision, marking when the observed speech grounds a tool and its required arguments. Through controlled task generation, speech synthesis and alignment, and gated annotation, we construct 26,694 dialogues totaling 786 hours across 48 domains, with disjoint training and test argument values. We additionally create an audio rendering of $τ^2$-bench for external evaluation. We train an omni-modal language model on this corpus, enabling causal, chunk-level decisions and partial-transcript generation for downstream call completion. The model achieves tool-state macro-F1 scores of 0.48 and 0.47 on held-out dialogues and the external benchmark, respectively. For correct calls, the estimated mean lead over a turn-end cascade, using the same formatter, is 1.8-2.0 seconds in-domain and 1.1 seconds on $τ^2$-bench. These results demonstrate a data-driven path toward spoken agents that act while listening. We will open-source the pipeline and the training scripts.

0.26