CS Paper Radar

Real-TurnTurk: A Multimodal Turkish Corpus for Turn-Taking Prediction

Ahmet Tuğrul Bayrak, Fatma Nur Korkmaz, Bekir Berker Türker, Mustafa Sertaç Türkel, Alper Kaplan

cs.CL cs.AI ·submitted 2026-08-22 18:29 UTC · v1
published

Abstract

Turn-taking is a basic organizational feature of human conversation and remains difficult to model in natural, synchronous dialog systems. While existing research has explored multimodal approaches and large language models for turn-ending prediction, there is a lack of naturalistic conversational corpora specifically addressing turn-taking dynamics in Turkish. This study introduces a multimodal Turkish conversational dataset of unscripted dyadic interactions, comprising synchronized front-facing video, per-speaker audio channels that allow overlapping speech to be attributed to individual speakers, and time-aligned transcriptions. Turn-taking prediction is formulated as a binary classification problem, and a Genetic Algorithm (GA) is employed to optimize interpretable decision rules derived from visual, acoustic, and linguistic features. A hybrid AND-OR rule representation is adopted in the proposed framework to represent the alternative cue combinations that precede a turn transition.

Comment
Accepted to INTCEC 2026. This is the author's pre-print version. The final authenticated version will be available through the conference proceedings

Closest papers

Nearest neighbours by meaning across the whole index.

TurnTurk-GPT: Text-Based Turn-Taking Prediction for Turkish

cs.CL ·2026-10-30 ·Ahmet Tuğrul Bayrak, Fatma Nur Korkmaz, Nur Yağmur Tuncer, Mustafa Sertaç Türkel
code/data similarity 0.77
Abstract

This paper presents TurnTurk-GPT, a text-based turn-taking predictor for Turkish obtained by fine-tuning a Turkish GPT-2 model with the TurnGPT objective on synthetic and real dialogue, and evaluates it on real Turkish conversation against rule baselines, a zero shot Turkish language model and a widely used multilingual end-of-turn detector. Evaluation uses word-level decision points on unpunctuated, lower-cased text approximating the output of streaming speech recognisers, recording-disjoint and speaker-disjoint folds, nested threshold selection and paired cluster bootstrap tests. TurnTurk-GPT outperforms the multilingual detector under both recording-disjoint and speaker-disjoint evaluation. Its gain over a zero-shot Turkish language model is largest for speakers seen in training, while on unseen speakers it matches this strong baseline. Removing the suffixes of the turn-final word reduces AP by up to 44%, against 2% for a sentence-internal word, locating the Turkish end-of-turn cue in verb-final morphology. Synthetic dialogue alone yields the weakest trained model, and when a speaker pauses the zero-shot and multilingual baselines remain close to chance. The model and the evaluation protocol are publicly released.

0.27

When Should a Voice Agent Speak? End-of-Turn Detection in Turkish

cs.AI cs.CL ·2026-11-19 ·Ahmet Tuğrul Bayrak, Fatma Nur Korkmaz, Mustafa Sertaç Türkel, Nur Yağmur Tuncer
similarity 0.73
Abstract

A voice agent that waits for a fixed silence before replying either interrupts users who pause mid-utterance or responds late to users who have finished. This study examines the decision at the onset of silence, using turn ends and turn-internal pauses extracted from the Real-TurnTurk corpus of Turkish dyadic video calls. A fine-tuned Turkish TurnGPT model, a zero-shot Turkish language model and a deployed multilingual detector are compared, together with voice-activity timing and facial features, under recording-disjoint and speaker-disjoint evaluation with paired cluster bootstrap tests. On held-out recordings, the fine-tuned model separates turn ends from pauses better than both text baselines and is the best calibrated of the three. Timing features of the current speaker are similarly informative, and their combination with the fine-tuned text score gives the best timing-based results; with logistic regression, this combination also outperforms the same combination with either text baseline. In a latency simulation, adding the text score to a silence timeout shortens the mean response delay at the same interruption rate. On automatic Whisper transcripts, the fine-tuned model retains its advantage over the zero-shot model.

0.04

Leveraging Dual-Process Theory for Multimodal Multiparty Turn-Taking Prediction

eess.AS cs.SD ·2026-11-14 ·Rutherford A. Patamia, Ming Liu, Wei Luo, Favour Ekong, Rehan Mohammed +2 more
similarity 0.70
Abstract · 13 pages, 5 figures. Preprint submitted to IEEE Transactions on Audio, Speech and Language Processing

In multiparty conversations, a pause does not always mean a speaker has finished, and another voice does not necessarily indicate a change in who holds the conversational floor. Reliable turn-taking therefore requires recognising not only whether a speaker is ready to yield the floor, but also when a possible transition deserves attention. We propose a dual-process-inspired framework that separates continuous candidate discovery from selective verification. A fast, recall-oriented stage identifies plausible transition points. In contrast, a deliberative stage combines acoustic, linguistic, speaker-related, and predicted future-activity evidence, together with multimodal cues where available, to reject false candidates and refine transition timing. We evaluate the framework through three distinct tasks: SHIFT/HOLD classification at supplied decision points, verification of automatically generated candidates, and continuous discovery and localisation of floor transfers. On VoxConverse, we derive action-aware turn-taking events from speaker-diarization annotations to examine unconstrained conversations with varying numbers of speakers. Multi-TPC provides complementary evaluation of controlled triadic interactions using speech, gaze, motion, and prosodic information. The findings reveal that identifying plausible transitions is considerably easier than reliably confirming and localising genuine floor transfers, while the contribution of individual evidence sources varies across conversational settings. These results highlight the importance of treating multiparty turn-taking as a continuous discovery-and-verification problem rather than classification at predefined boundaries.

0.19

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

eess.AS cs.CL ·2026-08-25 ·Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri, Simon Michael Vuch Williams +14 more
publishedbenchmarkhighly cited similarity 0.69
Abstract · 8 pages, 2 figures. Accepted to IEEE SLT 2026. v2: camera-ready version

Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited due to the lack of a consistent, linguistically grounded evaluation protocol and hand-annotated data covering diverse conversation types. To address this, we present TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection. We set conversation type as a controllable experimental variable, covering six distinct interaction styles, and triple-annotate each conversation. Benchmarking 14 heterogeneous turn-taking systems, we find end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in backchannel-dense interaction styles. Although in smooth floor transfers human listeners begin speaking a median 151 ms before the current turn ends, no current system performs equivalently without incurring excessive false positives. We release our corpus, a 104-hour training set, and a public leaderboard with an interactive dataset viewer at https://turnbench.sesame.com.

0.63

Less can be More: What Aspects of Speech Drive End-of-Turn Detection

eess.AS cs.AIcs.SD ·2026-09-10 ·Rini Sharon, Manickavela A, Kadri Hacioglu, Andreas Stolcke
top venuehot direction similarity 0.68
Abstract

In conversational AI, detecting when a speaker has finished talking is crucial for natural turn taking. While recent work incorporates semantics, the relative contribution of different modalities remains unclear. We present a controlled ablation of acoustic, prosodic, and semantic signals for streaming end of turn detection using a lightweight trimodal classifier. Under identical training conditions, the acoustic prosodic combination achieves the best balance of accuracy and latency, achieving utterance F1 of 0.93 with 7.8% false alarms at 400ms median latency. Adding text increases premature detections without improving performance. Feature space analysis confirms that prosodic features have the strongest class separability, while text representations overlap substantially. These findings suggest that turn-taking is primarily conveyed through intonation and silence patterns rather than semantic completeness, enabling faster and more reliable systems without expensive text inference.

0.42

Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness

cs.CL cs.SD ·2026-08-28 ·Mark Dourado, Karim Haddad, Henrik G. Hassager, Stefania Serafin
similarity 0.67
Abstract · Part of an industrial PhD collaboration between GN Group and Aalborg University

Smooth speaker transitions are fundamental to effective conversation and rely on an interlocutor's ability to predict when to enter the conversation. This ability depends on accurately interpreting and expressing the verbal and non-verbal cues that signal when a speaker wishes to take or relinquish the floor. The process becomes even more complex in noisy, natural, multi-party settings, with multiple interlocutors available. This study models how gaze and speech, together with perceived interpersonal closeness, signal conversational floor changes in free four-person dialogue. Using the GaMMA corpus, we trained logistic regression models using interpretable, behaviourally motivated features extracted before each turn-taking event to classify floor-transfer outcomes as gaps or overlaps. Predictors included gaze features such as transition motifs and behavioural contrasts, entropy, gaze-based addressee identity, and mutual gaze, alongside speech features derived from speaker loudness, as well as perceived interpersonal closeness (IOS) between speakers. Results show that gaze features capture predictive structure, and that combining them with loudness improves performance (ROC AUC = 0.76 +- 0.04). Loudness reflected speaker control, while gaze dispersion and addressing indexed listener readiness and competitive entry. Performance remained robust across noise conditions, indicating that gaze provides a complementary, noise-resilient cue to turn-taking dynamics.

0.17

AVCC: Audio-Visual Conversation Corpus for Turn-Taking Prediction with a Human Benchmark

cs.CL cs.HCcs.SD ·2026-11-10 ·Haotian Qi, Gabriel Skantze
benchmarkcode/data similarity 0.67
Abstract

} Predicting who will speak next, and when, is a prerequisite for conversational agents and social robots that take turns without waiting for silence. Multiparty next-speaker prediction (NSP) has been studied on corpora with per-speaker sensors, each with its own event definitions. No public benchmark covers the single camera and microphone a robot has, and no model has been compared against how well people perform the same prediction. We turn AVCC, a 31-hour audio-visual corpus of two- and three-speaker online conversation recorded with a single static camera and microphone, into a public NSP benchmark. We add a speaker-disjoint seven-video test partition with 865 hold and 865 shift events at mutual-silence gaps, word-level forced-aligned transcripts for the whole corpus, and a human reference benchmark collected at the same events. We score an audio-only Role-Relative VAP model (RoleVAP) and the multimodal MuVAP model. Labels, transcripts, per-response human data, and the scripts that reproduce every number are released.

0.22

The Importance of Speech Genre for End-of-Turn Detector Training

eess.AS cs.AI ·2026-10-29 ·Rini Sharon, Kadri Hacioglu, Jeena Prakash, Andreas Stolcke
code/data similarity 0.67
Abstract

End-of-turn (EoT) detection has been studied mostly through features, models, and modalities, while the speech genre of the training data has drawn far less attention. We isolate this factor. Using one reproducible annotation pipeline, we build a matched pair of corpora, multi-turn human-human conversational and single-turn human-machine, and train the same streaming ConvLSTM EoT detector on each. Across four test sets the conversational model reaches up to +0.45 higher F1, driven by fewer false alarms on the mid-turn pauses and hesitations that are rare in single-turn corpora. This conversational-training effect is not specific to frame-level decoding. Under a matched complete-or-not test it also appears at the utterance level, across two encoders, but only on conversational test data. Testing only on single-turn speech, as most prior work does, therefore hides the benefit of conversational training data. Against open-source baselines, our 1.8M-parameter model attains the highest F1 on every test set.

0.13