CS Paper Radar

When Should a Voice Agent Speak? End-of-Turn Detection in Turkish

Ahmet Tuğrul Bayrak, Fatma Nur Korkmaz, Mustafa Sertaç Türkel, Nur Yağmur Tuncer

cs.AI cs.CL ·submitted 2026-11-19 04:26 UTC · v1

Abstract

A voice agent that waits for a fixed silence before replying either interrupts users who pause mid-utterance or responds late to users who have finished. This study examines the decision at the onset of silence, using turn ends and turn-internal pauses extracted from the Real-TurnTurk corpus of Turkish dyadic video calls. A fine-tuned Turkish TurnGPT model, a zero-shot Turkish language model and a deployed multilingual detector are compared, together with voice-activity timing and facial features, under recording-disjoint and speaker-disjoint evaluation with paired cluster bootstrap tests. On held-out recordings, the fine-tuned model separates turn ends from pauses better than both text baselines and is the best calibrated of the three. Timing features of the current speaker are similarly informative, and their combination with the fine-tuned text score gives the best timing-based results; with logistic regression, this combination also outperforms the same combination with either text baseline. In a latency simulation, adding the text score to a silence timeout shortens the mean response delay at the same interruption rate. On automatic Whisper transcripts, the fine-tuned model retains its advantage over the zero-shot model.

Closest papers

Nearest neighbours by meaning across the whole index.

TurnTurk-GPT: Text-Based Turn-Taking Prediction for Turkish

cs.CL ·2026-10-30 ·Ahmet Tuğrul Bayrak, Fatma Nur Korkmaz, Nur Yağmur Tuncer, Mustafa Sertaç Türkel
code/data similarity 0.78
Abstract

This paper presents TurnTurk-GPT, a text-based turn-taking predictor for Turkish obtained by fine-tuning a Turkish GPT-2 model with the TurnGPT objective on synthetic and real dialogue, and evaluates it on real Turkish conversation against rule baselines, a zero shot Turkish language model and a widely used multilingual end-of-turn detector. Evaluation uses word-level decision points on unpunctuated, lower-cased text approximating the output of streaming speech recognisers, recording-disjoint and speaker-disjoint folds, nested threshold selection and paired cluster bootstrap tests. TurnTurk-GPT outperforms the multilingual detector under both recording-disjoint and speaker-disjoint evaluation. Its gain over a zero-shot Turkish language model is largest for speakers seen in training, while on unseen speakers it matches this strong baseline. Removing the suffixes of the turn-final word reduces AP by up to 44%, against 2% for a sentence-internal word, locating the Turkish end-of-turn cue in verb-final morphology. Synthetic dialogue alone yields the weakest trained model, and when a speaker pauses the zero-shot and multilingual baselines remain close to chance. The model and the evaluation protocol are publicly released.

0.27

IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian Languages

cs.CL ·2026-09-25 ·Rajarshi Roy, Shobhit Banga, Jonathan Raiman, Supriya Paul, Bhaskar Singh +10 more
similarity 0.75
Abstract

Full-duplex voice agents must handle pauses, take turns, backchannel, and respond to user interruptions in real time. Full-Duplex-Bench evaluates these behaviors, but its English-only corpus and reliance on word-timestamped ASR and an English-prompted LLM judge make it difficult to extend to Indian languages. We introduce IndicFDB, which extends it to ten languages spoken in India with 12,350 samples, nearly 17 times as many as the original. We address three challenges: finding conversational events in multilingual speech, evaluating their timing without reliable word-level alignment, and judging responses across languages. We mine pause handling, turn taking, and backchanneling samples from roughly 50,000 hours of channel-separated conversations using voice activity detection (VAD), and construct human-validated synthetic user interruption samples. Language-independent VAD heuristics evaluate timing, while an open-weight transcription and translation pipeline converts responses to English for LLM ratings of relevance and quality. Across seven voice agents, commercial APIs show unexpectedly consistent behavior across languages but are either fast or robust to pauses, never both, while monolingual open full-duplex models expose further tradeoffs among backchanneling, response quality, and latency.

0.40

Real-TurnTurk: A Multimodal Turkish Corpus for Turn-Taking Prediction

cs.CL cs.AI ·2026-08-22 ·Ahmet Tuğrul Bayrak, Fatma Nur Korkmaz, Bekir Berker Türker, Mustafa Sertaç Türkel, Alper Kaplan
published similarity 0.73
Abstract · Accepted to INTCEC 2026. This is the author's pre-print version. The final authenticated version will be available through the conference pr…

Turn-taking is a basic organizational feature of human conversation and remains difficult to model in natural, synchronous dialog systems. While existing research has explored multimodal approaches and large language models for turn-ending prediction, there is a lack of naturalistic conversational corpora specifically addressing turn-taking dynamics in Turkish. This study introduces a multimodal Turkish conversational dataset of unscripted dyadic interactions, comprising synchronized front-facing video, per-speaker audio channels that allow overlapping speech to be attributed to individual speakers, and time-aligned transcriptions. Turn-taking prediction is formulated as a binary classification problem, and a Genetic Algorithm (GA) is employed to optimize interpretable decision rules derived from visual, acoustic, and linguistic features. A hybrid AND-OR rule representation is adopted in the proposed framework to represent the alternative cue combinations that precede a turn transition.

0.65

The Importance of Speech Genre for End-of-Turn Detector Training

eess.AS cs.AI ·2026-10-29 ·Rini Sharon, Kadri Hacioglu, Jeena Prakash, Andreas Stolcke
code/data similarity 0.71
Abstract

End-of-turn (EoT) detection has been studied mostly through features, models, and modalities, while the speech genre of the training data has drawn far less attention. We isolate this factor. Using one reproducible annotation pipeline, we build a matched pair of corpora, multi-turn human-human conversational and single-turn human-machine, and train the same streaming ConvLSTM EoT detector on each. Across four test sets the conversational model reaches up to +0.45 higher F1, driven by fewer false alarms on the mid-turn pauses and hesitations that are rare in single-turn corpora. This conversational-training effect is not specific to frame-level decoding. Under a matched complete-or-not test it also appears at the utterance level, across two encoders, but only on conversational test data. Testing only on single-turn speech, as most prior work does, therefore hides the benefit of conversational training data. Against open-source baselines, our 1.8M-parameter model attains the highest F1 on every test set.

0.13

TamilEOT: A Dataset and Model for Semantic End-of-Turn Detection in Tamil Telephone Speech

cs.CL cs.AIcs.SD ·2026-09-04 ·Santhoshkumar V
code/data similarity 0.69
Abstract · 14 pages, 7 tables. Code, data and models: https://github.com/santhosh-005/tamil-eot

A voice agent must decide, at each pause, whether the speaker has finished or is still thinking. Without a model of the language, systems fall back to a fixed silence timeout: too short, and the agent interrupts; too long, and every turn pays the full wait. We release TamilEOT, to our knowledge the first open dataset and detector for semantic end-of-turn detection in a South Indian language: 18,485 labelled turn boundaries from 116 real Tamil telephone conversations, two audio-only detectors built on Smart Turn v3's architecture, and a turn-level companion set of 4,774 whole turns with their pause structure. On a held-out split of 4,168 clips from 30 unseen calls, accuracy rises from 70.30% zero-shot to 83.71% (8.7 MB) and 86.13% (21 MB); ROC-AUC rises from 0.751 to 0.921. Both run in under 150 ms single-threaded on a laptop CPU. We also report what building it cost. Our rule-derived negative-class labels scored 44.4% against a blind human pass, below chance, because the rule answered a different question than the model is asked; an audio-LLM labeller at 97.5% agreement replaced them for US$5.69. Of six training levers, only encoder capacity moved the result: three identical runs spanned 0.87 accuracy points, the floor below which nothing else here is a finding. Replaying labelled boundaries through the production VAD and streaming adapter costs a further 2.60 points, and 7.8% of boundaries never reach the model at all. Data, weights, code and every negative result are public.

0.39

From Metrics to Natural Dialogue: French Full-Duplex Benchmark for Spoken Dialogue Models

eess.AS ·2026-09-09 ·Hamid Soltani, Gilles Boulianne
top venuebenchmark similarity 0.68
Abstract · 9 pages, 0 figures. Accepted for publication in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP): I…

Full-duplex spoken dialogue models aim to make voice agents more natural by allowing them to listen, speak, pause, and respond during ongoing conversation. However, it is not clear whether full-duplex benchmarks behave the same way when models are evaluated in a different language. To investigate this, we introduce a French full-duplex benchmark (FDB) with two variants, CALLFC-FDB for Canadian French and MEDIA-FDB for European French, and compare them with an English FDB. Built from real spoken resources, these benchmarks evaluate key full-duplex skills, including pause handling, turn-taking, backchannels, and interruptions. Beyond introducing French FDBs, we evaluate human--human conversations with FDB metrics to better understand the values these metrics take in real-world dialogue. Our analysis reveals that most timing-based metrics behave similarly across languages, while content-based evaluation degrades under language mismatch. We also find a trade-off between optimizing benchmark metrics and preserving conversational naturalness.

0.67

People Making Conversation: Evaluating How Naturally Speech Models Converse

cs.CL ·2026-10-08 ·Arya Bastani, Iman Modarressi
published similarity 0.67
Abstract · Accepted at the NeurIPS 2026 Workshop on Real-Time Conversational Agents

A conversational speech model has to decide, at every pause, whether to speak. Existing evaluations test boundary detection, policy adherence, or aggregate statistics, not this decision. We place real-time speech systems at 427 pauses and 482 turn ends from 176 unscripted conversations in Seamless Interaction, give each the conversation so far, and score what it generates against what the person did: whether and how soon it speaks at a pause; its latency, length, and wording at a turn end; and whether it can answer a question about the preceding 30 to 120 seconds. Across six systems, the hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause whichever endpointer they use; only their latency, about 1 to 4 s with silence-based endpointers and about 8 s with semantic turn detection, decides whether the words land inside the pause. The full-duplex models (PersonaPlex, Moshi) are the only systems that regularly stay silent and the only ones that interrupt within a human reaction time. Hosted systems reply longer than the person. Given the history as text, hosted systems recall 41 to 60% of conversation facts under a Claude Opus 5.5 judge and 19 to 26% under a strict word matcher; hearing it as audio, full-duplex models recall 17 to 22% and 12 to 15%. We release the labels, points, harness, and every output.

0.14

Does the Voice Matter? Evaluating Frustration Detection on Deployed Clinical Voice-Agent Calls

cs.CL cs.AI ·2026-11-16 ·Annika Michael, Zachary Ellis, Yuxiang Zhou, Samuel Thio, Yajie Vera He +2 more
published similarity 0.67
Abstract · Extended version of papers accepted at ML4H 2026 (Proceedings) and GenAI4H workshop @ NeurIPS 2026

Autonomous voice agents now conduct routine clinical follow-up calls, so some automated triage is unavoidable. A patient becoming frustrated is one thing that triage should catch. These systems typically read only the transcript the speech-to-text stage produces, not the voice, and whether that is enough is largely untested on real clinical speech. We study per-turn frustration detection on real patient calls from a deployed clinical voice agent: 150 turns from six calls labelled by three annotators, with a length-matched acted corpus as a control, and 714 turns from 30 calls. Patient turns are overwhelmingly short: across the deployed calls, roughly 59% are one or two words, against 13% in the full acted corpus, and a corpus accuracy gap remains even after the acted set is length-matched. On one-word turns, transcript-only detection does not exclude chance. Across five audio-capable language models, the transcript flags 3 or 4 of the 39 frustrated one-word turns in the 30 calls, and a classifier given only the word count reaches 57.3%. Whether audio helps depends on the model and on the instruction. Asked to attend to delivery, two Gemini 2.5 models gain 3.0 pp over the transcript across 30 calls (-1.0 to 7.5 pp). Of the turns they confidently clear, 6.9% were in fact frustrated for audio against 13.0% for the transcript, a difference of 6.1 pp (2.2 to 10.4 pp). Across the 30 calls, given one prompt identical for both inputs, no model's audio-text interval excludes zero. Under the input-specific prompts, a newer Gemini model and two smaller open models do worse with audio than with the transcript (-3.5, -8.6 and -10.6 pp). A WavLM probe trained on the clinical calls matches the best prompted model overall (71.6%) but misses every frustrated one-word turn in the 24 added calls. When a patient's turn is short, the transcript alone is not enough to rule out frustration.

0.25