CS Paper Radar

Reference-Guided Turn-Taking Style Transfer for Full-Duplex Spoken Language Models

Yuxuan Chen, Yang Xiao, Jack Sun, Chak Sun, Brick Li, Sam

cs.CL cs.AIcs.SD ·submitted 2026-10-14 18:12 UTC · v1

Abstract

Full-duplex spoken language models decide at every frame whether to take the floor, defining a conversational personality -- yet each set of weights encodes exactly one, which human speakers bracket rather than share. Existing controls mostly act from outside; none identifies the quantity that sets how much the model speaks. We identify it: a single late-layer residual-stream direction, the initiative vector, extracted from the model's own conversations without training or annotation. The same direction reads the decision, anticipating onsets up to 800 ms ahead, and shifts it: scaled during generation, it moves speech ratio from 6-12% to 42-61% (no steering: 21-30%), consistently across conversations, weights, and corpora. A short reference recording suffices to transfer a speaker's turn-taking style to the model, and the transfer is reference-specific: swapping in another speaker's reference worsens the match in 93-98% of cases, whereas the best uniform dose is indistinguishable from no steering. Turn-taking style is not frozen into the weights but a direction that can be read, shifted, and calibrated per person. Audio examples are available at https://elpsykongloo.github.io/initiative-vector/.

Comment
5 pages, 4 figures. Submitted to ICASSP 2027. Audio examples: https://elpsykongloo.github.io/initiative-vector/

Closest papers

Nearest neighbours by meaning across the whole index.

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

eess.AS cs.CL ·2026-08-25 ·Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri, Simon Michael Vuch Williams +14 more
publishedbenchmarkhighly cited similarity 0.75
Abstract · 8 pages, 2 figures. Accepted to IEEE SLT 2026. v2: camera-ready version

Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking evaluation remains limited due to the lack of a consistent, linguistically grounded evaluation protocol and hand-annotated data covering diverse conversation types. To address this, we present TurnBench, a multi-domain benchmark that pairs a 30-hour, hand-labeled corpus of dyadic human conversation with a standardized evaluation protocol for end-of-turn and interruption detection. We set conversation type as a controllable experimental variable, covering six distinct interaction styles, and triple-annotate each conversation. Benchmarking 14 heterogeneous turn-taking systems, we find end-of-turn recall stable across types, while interruption false positives are strongly type-dependent and concentrated in backchannel-dense interaction styles. Although in smooth floor transfers human listeners begin speaking a median 151 ms before the current turn ends, no current system performs equivalently without incurring excessive false positives. We release our corpus, a 104-hour training set, and a public leaderboard with an interactive dataset viewer at https://turnbench.sesame.com.

0.63

Coupled but Late: Turn-Taking Between Full-Duplex Speech Models in Unscripted Dialogue

cs.AI ·2026-10-06 ·Lichen Zhu, Yueqian Lin, Yiheng Wang, Hai "Helen" Li, Yiran Chen
similarity 0.73
Abstract · The paper is under review

Full-duplex speech models are trained to converse with a person, but they are increasingly made to converse with each other, in self-play data generation, agent societies, and model-based evaluation. In that loop no human absorbs a timing error: each model's turn-taking is the other's input. We ask what timing the loop settles into. Two PersonaPlex-7B instances exchange audio tokens on a shared clock in unscripted conversation, and one floor-transfer rule is applied to them and to Switchboard. Their timing is coupled: re-pairing speakers across conversations destroys it. But the floor changes hands late, at a median of 400-560 ms against 137 ms for humans, and the last 120 ms of the partner's turn, where human projection places a tenth of its transfers, holds 1% of theirs. Delaying one direction of the channel shifts the response one-for-one and leaves the run-up to it empty, consistent with a reactive wait after the perceived end rather than the turn-end projection human timing requires.

0.10

Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agents

cs.CL cs.SD ·2026-09-11 ·Yunqi Lu, Tyler Baumgartner, Nikhil Johri, Brandon Tai, Candice Fan +4 more
hot direction similarity 0.72
Abstract · 12 pages, 8 tables

Full-duplex evaluation often emphasizes whether an agent keeps speaking or stops. That binary cannot express a third response humans use routinely: continuing to speak while incorporating what the listener just contributed. The contribution may be a missing word, a correction or a clarification. We introduce Duplex Cue, an evaluation of this \emph{in-turn adaptation} in full-duplex voice agents. Duplex Cue separates listener intent (backchannel, collaboration, or interruption) from speaker behavior: continuing unchanged, adapting within the turn, or yielding. Adaptation includes acknowledgment as well as content revision. In a single-model case study using 300 human-confirmed cues from unscripted English conversations, we compare recorded human responses with PersonaPlex continuations generated while replaying the listener's audio. We retain 208 pairs with the ongoing speaker active at cue onset and a scorable response in each condition. On the 66 collaborative pairs, recorded speakers adapt in 68.2\% of cases, compared with 34.8\% for PersonaPlex. The model otherwise continues unchanged (42.4\%) or yields (22.7\%). These findings show why evaluating natural voice interaction requires measuring how an agent responds to a listener's contribution as well as whether it keeps speaking.

0.49

Leveraging Dual-Process Theory for Multimodal Multiparty Turn-Taking Prediction

eess.AS cs.SD ·2026-11-14 ·Rutherford A. Patamia, Ming Liu, Wei Luo, Favour Ekong, Rehan Mohammed +2 more
similarity 0.72
Abstract · 13 pages, 5 figures. Preprint submitted to IEEE Transactions on Audio, Speech and Language Processing

In multiparty conversations, a pause does not always mean a speaker has finished, and another voice does not necessarily indicate a change in who holds the conversational floor. Reliable turn-taking therefore requires recognising not only whether a speaker is ready to yield the floor, but also when a possible transition deserves attention. We propose a dual-process-inspired framework that separates continuous candidate discovery from selective verification. A fast, recall-oriented stage identifies plausible transition points. In contrast, a deliberative stage combines acoustic, linguistic, speaker-related, and predicted future-activity evidence, together with multimodal cues where available, to reject false candidates and refine transition timing. We evaluate the framework through three distinct tasks: SHIFT/HOLD classification at supplied decision points, verification of automatically generated candidates, and continuous discovery and localisation of floor transfers. On VoxConverse, we derive action-aware turn-taking events from speaker-diarization annotations to examine unconstrained conversations with varying numbers of speakers. Multi-TPC provides complementary evaluation of controlled triadic interactions using speech, gaze, motion, and prosodic information. The findings reveal that identifying plausible transitions is considerably easier than reliably confirming and localising genuine floor transfers, while the contribution of individual evidence sources varies across conversational settings. These results highlight the importance of treating multiparty turn-taking as a continuous discovery-and-verification problem rather than classification at predefined boundaries.

0.20

SpeakerTurn: Target-Speaker Semantic Turn Detection in Multi-Speaker Environments

eess.AS ·2026-10-29 ·Nan Li, Hetao Shan, Jintao Liang, Jiahao Yan, Longbiao Wang +2 more
similarity 0.71
Abstract · 6 pages, 2 figures, 3 tables

Semantic completeness alone cannot determine when an assistant should respond in a multi-speaker environment: a completed utterance may belong to an interfering speaker, while the designated user is still speaking, acknowledging, or requesting time. We therefore study semantic turn inference conditioned on speaker identity. We propose SpeakerTurn, a framework that jointly predicts the target transcript and turn state from a speech mixture and an independent target-speaker reference. Built on CocktailASR-1, it links semantic decisions to the corresponding target content through joint supervision, distinguishing complete, incomplete, backchannel, wait, and target-absent states, with a separate interruption task. We also examine whether structured chain-of-thought (CoT) supervision improves decisions by representing speaker presence and semantic cues. On a synthetic multi-speaker test set, SpeakerTurn achieves five-state macro F1 of 0.811 and target-transcription character error rate (CER) of 4.20%, correctly resolving all ten reference-swapped interruption pairs. Joint-format transcription and native-interface recognition nevertheless behave differently, and CoT improves validation transcription without improving turn macro F1. These results show why target attribution, semantic turn discrimination, and recognition retention require separate assessment when adapting speech language models for speaker-conditioned interaction.

0.09

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

cs.SD cs.AIcs.CL ·2026-10-14 ·Puneet Mathur, Dinesh Manocha
benchmark similarity 0.71
Abstract · Under review

Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel without taking it. Existing evaluations study dyadic turn-taking or end-to-end agent behavior, but lack a unified, controlled evaluation of genuine floor transitions against confusable non-events in multi-party settings. We introduce MultiPartyTurnBench (MPTB), a benchmark of 240 synthetically generated, channel-separated conversations spanning two to five speakers and six interaction types. MPTB evaluates two complementary tasks: End-of-Turn (EOT) detection distinguishes genuine turn completion from temporary holds, while Interruption (INT) detection distinguishes successful floor takeovers from non-floor-taking speech. Exact event-conditioned annotations enable causal evaluation using false-positive-constrained recall and signed latency. Across acoustic, semantic, and turn-projection baselines, we find a stark asymmetry: learned systems achieve 82.5-92.6% EOT recall at 4.3-11.8% false-positive rates, while INT detection suffers from missed takeovers or excessive false activations. This asymmetry extends to five full-duplex agents which achieve 86.1-95.8% balanced accuracy on turn entry, yet perform near chance (46.9-54.7%) on selective interruption handling. Gemini-3.8-Live yields to 97.7% of backchannels, whereas MiniCPM-o-4.5 yields to only 1.4% of genuine interruptions. These findings establish selective floor control as a distinct, unresolved challenge for reliable multi-party voice interaction.

0.04

Kezek: Splitting Mono Two-Person Conversations into Per-Speaker Tracks for Full-Duplex Dialogue Data

eess.AS cs.AI ·2026-11-12 ·Bagzhan Karl, Alizhan Karl
code/data similarity 0.70
Abstract · 18 pages, 15 tables, 2 figures. Code: https://github.com/Manifestro/kezek ; weights: https://huggingface.co/manifestro/kezek-v1 ; listening …

Full-duplex spoken dialogue models learn turn-taking, overlaps and backchannels from conversations recorded with one channel per speaker, but most available conversational audio is a single mono mix. We present Kezek, a model that turns a mono recording of exactly two people into one track per person with the original length and timing; overlaps and short replies ("yeah", "mm-hm") stay on the track of the person who said them. Kezek decides who speaks when and who is who once for the whole recording (stage 1: a two-voice activity model guided by a speaker fingerprint trained to be insensitive to phase), and a conditioned time-domain separator only carries out that decision (stage 2), with outputs that always sum to the input. Standard separation metrics miss the failures that matter for dialogue data, so we report short-reply retention, leakage onto the other person's solo, channel swaps and coverage. Against the open pipeline pyannote -> DialogueSidon on the same recordings, Kezek keeps more short replies (0.83 vs. 0.69 on Open Yap test; 0.80 vs. 0.71 over the other's speech on unseen telephone speech from CallHome English), mostly because it returns the whole recording while the external pipeline drops part of it (on the audio both return, reply retention is about the same: 0.859 vs. 0.865 on Open Yap); it also leaks less, swaps channels less often, and runs 4-5 times faster (about 230 hours of audio per GPU-hour). We also report negative results and an ideal-mask ceiling that locate the remaining gap: quiet replies under a louder voice. Code and weights are released under permissive licences.

0.50

The Importance of Speech Genre for End-of-Turn Detector Training

eess.AS cs.AI ·2026-10-29 ·Rini Sharon, Kadri Hacioglu, Jeena Prakash, Andreas Stolcke
code/data similarity 0.69
Abstract

End-of-turn (EoT) detection has been studied mostly through features, models, and modalities, while the speech genre of the training data has drawn far less attention. We isolate this factor. Using one reproducible annotation pipeline, we build a matched pair of corpora, multi-turn human-human conversational and single-turn human-machine, and train the same streaming ConvLSTM EoT detector on each. Across four test sets the conversational model reaches up to +0.45 higher F1, driven by fewer false alarms on the mid-turn pauses and hesitations that are rare in single-turn corpora. This conversational-training effect is not specific to frame-level decoding. Under a matched complete-or-not test it also appears at the utterance level, across two encoders, but only on conversational test data. Testing only on single-turn speech, as most prior work does, therefore hides the benefit of conversational training data. Against open-source baselines, our 1.8M-parameter model attains the highest F1 on every test set.

0.13