CS Paper Radar

CounselPlex: A Stressor-Aware Full-Duplex Counseling Model

Yeonju Kim, Bella Godiva, Yong Man Ro

cs.CL cs.SD ·submitted 2026-10-23 14:26 UTC · v1

Abstract

Full-duplex speech models can listen and speak at the same time, enabling natural, low-latency interaction. In counseling, however, these models often focus on the latest utterance and lose track of the seeker's underlying problem. We call this failure Stressor Blindness. We propose CounselPlex, a stressor-aware full-duplex framework that uses two operations. CounselPlex updates a short stressor state during seeker silence and injects this state into <PAD> positions while listening. We also introduce RoPE Position Shift Augmentation (RPSA), which broadens positional coverage during training and improves performance at longer turn depths in our evaluation. LLM-judge evaluations show improved counseling quality and stressor grounding over strong baselines.

Comment
21 pages, 4 figures, 29 tables

Closest papers

Nearest neighbours by meaning across the whole index.

DuplexDrama: A Synthesized Dialogue Dataset with Scenarios, Full-Duplex Behaviors, Expressive Speech, and Sound Events

cs.CL ·2026-09-11 ·Qingxiang Guo, Wenke Fan, Shuofeng Zhao, Dawei Yang, Zhiyang Zhou +7 more
hot directionhighly cited similarity 0.62
Abstract · 5 pages, 5 figures, 5 tables, 19 references. Demo: https://dunjie5465.github.io/duplexdrama-demo/

We present DuplexDrama, the first synthesized spoken dialogue dataset that simultaneously covers four dimensions: (i) complete persona and scenario settings; (ii) three full-duplex behaviors (interruption, backchannel, incomplete); (iii) expressive speech with persona-aligned emotion labels; and (iv) script-aware sound events. DuplexDrama is built via a 4-stage pipeline; quality validation on both scripts and synthesized audio confirms its quality. We have produced more than 2,000 hours audio data with a 64-voice timbre pool spanning 13 personas and 5 age buckets; 3.8% of all turns carry at least one full-duplex behavior. This data has been validated through internal full-duplex model training. We will release a curated subset of 6,400 bilingual dialogues (800 h, Chinese ~500 h + English ~300 h) to advance full-duplex spoken dialogue model research. Data samples are available at our demo page and LLM-judge evaluation prompts will be released with the dataset.

0.57

DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents

cs.AI ·2026-09-03 ·Puneet Mathur, Dinesh Manocha
publishedbenchmark similarity 0.61
Abstract · Under Submission at ICLR 2027

Full-duplex voice agents must continuously decide when to speak, listen, backchannel, interrupt, overlap, and yield the conversational floor. Existing benchmarks evaluate these behaviors through explicit turn-management instructions, whereas voice agents are often configured through roles or personas from which appropriate conversational behavior must be inferred. We introduce DuplexSpeechBench-IFEval (DSB-IFEval), a benchmark for evaluating implicit instruction following in real-time spoken interaction. DSB-IFEval comprises 1,038 test cases derived from 240 controlled conversations spanning eight behaviorally contrastive assistant roles and five conditioning protocols: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona-rule conditioning, and instruction conflict. We measure real-time floor management using the deterministic Instruction Adherence Score (IAS) and persona-consistent response content using the LLM-judged Persona Adherence Score (PAS). Across eleven real-time speech models, we find that executing an explicit floor-management policy does not reliably imply the ability to infer the same policy from a persona. Moreover, even frontier models such as GPT-Live-1 and Gemini-3.8-Live adapt their dialogue language to the assigned persona without consistently translating that persona into the appropriate full-duplex floor-management behavior. Finally, models that successfully resolve benign instruction conflicts often fail when safety-relevant role behavior should override an explicit directive. These results show that inferring role-implied behavior, executing it in real time, and resolving instruction conflicts remain distinct challenges for full-duplex voice agents.

0.60

DuplexAct-Bench: Broadening Full-Duplex Speech Evaluation toward Proactive Interaction across Diverse Behavioral Requirements

cs.CL cs.HC ·2026-09-30 ·Keyue Xing, Wentao Ding, Mengmeng Wang, Wenming Tu, Zilong Zheng +1 more
benchmarkcode/data similarity 0.61
Abstract

Existing full-duplex speech benchmarks cover only subsets of real-time interaction behaviors, often under limited contextual conditions. We introduce DuplexAct-Bench, a bilingual benchmark that systematically covers six complementary behaviors, from interruption and yielding to proactive initiation, active silence, and backchanneling, across Pre-session, In-session, and No-explicit conditions. Across 1,290 English and Chinese streaming trials, we evaluate 12 full-duplex speech systems on both Timing and Content. Results reveal substantial variation across behaviors, conditions, and systems, as well as frequent mismatches between semantic quality and behavioral timing. These findings show that current systems remain far from robustly managing when, whether, and how to participate as real-time interaction unfolds. Project page: https://alitaxky.icu/DuplexAct-Bench/

0.16

Duplex-MPE: Benchmarking Multi-Party Interaction in Full-Duplex Dialogue

cs.SD eess.AS ·2026-09-25 ·Chengqian Ma, Wenhao Feng, Weixuan Jin, Gaole Dai, Tianyu Xie +5 more
benchmarkcode/data similarity 0.61
Abstract · 28 pages, 3 figures. Project page: https://step-out.github.io/Duplex-MPE-Page/. v2: adds three open-weight systems (Realtime-Venus-Audio, Mo…

Real-time full-duplex speech models can listen while speaking, enabling natural interaction without rigid turn boundaries. Existing benchmarks evaluate turn-taking, interruption handling and multi-round dialogue, but largely centre on a designated user rather than an assistant participating in a shared conversation among several people. We introduce Duplex-MPE to evaluate when such an assistant should answer, remain silent or stop speaking. The benchmark contains 2,000 scenarios with three or four human speakers and one assistant, each paired across explicit and implicit addressing of the same request. Models receive continuous conversation audio without transcripts or supplied turn boundaries. Four scores measure fresh response initiation, answer accuracy, silence preservation and stopping when a human resolves a request. We evaluate eight open-weight speech systems: MiniCPM-o 4.5, Moshi, FLM-Audio, Voila, Freeze-Omni, Realtime-Venus-Audio, Moshika-RL-Seamless and PersonaPlex. MiniCPM-o 4.5 leads on three scored capabilities, while frequent speech from other systems can coexist with inaccurate answers or failures to remain silent. A transcript-based Gemini 3.1 Pro reference responds 64.3 percentage points more often to explicit than implicit requests; paired tests detect no significant response-presence difference for the speech systems.

0.57

Diagnosing and Mitigating Long-Session Degradation in Full-Duplex Speech Models

cs.CL ·2026-10-26 ·Junho Kim, CheHyun Lee, Donghun Kang, Youngho Cho, Dongwook Lee +2 more
code/data similarity 0.60
Abstract · The project page is available at https://kimjunho333.github.io/dmlsd-fdsm/

End-to-end full-duplex spoken dialogue models enable natural, real-time spoken interactions with interruptions, backchannels, and overlapping speech. Prior evaluations have assessed diverse interaction capabilities and identified some degradation patterns across conversational turns. However, how model behavior evolves over extended sessions remains insufficiently understood. In this work, we address this gap by systematically characterizing long-session degradation and developing efficient methods to mitigate it. We introduce a closed-loop evaluation framework for long-session stability and apply it to several publicly available full-duplex speech models, revealing degradation over time in responsiveness, spoken content quality, and voice consistency. To mitigate this degradation, we propose a self-distillation-based training framework for lightweight LoRA adaptation that improves long-session stability with a small amount of additional training. We further introduce a two-resolution context (TRC) architecture that preserves recent interactions in full detail while compressing older interaction history into key-value representations. We evaluate our methods on three models with distinct architectures, assessing long-session stability, information retention, and computational efficiency. Across all three models, we show that LoRA adaptation mitigates performance degradation over ten-minute sessions, while adding TRC extends these gains beyond the training session length and reduces KV-cache growth by more than 90%.

0.20

Voice Agents under Acoustic Stress: From Signal Degradation to Interaction and Action

eess.AS cs.HC ·2026-09-24 ·Amir Ivry, Kai-Wei Chang, Lin Zhang, Sharon Gannot, Carlos Busso
similarity 0.58
Abstract · This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may…

Voice agents must complete users' tasks despite noise, reverberation, and competing speech. Evaluating agents' robustness therefore requires following how acoustic conditions affect the conversation and the actions taken on the user's behalf. This overview examines what existing benchmarks reveal about agents' ability to complete tasks under acoustic stress and where further task-based evaluation is required. We then introduce TRACE, a practical workflow for designing, running, and interpreting evaluations of acoustic robustness in task-oriented human-agent interactions: the same agent attempts a specified task with an original recording and an acoustically stressed copy, and the resulting conversations are scored for task completion, wrong actions, recovery, and user effort. Finally, we explain how results from these evaluations can guide changes to an agent to prevent wrong actions and improve recovery.

0.49

ConversationalVoice: Full-Duplex Speech Data from Real Conversations through Source-Faithful Reconstruction and Conversation-Grounded Expansion

cs.CL cs.SDeess.AS ·2026-09-08 ·Richard Yucheng He, Baodong Cao, Chen Xu, Yihang Liu, Tairan Chen
hot direction similarity 0.58
Abstract · 12 pages, 3 figures, 1 table, preprint

Full-duplex speech models require training data that preserves turn-taking, overlap, interruption, and backchannel behavior, yet these signals are entangled across speakers in noisy real-world recordings. We present Conversational Voice, a pipeline that converts real two-speaker excerpts into three complementary training-data artifacts. (1) Separation recovers speaker-specific tracks with stable speaker assignments, a canonical transcript, and naturally observed interaction timing. (2) Reconstruction generates speech in matched voices from a fixed source transcript, reconstructs the source turn order, pauses, and overlaps, and adds word-level alignment and delivery instructions. (3) Expansion generates new dialogue constrained by the source context, speakers, and observed interaction pattern. Automatic speaker-verification metrics remain strong across stages, with same-speaker similarity of 0.983-0.991 and positive discrimination margins of 0.199-0.209. Predicted speech quality (NISQA MOS) is 3.56 for separation, 4.41 for reconstruction, and 4.61 for expansion. A Gemini-based automatic evaluator assigns expansion mean scores of 4.94/5 for contextual coherence and 4.80/5 for dialogue naturalness. Expansion and reconstruction exhibit broadly similar interaction profiles; expansion's turn, overlap-event, backchannel, and interruption rates are 4.6%, 8.0%, 13.2%, and 16.0% lower, respectively. We evaluate data properties only; downstream gains in full-duplex model training remain for future work.

0.51

Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents

cs.SD cs.AIcs.CLcs.MM+1 ·2026-09-26 ·Yanjie Zhang, Nanchen Hu, Yushi Sun
benchmark similarity 0.58
Abstract

Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBench, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer. Speaker-switch pairs hold the specified words fixed while source, distance rendering, and a temporal boundary define a controlled wearer-to-bystander shift. Six raw Audio LLMs and three training-free adaptations often identify the target tool yet rarely withhold action under this shift; the highest raw switch mute rate is 14%. We then use VoxGate as a post-training case study. Supervised training mutes 91.3% of switched commands while choosing the correct tool for all nearby wearer commands and text-only controls. An exploratory GRPO stage has similar switch performance; side-talk accuracy rises from 68.4% to 70.9%, and self-talk muting from 52.0% to 60.0%. Factorized controls identify an independent source-change effect, while sensitivity to the far-field manipulation varies across acoustic renderings. The benchmark therefore measures multi-cue acoustic-context gating rather than isolated speaker identity.

0.27