CS Paper Radar

ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation

Zesheng Wei, Mengfan Li, Wenhao Liu, Yixin Zhang, Zilei Wang, Yang Deng

cs.CL ·submitted 2026-09-10 05:26 UTC · v1
top venuebenchmarkdatasetcode/data

Abstract

Dispute mediation is essential for maintaining social harmony and resilience, yet developing skilled mediators is costly and time-consuming. Existing LLM-based mediation research remains limited by unrealistic task formulations, low-fidelity datasets, and coarse evaluation metrics that obscure turn-by-turn dynamics. To address these gaps, we introduce ProMediConv, a novel benchmarking framework that models mediation as a proactive, multi-stage, and party-aware dialogue process incorporating 11 mediation strategies and four party behavior pattern (BP) states. Using 972 complete real-world cases, we construct a high-fidelity mediation dataset with utterance-level annotations of strategies and BP states. Furthermore, to better assess agent impact, we propose MAD (Mean Attribute Difference), a fine-grained metric that captures BP shifts throughout the dialogue. Leveraging this framework, we establish a comprehensive benchmark by evaluating diverse models alongside our tailored baseline ProMediAgent. Extensive empirical analyses reveal critical behavioral phenomena and underscore the persistent challenges current models face in dynamic, multi-party mediation. Ultimately, ProMediConv provides a rigorous foundation and a vital quantitative standard for advancing AI-assisted conflict resolution. Our dataset and codebase are accessible at https://github.com/ZsWei66/ProMediConv_repo.

Comment
Accepted to Findings of EMNLP2026

Closest papers

Nearest neighbours by meaning across the whole index.

DIPLOMAT: Dialogue-Span-Aware Direct Preference Optimization for Polite Persuasive Workplace Negotiation Dialogues

cs.CL cs.AI ·2026-09-06 ·Bibhuti Jha, Rishikant Chigrupaatii, Priyanshu Priya, Asif Ekbal
top venuedataset similarity 0.64
Abstract

Effective workplace negotiation requires balancing multiple objectives, including achieving task goals, preserving professional relationships, and resolving conflicts constructively. However, misunderstandings, misaligned preferences, and interpersonal friction often impede successful outcomes. Politeness mitigates these challenges by fostering trust, reducing tension, and preventing escalation, and persuasive communication helps overcome resistance, align preferences, and guide participants toward mutually beneficial agreements. Motivated by these insights, we present DIPLOMAT, a dialogue system for polite and persuasive workplace negotiation. To support its development, we introduce PROWESS, a dataset of multi-turn workplace negotiation dialogues generated via a multi-agent framework and enriched withnegotiation strategies, politeness levels, persuasive strategies. DIPLOMAT is trained using Dialogue-Span-Aware Direct Preference Optimization (DSA-DPO), a novel preference learning objective that identifies key dialogue spans for preference alignment. This enables DIPLOMAT to generate contextually coherent responses that employ intended negotiation strategies, maintain politeness, and incorporate effective persuasion strategies throughout interactions. Automatic and human evaluation on PROWESS confirm that DIPLOMAT consistently outperforms baselines in generating coherent, polite, and persuasive negotiation responses.

0.42

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

cs.CL cs.AIcs.LGeess.AS ·2026-08-27 ·Yen-Ju Lu, Yuzhe Wang, Yaohan Guan, Xiluo He, Jiarui Hai +6 more
benchmark similarity 0.63
Abstract · 24 pages, 4 figures

Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.

0.49

LexPilot: Benchmarking the Proactive Interaction Ability of Legal LLMs with Laypeople in Chinese Civil Case Consultation

cs.AI ·2026-10-16 ·Wenbin Zhang, ShangZhan Li, Bingxiang He, Ye He, Yuxin Zhou +3 more
benchmark similarity 0.62
Abstract

Large Language Models (LLMs) have greatly assisted legal professionals. However, our empirical studies reveal that, when faced with underspecified consultations from non-experts, current LLMs often produce irresponsible responses with limited utility despite lacking sufficient case details for definitive conclusions. To better assist laypeople, legal LLMs must identify focal points of dispute and proactively elicit crucial context from users. Unfortunately, existing data resources often focus on interaction and response quality, offering limited insight into how proactive interaction affects LLMs' downstream claim-level judgment prediction in actual civil cases. To address this gap, we introduce \textbf{LexPilot}, a benchmark linking proactive legal interaction to claim-level verdict prediction using authentic Chinese civil judgments. Constructed from 432 real-world judgments across 36 common causes of action, LexPilot comprises expert-reviewed pairs of complete and underspecified consultation histories with corresponding court verdicts. Experiments reveal that current LLMs struggle with proactive legal interactions in underspecified consultations, with downstream judgment prediction substantially underperforming that in the fully specified consultation setup (-23.84% on average). Further analysis identifies critical bottlenecks hindering these capabilities. Our work highlights the urgent need for more user-centered legal AI to better serve the public.

0.16

Legal Guidance in the Wild: How Users Seek Legal Help in Real-World LLM Conversations

cs.CL cs.CY ·2026-10-18 ·Amirreza Mohseni, Gerasimos Spanakis
publishedcode/data similarity 0.62
Abstract · 12 pages, 6 tables. Accepted for publication at JURIX 2026, Toulouse, France. This is the preprint version. Code and data available at: http…

People ask conversational AI models about their legal rights, obligations, procedures, and personal situations that may have legal consequences. To study this behavior, we first need to distinguish requests for legal guidance from other legally themed discussions and unrelated conversations. We contribute a multi-axis taxonomy of legal queries capturing whether legal guidance is sought, how a request is formulated, and which legal domains it concerns; apply it to 2,122 naturally occurring WildChat-1M conversations; and create a human-reviewed gold evaluation set of 200 conversations. All annotations, model checkpoints, and evaluation code are publicly available. In this enriched sample, 71.1% of conversations containing legal content seek personalized or practical guidance. Rights-and-options advice is the most common primary request type among legal-content conversations, and 25.3% of guidance-positive conversations span multiple legal topics. The gold evaluation set supports a two-stage comparison of ModernBERT-base and ModernBERT-large against a TF-IDF cascade. Our framework supports large-scale studies of legal guidance in conversational AI, and the public release of the annotated data facilitates reproducible research and future model development.

0.43

Seeing the Forest for the Trees: Emergent Agreement from Individual Conversations

cs.CL ·2026-10-21 ·Adi Simhi, Yaniv Nikankin, Jonathan Herzig, Idan Szpektor, Yonatan Belinkov +1 more
code/data similarity 0.61
Abstract

As LLM agents interact in the wild and shape each other's decisions, understanding how they converge toward agreement becomes important. We study this in multi-turn conversations where two LLMs discuss subjective matters, can agree or disagree, and may change their stances. Using a linear classifier on individual conversation activations, we identify an internal representation of agreement. We find that it varies substantially across turns and conversations and does not reliably track external agreement at the individual conversation level. We therefore move beyond the representation of agreement in individual conversations, by aggregating the agreement representation across conversations. Externally, this aggregate view reveals a simple structure: when pooled across conversations, agreement can be predicted well with a Markov process, with agreement at each turn being a direct function of agreement at the previous turn. Similarly, when internal agreement is aggregated, it couples with the external Markov agreement through a power-law mapping, exposing Markovian dynamics in internal representations. Additionally, we find that after a change of stance, models express agreement externally before their internal representations reflect the same shift. These results reveal a joint structure between internal and external agreement that emerges only when internal features are aggregated across conversations. Code available at https://github.com/technion-cs-nlp/SeeingTheForestForTheTrees.

0.35

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

cs.CL eess.AS ·2026-10-26 ·Yizhou Peng, Chen Chen, Donghang Wu, Hexin Liu, Eng Siong Chng
benchmarkcode/data similarity 0.61
Abstract · Submitted to ICASSP 2027

Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete delays execution even when the request is already actionable. However, neither existing spoken agents nor spoken dialogue corpora define the earliest point at which a request becomes actionable. We introduce a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision, marking when the observed speech grounds a tool and its required arguments. Through controlled task generation, speech synthesis and alignment, and gated annotation, we construct 26,694 dialogues totaling 786 hours across 48 domains, with disjoint training and test argument values. We additionally create an audio rendering of $τ^2$-bench for external evaluation. We train an omni-modal language model on this corpus, enabling causal, chunk-level decisions and partial-transcript generation for downstream call completion. The model achieves tool-state macro-F1 scores of 0.48 and 0.47 on held-out dialogues and the external benchmark, respectively. For correct calls, the estimated mean lead over a turn-end cascade, using the same formatter, is 1.8-2.0 seconds in-domain and 1.1 seconds on $τ^2$-bench. These results demonstrate a data-driven path toward spoken agents that act while listening. We will open-source the pipeline and the training scripts.

0.26

Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference

cs.MA cs.CLcs.SD ·2026-09-09 ·Yaohan Guan, Yen-Ju Lu, Yuzhe Wang, Junhyeok Lee, Jesus Villalba +3 more
similarity 0.61
Abstract

Inferring speaker relationships from spoken conversations is an important step towards socially aware speech understanding. However, this task remains underexplored, and supervised modeling is costly to train and scale. At the same time, existing inference-time LLM approaches provide limited structure for handling subtle, distributed, and multimodal relational cues that may support multiple plausible interpretations. To address these limitations, we introduce a training-free multi-agent reasoning framework that organizes inference through structured interaction among LLM agents, allowing relationship judgments to be proposed, challenged, and adjudicated without task-specific training. We instantiate this framework with two complementary designs. We propose Multi-Role Multi-Agent Debate as a task-specific adaptation of standard multi-agent debate for speaker relationship inference, assigning agents complementary roles or social-theory-grounded perspectives rather than a single undifferentiated viewpoint. In contrast, we introduce Multi-Agent Compete, a competition-based protocol that compares agent judgments through pairwise adjudication, eliminates weaker candidates, and retains the most defensible one. We evaluate these methods on the Seamless Interaction dataset across different modality settings, covering both binary classification and fine-grained relationship-detail prediction. Results suggest that they improve over zero-shot and existing multi-agent baselines in most cases. Human evaluation further suggests that this task is challenging even for people. LLM methods can sometimes outperform human annotators in text-included settings but are less competitive in the audio setting. Together, these findings suggest that relationship inference benefits from structured inference-time interaction among agents, while acoustic cues are not yet fully captured by current models.

0.37

Calibrating LLM Judges for Human and AI Conversations

cs.HC ·2026-09-24 ·Maike Züfle, Patrícia Schmidtová, Vilém Zouhar, Shree Harsha Bokkahalli Satish, Erica Cooper +7 more
similarity 0.61
Abstract

Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.

0.43