CS Paper Radar

On the possibilities and limitations of LLM-based multi-agent systems for multi-criteria decision-making in healthcare operations management

Marco Boresta, Tommaso Giovannelli, Sepehr Namirad, Giulia Riccardi

cs.AI math.OCcs.MA ·submitted 2026-10-16 16:05 UTC · v1

Abstract

Multi-criteria models for healthcare operations need weights that express how stakeholders trade off patient waiting time, provider workload, and operating cost, but these weights are difficult to quantify. Teams of large language model (LLM) agents, each prompted to represent a stakeholder, have been used to supply them. We propose a framework for assessing the reliability of LLM-generated preferences as inputs to multi-objective optimization and quantifying how their uncertainty affects decisions. We study explicit preferences by asking agents to rate the importance of the criteria in an outpatient appointment scheduling problem, and implicit preferences by inferring trade-offs from the agents' pairwise choices between emergency department configurations. The computational experiments show that the explicit preferences respond to the clinic's operating conditions and differ across roles, but deliberation among the agents can erase these differences, whereas simple averaging preserves them. The pairwise choices used to reveal the agents' implicit preferences depend largely on which option is presented first, and we find limited evidence that a statistical model using contextual information alongside presentation order predicts the agents' choices on new comparison pairs better than a model using presentation order alone. We then formulate robust weighted-sum optimization models to account for preference uncertainty. In most cases, these models select different schedules or configurations from those obtained using average preference weights. These results indicate that LLM-generated preferences need an audit of this kind before they are used as optimization weights in healthcare decision support.

Closest papers

Nearest neighbours by meaning across the whole index.

Reliability Stress Tests and Decision-Time Routing for Chest X-ray Vision-Language Models

eess.IV cs.CV ·2026-10-01 ·Xinye Yang, Zhusi Zhong, Scott Collins, Grayson Baird, Xuyu Wang +1 more
publishedbenchmarkcode/data similarity 0.62
Abstract · 6 pages, 4 figures, 5 tables. Accepted version of a workshop paper presented orally at the 2026 IEEE/ACM Conference on Connected Health: App…

Medical vision-language model (VLM) evaluation is sensitive to workflow design, prompting strategy, and benchmark construction, yet most studies treat these factors in isolation. We introduce a reliability stress test for chest X-ray interpretation built on two balanced datasets (a private report-backed set and a curated MIMIC subset). Three medical VLMs (CheXagent, MedGemma-4B, and MedGemma-27B) are evaluated across three prompt styles and two workflows (single-VLM and multi-agent), producing 36 configurations. We show that exact-match accuracy alone can overstate the effectiveness of conservative models that default to "Normal" predictions. Diagnostic reliability also depends heavily on model family and scale: multi-agent reasoning helps some configurations but hurts others. Building on these observations, we propose a decision-time routing framework that selectively escalates to multi-agent inference only when beneficial, improving the cost-quality trade-off over fixed workflows. Our results highlight the need for evaluation protocols that jointly consider prompt sensitivity, failure-mode diversity, and workflow choice before clinical deployment.

0.70

Ask, Relax, or Act? Evaluating Actionable Indeterminacy in LLM Preference Reasoning

cs.CL cs.AI ·2026-10-02 ·Ang Li, Yue Lin, Feifei Kou, Zhan Su, Prayag Tiwari +4 more
benchmark similarity 0.61
Abstract · 55 pages, 5 figures

An LLM agent can recognize uncertainty yet still choose the wrong next step: asking when action is already justified, or seeking clarification when the constraints must change. We formalize actionable indeterminacy: act when an accepted action is shared across all admissible preferences or objectives, clarify when each possibility is feasible but no action is shared, and propose a minimum-cost permitted constraint repair when the request is infeasible. We construct a solver-grounded benchmark spanning object allocation, meeting scheduling, apartment choice, and stable matching. Matched pairs retain the same source while changing whether intervention is necessary, and evaluation separates decision correctness, matched-pair reliability, and fully correct responses. Our findings reveal a recurring difficulty in recognizing when intervention is unnecessary: models can identify situations requiring clarification or repair yet still intervene when a justified action already exists. Correct decision labels also fail to guarantee usable actions, questions, or repairs. Crucially, response requirements shape not only how decisions are expressed but also which decisions are made. Making the required content explicit substantially improves fully correct responses and can change intervention decisions, even when outputs are already parseable. These findings highlight that reliable agency requires more than recognizing uncertainty: it requires intervening only when necessary and translating the chosen next step into a verifiable response.

0.41

When Is Smaller Greener? Right-Sizing Language Models for Agentic AI

cs.AI cs.MA ·2026-10-14 ·Sajid Siraj, Maria Barbati
similarity 0.60
Abstract

AI agents use a large language model (LLM) to plan and act over many steps. Which model should an agent use, so that its tasks are completed reliably with the least energy, cost and time? We treat this as a multi-criteria decision problem in which comparisons per call can mislead: a task succeeds only if all its steps succeed, and failed attempts still use resources. We therefore compare models on resources per successfully completed task. We derive crossover task lengths at which two models exchange their efficiency ranking, determined by their per-step reliability and resource use. Each objective has its own crossover. Below the shortest crossover the less reliable model dominates, above the longest the more reliable one dominates, and in between both are Pareto-efficient and preferences decide. This supports a two-stage framework: a preference-free dominance screen, followed by a multi-criteria ranking of the efficient models, subject to reliability and feasibility constraints. We also give conditions under which this structure holds when reliability and resource use change during a task. In a pilot with 15 open-weight models on two synthetic agent tasks, with measured GPU energy, the predicted crossover fell within the observed range for 72 of the 88 observed crossovers. At 20 steps, a 3-billion-parameter model used a tenth of the energy of a 27-billion one per completed task; at 100 steps, a 1.7-billion model used almost four times as much as an 8-billion one. Smaller is greener only up to a task length that can be predicted.

0.48

Learning When Human Decisions Matter: Executor-Relative Consultation Value for LLM Agents

cs.CL cs.AI ·2026-11-04 ·Shuli Jiang, Khushbu Pahwa, Yuting Zhang, Yi Zhang, Wei Xia +1 more
similarity 0.59
Abstract · 31 pages, 20 tables, 2 figures

Large language models increasingly automate complex tasks, but human decision capacity does not scale with their capabilities. In many applications, humans remain essential for providing private context, authority, or judgments unavailable to the agent. We formulate human decision capacity allocation: given multiple consultation opportunities and a limited budget, at which decisions should an LLM agent consult a human? The central challenge is estimating consultation value, which is executor-dependent because the same human input may benefit different downstream agents differently. We introduce Consultation Value Distillation (CVD), which estimates consultation values offline using Monte Carlo executor rollouts and distills these values into lightweight predictors for budgeted allocation, requiring no rollout-based value estimation at deployment. Across five batch and three sequential settings with two frontier executors, CVD improves allocation through joint and continuation-aware value learning, while independent consultation values remain competitive in several settings. CVD also broadly outperforms direct prompting and Value of Information where applicable, remains robust with substantially less rollout supervision, and enables nearly lossless approximate allocation at low search cost.

0.16

TandemBench: Benchmarking Decision Models for Multi-Agent Collaboration

cs.CL ·2026-11-04 ·Baohang Li, Xiaocheng Feng, Bing Qin
benchmarkcode/data similarity 0.58
Abstract

Decision models choose among predefined options in tasks such as routing and action selection. In multi-agent collaboration, a decision determines how members divide work, exchange resources, and act in parallel. We introduce TandemBench, a benchmark of 1,200 questions in multi-agent cooking scenarios where coordination choices affect goal completion. Models receive complete global information and candidate team plans, with reference answers established through execution and exact search. Models select an optimal team plan and, in paired variants that retain or remove it, judge whether better arrangements exist beyond the supplied options. We evaluate 40 native decision models and compare them with 10 general-purpose language models. The best native decision model achieves 62.50% decision accuracy. Native decision models remain close to chance in optimality verification. Paired tests reveal that some report better alternatives regardless of whether an optimal plan is already present. These findings show that team-plan decision accuracy alone can conceal systematic errors in judging whether the supplied options contain an optimal arrangement. The dataset and evaluation code are publicly available at https://github.com/lililibaohang/TandemBench.

0.29

Role-Specialized Mixture-of-Agents with Open-Weight LLMs for Clinical Prediction

cs.AI cs.LG ·2026-08-23 ·Jun Hou, Yi Fang, Xuan Wang
similarity 0.58
Abstract · COLM 2026 DAIH Workshop

Large Language Models (LLMs) are increasingly applied to clinical prediction tasks such as in-hospital mortality and readmission from electronic health records (EHRs). Privacy and compliance constraints motivate systems that can be deployed locally, which has increased interest in open-weight multi-agent designs. However, most medical multi-agent systems are evaluated as a single block, leaving unclear which agent role contributes to prediction and whether retrieval drives observed gains. We study a role-specialized Mixture-of-Agents (MoA) that combines medical knowledge retrieval with contrastive similar-patient reasoning. By varying the role design while holding the retrieval setup fixed, we localize the main effect to the final integrator. Pairing large open-weight analysts with a small open-weight integrator matches closed-model prompting on F1 for mortality prediction while flagging substantially more true high-risk patients. Mechanism analysis shows the role assignment directly yields a high-recall operating point without threshold tuning. The effect is task-dependent, with smaller gains for readmission because the available records correlate weakly with this longer-horizon outcome. These results position role design as a key factor in privacy-constrained, training-free clinical LLM prediction.

0.26

From Questions to Decisions: Benchmarking How LLM Agents Acquire Decision-Relevant Information from Human Beings

cs.AI ·2026-11-09 ·Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui +7 more
benchmark similarity 0.58
Abstract

LLM agents are increasingly asked to make plans for people, such as scheduling drivers or assigning nurses. However, people's requests are often incomplete, and the supporting information for decision-making may be outdated, requiring agents to ask for clarification. Existing benchmarks typically provide fully specified problems or rely on uncalibrated LLM-simulated users, leaving this ability insufficiently tested. We introduce Ask2Decide, a benchmark of 43 planning tasks that evaluates whether agents ask for missing information that matters for their decisions. An operator holds facts absent from the agent's request and records, and optimization solvers determine which facts change the optimal plan. We also include similarly worded facts that leave the optimal plan unchanged to test whether agents distinguish useful information from irrelevant details. We collect 79 interaction sessions from 19 participants and find that, after an incorrect plan, human replies omit the correcting fact far more often than replies from an uncalibrated simulated user (43% versus 7%). Using these interactions, we calibrate a simulated operator that more closely matches human reply patterns. Evaluating four frontier LLMs, we find that their initial plans are infeasible on 58-98% of tasks when information is missing, compared with 5-28% when all facts are provided upfront. Agents uncover facts that change the optimal plan only slightly more often than facts that leave it unchanged (39-52% versus 32-41%). These results show that obtaining decision-relevant information through interaction remains a major bottleneck for LLM planning. This is ongoing work and will be updated. The date of this version is shown on the first page.

0.15

A Multi-Agent LLM Framework for Personalized Health Checkup Interpretation and Guidance

cs.AI cs.MA ·2026-10-01 ·HyungJun Kim, Taehan Lee, Soojin Cheon
similarity 0.57
Abstract · 16 pages, 2 figures, 8 tables and Appendix

Personalized interpretation of health checkup results requires reasoning across longitudinal records, medical knowledge, lifestyle guidance, and healthcare navigation. We present a multi-agent large language model (LLM) system that identifies multiple intents, maps each to a task-specific agent, executes them in parallel, and synthesizes their outputs. We compared answers generated in Single Agent and Multi Agent settings on 120 Korean compound queries combining two to four requirements, using synthetic health checkup records. The Multi Agent improved the weighted LLM-judge score from 1.695 to 1.797 (p = 0.027), and three additional LLM judges showed consistent improvements ($Δ$ = +0.111 to +0.186, all p < 0.05). The gains came from usefulness, consistency, and the handling of every requirement in compound queries, whereas numerical accuracy and grounding improved significantly under only one of the four judges and medical safety did not differ, and critical failures occurred at similar rates (Single Agent 15.0% vs. Multi Agent 13.3%). Two human evaluators preferred Multi Agent in 66.7% and 68.3% of pairwise comparisons. Multi Agent execution increased latency and cost by 1.31$\times$ and 2.02$\times$, respectively. In exploratory subgroup analyses, the improvement was concentrated in queries involving personal-record lookup.

0.23