CS Paper Radar

A Dynamic Inventory Rationing Model for Last-Mile Delivery of Medication in Liberia's Community Health Program

Baris Ata, Yufan Chen, Robert Montgomery

cs.LG math.OCeess.SYmath.AP ·submitted 2026-10-23 01:43 UTC · v1
benchmark

Abstract

We study the dynamic allocation of scarce medications in Liberia's community health program. A central warehouse supplies a health facility and the Community Health Assistants (CHAs) it supervises. When a CHA runs out of medication, patients are referred to the facility, incurring travel costs if they complete the referral and remain untreated otherwise. Allocating inventory to CHAs allows more patients to receive treatment without traveling, while retaining inventory centrally preserves flexibility to respond to uncertain demand. The allocation problem must balance these considerations while accounting for fairness across locations. We formulate this problem as a finite-horizon stochastic control problem whose objective combines the costs of referrals and untreated patients with a demand-weighted fairness utility. Because standard dynamic programming methods are computationally intractable in this setting, we develop a continuous-time approximation that involves controlling the drift rate of a reflected Brownian motion. We then adapt the deep splitting method, using deep neural networks to approximate the value function and its gradient and compute a dynamic allocation policy. We calibrate the model using data on malaria cases among children in the Fangonda region of Lofa County, Liberia, and evaluate the proposed policy in the original discrete-time model. In the baseline setting, our policy reduces operating costs by about 6% relative to the strongest benchmark, the fairness-only policy, while retaining approximately 98% of its fairness utility. Additional experiments show that higher referral costs shift inventory toward CHAs and reduce local unmet demand, while greater demand variability increases our policy's performance over the fairness-only benchmark. Our model and computational approach provide a framework for studying similar last-mile allocation problems.

Comment
56 pages, 8 figures, 8 tables, including appendices

Closest papers

Nearest neighbours by meaning across the whole index.

Dynamic Skill-Based Routing for Call Centers with Transfers: A Value-Based Framework

eess.SY ·2026-10-18 ·Baris Ata, Ebru Kasikaralar
benchmarkcode/data similarity 0.61
Abstract

We consider a multi-class, multi-pool queueing model of a telephone call center in which a service completion does not always end the call. The caller may be transferred to another agent for a second service, so the assignment of an agent to a call affects not only the caller's waiting time but also whether the call is resolved and how much further work it creates. We model these transfers as Markovian routing from original to rework classes, and the system manager dynamically assigns agent pools to classes so as to maximize the value generated by service, net of transfer penalties and waiting and abandonment costs, over a finite horizon. Focusing on the Halfin-Whitt heavy-traffic regime, we derive an approximating Brownian control problem. Its Hamilton-Jacobi-Bellman equation is a semilinear partial differential equation whose dimension equals the number of classes. Building on the deep splitting method of Beck et al. (2021), we develop a simulation-based computational method that approximates the value function and its gradient by neural networks, and we use it to construct an implementable routing policy for the original (prelimit) system. We calibrate the model to publicly available data from a large U.S. bank call center. In all test problems we consider, the proposed policy outperforms the best available benchmark policy we could find.

0.34

Learning a patient world model for health system simulation in intensive care

cs.AI ·2026-11-05 ·Yingtai Li, Ruoyan Wang, Yu Li, Shaohua Kevin Zhou
similarity 0.58
Abstract

Designing healthcare strategies requires anticipating their clinical and operational consequences before implementation, a task that often relies on health system simulation. Such a simulation necessitates modeling how patient states evolve and generate care demands. However, conventional approaches typically tailor compact state transitions to a single policy question and omit history-dependent dynamics, thus requiring the redefinition of states and transitions when there is a new question that entails different clinical information. We propose to learn a patient world model from longitudinal EHRs using a neural network to provide a shared clinical representation of patient trajectories that can serve simulation questions across scenarios. Taking ICU operations as an example, we train the model on MIMIC-IV, freeze it, and couple its time-queryable predictions to shared ICU calendars, resources, and queues via scenario-specific mappings, so that one model renders admission demand, care tasks, and discharge proposals across simulations evaluating ICU bed capacity, care dispatch, and post-ICU transfer capacity. Experiments show that our model matches Markov and semi-Markov comparators in aggregate occupancy and directional responses to capacity and dispatch policies. It also reproduces policy gains from replayed observed trajectories. For patients entering the same support state from different prior trajectories, the learned representation captures a 24-h recovery difference (-6.4 predicted versus -6.7 observed percentage points), a contrast missed by compact state models, which propagates into distinct simulated care demand, queue burden, and bed use. These results indicate that a learned world model of patient dynamics has the potential to replace question-specific state models, allowing new care policies to be evaluated by reconfiguring the scenario without model rebuilding.

0.12

CoRe-MARL: Cooperative Redistribution Under Unknown Dynamics Using Recurrent Multi-Agent Reinforcement Learning

cs.LG cs.AI ·2026-09-16 ·Naimur Rahman Chowdhury, Shatabdi Sen Prapti, Md. Salehin Seyam, Limon Bin Hossain
similarity 0.55
Abstract

Emergency management assistance programs, such as relief distribution, are essential for delivering necessary supplies to affected communities. However, these programs operate in a decentralized network of local centers that face uncertain local demand and supply dynamics, resulting in inconsistent avail- ability of local services. Redistribution of supplies among these local centers reduces these imbalances, but the centers often make decisions independently, with limited information and disrupted transportation. This study develops CoRe-MARL, a cooperative multi-agent reinforcement learning (MARL) framework, by formulating a decentralized partially observable Markov decision process (Dec-POMDP). We treat each center as an agent that learns a redistribution policy to improve the service in the worst-case region and reduce the service gap across regions while protecting network-wide service. We incorporate a recurrent network that captures evolving supply and demand dynamics without direct observation, while multi-agent proximal policy optimization (MAPPO) enables centralized training and decentralized execution (CTDE). We evaluate the framework in a simulated environment with diverse trajectories, where exact dynamics are not observed by actors and the MAPPO critic. We compare the recurrent MAPPO with the recurrent independent PPO (IPPO) and a local only heuristic, and find that MAPPO reduces the service gap across local centers and enhances service for the worst-served center while maintaining competitive network-wide service. The recurrent MAPPO also shows consistent performance across diverse trajectory patterns, demonstrating its ability to adapt to evolving dynamics. The findings demonstrate the capability of cooperative learning for decentralized redistribution and improving equitable service under uncertain and evolving dynamics.

0.18

Learning through Capacity Awareness to Minimize Hospital Readmissions

cs.LG cs.CYstat.APstat.ML ·2026-11-08 ·Lydia T. Liu, Kara Schechtman, Pengyi Shi, Angela Zhou
theory similarity 0.54
Abstract

Predictive algorithms are increasingly used to guide hospital discharge decisions, yet these models can exhibit censoring bias for vulnerable patients due to insufficient historical data or past discharge decisions. When biased readmission risk predictors are used to inform discharges, the training data generated by discharge decisions may perpetuate similar biases when predictors are re-trained, leading to a vicious cycle. We extend a queueing-based Markov decision process for hospital discharge to incorporate periodically retrained readmission risk predictors. Our key insight is that cost-optimal, "capacity-aware" operational decision policies can naturally diversify the lengths of stay at which patient outcomes are observed, enabling consistent estimation of readmission risk without explicit fairness constraints. We prove that repeated retraining under discharge-up-to policies converges to unbiased risk estimates under well-specification. In synthetic experiments and calibrated simulations based on patient-level data from a partner hospital, we find that where commonly used threshold-on-risk policies can remain persistently biased, discharge-up-to policies reduce learning error over status quo decision policies (such as thresholding on risk) by 24%-25%. These improvements to estimation also mitigate some forms of bias, reducing readmission cost disparities (adjusted for baseline risk differences between demographic groups) by 38%-63%. Our results demonstrate how capacity-aware decision policies can improve both predictive performance and fairness by addressing censoring bias.

0.09

Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

cs.LG math.NAmath.OC ·2026-09-14 ·Jin Ma, Gaozhan Wang, Jianfeng Zhang, Xunyu Zhou
similarity 0.54
Abstract

We study continuous-time and possibly high-dimensional stochastic control problems where drift coefficients and running reward functions are unknown. Due to these missing model primitives, we take the exploratory, reinforcement learning (RL) framework of Wang, Zariphopoulou, and Zhou(2020) with relaxed controls and entropy regularization. The objective is to develop theoretically grounded, efficient and scalable RL algorithms to learn both the optimal value functions (which also solve the exploratory HJB equation) and optimal exploratory feedback control policies. When the diffusion coefficients do not contain control, we employ probabilistic representations of both the optimal value function and its gradient based on an auxiliary state process depending only on the diffusion part of the original dynamics. With a delicate analysis on some properly defined mappings and their fixed points, this leads to the introduction of our policy iteration algorithms and their convergence. We demonstrate the performance of our algorithms through various numerical examples. Finally, we study a special control-dependent diffusion case where probability representation of the Hessian is called for.

0.39

Policy Synthesis for Finite Populations of MDP Agents under Aggregate Reach-Avoid Chance Constraints

eess.SY cs.MA ·2026-10-08 ·Jie Fu, Anamika Dubey
similarity 0.54
Abstract · 8 pages, 2 figures. Submitted to the 2027 American Control Conference

Consider a finite population of agents with decoupled Markov transition dynamics and empirical-density feedback, subject to the following constraints: with probability at least $1-δ_r$, at least a fraction $α_r$ of agents must reach a target region at some time $t^*$, while, at each time up to $t^*$, the unsafe population fraction must remain below $β_u$ with probability at least $1-δ_u$. However, standard mean-field methods enforce these constraints only in expectation, which fails to account for stochastic fluctuations at finite fleet size $N$. To address this control problem, we propagate the second-order moment (variance) of the empirical density alongside the mean-field trajectory via a discrete-time Lyapunov recursion, and apply the Cantelli inequality to convert chance constraints into tractable deterministic conditions on the moments of the empirical density. We then incorporate these moment-based surrogate constraints into a gradient-based sequential convex approximation procedure for density-feedback policy synthesis. We further introduce additional moment-error bounds to construct a rigorous finite-$N$ certificate. The method is evaluated on a gridworld environment and a power-system EV-charging aggregation problem and compared with a standard deterministic population-level LP baseline.

0.29

Data-Driven Modeling and Predictive Control of Chronic Diseases: An Ulcerative Colitis Application

eess.SY ·2026-10-03 ·Fionna B. Kopp, Ann-Kathrin Schalkamp, Shadera Azzam, Vivek Rudrapatna, Francesco Borrelli
similarity 0.53
Abstract

The clinical management of chronic disease is naturally structured as a feedback control problem. Physicians observe the disease state at clinic visits and make decisions based on the patient's history and current state. We propose a framework that models the dynamics of chronic disease in response to therapy as an event-based controlled stochastic system, in which treatment decisions are made at discrete, irregularly spaced events while the disease evolves between events as a continuous-time Markov chain (CTMC) with state and input dependent transition rates. We demonstrate this framework in the setting of ulcerative colitis, where we identify the model from real-world clinical data in the electronic health record (EHR) of more than 3,000 patients with ulcerative colitis. Using the learned model, we formulate therapy selection as a model predictive control problem (MPC) that maximizes the expected time spent in inactive disease states, and we evaluate the resulting policy. In closed-loop simulation on 10 patients, the MPC achieves a cost at or below the observed clinical course in the majority of its realizations for 4 of the 10 patients, and its outcomes in all cases are comparable to observed care and within clinical guidelines.

0.20

CVaR Deep Optimal Stopping

cs.LG math.OC ·2026-11-08 ·Yechan Jeong, Youngjoon Hong
similarity 0.53
Abstract · Neurips 2026

Risk-sensitive optimal stopping is useful when poor stopping outcomes carry substantial costs, as in option exercise, sequential medical treatment, and safety-critical control. A natural objective is Conditional Value at Risk (CVaR) of the stopped reward. We develop a theoretical framework and algorithm for CVaR deep optimal stopping. The framework and algorithm address four difficulties in applying risk-sensitive reinforcement learning to optimal stopping: state augmentation, score-function pathology, blindness to success, and dependence on auxiliary networks. We isolate three structural properties of optimal stopping: single-point reward, action-independent dynamics, and sum-zero gradient. We exploit these properties to establish Markov optimality without state augmentation and derive a pathwise gradient estimator. Under bounded policy derivatives, the pathwise estimator has bounded second moments near deterministic policies, avoiding the divergence that can occur with score-function estimators. Building on this framework, we propose DIOS (Distribution-Informed Optimal Stopping) to address the remaining two difficulties. A per-time-step soft penalty combined with tail-level annealing mitigates blindness to success. Estimating the reward quantile directly from each batch removes the need for an auxiliary critic. For a fixed tail level, we establish a finite-time stationarity guarantee under regularity assumptions along the training iterates. Across the 24 main Bermudan max-call configurations, DIOS achieves the highest mean CVaR and the shortest recorded training time among the tested CVaR baselines.

0.22