CS Paper Radar

Stability-Preserving Reinforcement Learning for Heterogeneity-Aware Composite Max-Pressure Traffic Signal Control

Jichen Zhu, Chaopeng Tan

eess.SY ·submitted 2026-10-22 10:16 UTC · v1

Abstract

Reinforcement learning (RL) provides a flexible mechanism for adapting traffic signal control to heterogeneous traffic and sensing conditions, but directly learned signal actions do not automatically inherit the analytical properties of structured traffic-control methods. This paper proposes Com-MP, a theory-guided RL framework that learns a context-dependent composition of multiple max-pressure (MP) control preferences while retaining decentralized MP phase selection. The learned policy adapts the relative influence of constituent MP formulations according to local traffic and sensing conditions, allowing data-driven adaptation to operate within a structured traffic-control framework. We show that such intersection-specific and time-varying composition preserves strong queue stability for every demand in the strictly stabilizable region under any admissible non-anticipative composition policy, thereby separating the performance-adaptation role of learning from the stability property of the underlying controller. In a representative implementation combining detector-supported queue-based MP and connected-vehicle-supported travel-time MP, experiments on a real-world 45-intersection network show that Com-MP reduces average delay by 15.6\% and 8.6\% relative to the two constituent controllers, respectively, and by about 7.0\% relative to the best fixed composition. The learned composition responds systematically to local traffic and sensing conditions, remains effective under heterogeneous and degraded sensing, and can be transferred without retraining to an unseen network with an average delay only 2.4\% higher than that of a policy trained directly on the target network. These results show how learning-based adaptation can be introduced within a stability-preserving traffic-control structure, providing a theory-guided pathway toward trustworthy learning-enabled traffic signal control.

Closest papers

Nearest neighbours by meaning across the whole index.

Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control

cs.LG eess.SY ·2026-09-15 ·Giray Önür, Azita Dabiri, Bart De Schutter
similarity 0.70
Abstract

Integrated deep reinforcement learning (DRL) and model predictive control (MPC) methods are increasingly used to control autonomous systems by combining their complementary capabilities. DRL learns control policies through interaction with the environment. MPC uses a system model to optimize control inputs while accounting for constraints. In DRL-MPC frameworks with shared control authority, both the DRL agent and the MPC controller each determine part of the control inputs. However, common learning formulations treat MPC as part of the environment and therefore do not explicitly account for MPC's contribution to control or its interaction with the DRL agent. This paper proposes a novel composite-gradient learning (CGL) method that integrates the MPC controller into the learning process by representing the DRL and MPC control inputs as a joint action and accounting for their interaction when updating the DRL agent during training. CGL is evaluated on two multi-class freeway traffic networks with different strengths of interaction between the DRL and MPC control inputs and it is compared with alternative methods that treat MPC as part of the environment or that only partially incorporate MPC into learning. The results show that CGL offers limited benefit under weak interaction, but learns higher-performing control policies than the alternative methods in a subset of training runs under strong interaction, although the average control-performance gains remain modest.

0.06

HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations

cs.LG ·2026-11-12 ·Chenkai Hu, Yingqian Zhang, Ming Xie, Yi Fang, Chengxiang Jin +1 more
code/datahot direction similarity 0.70
Abstract

Multi-agent reinforcement learning has shown great promise for traffic signal control. However, most existing methods rely on two key assumptions: (i) agents are assumed to be homogeneous with unified observation/action spaces; (ii) full and reliable lane-level traffic observations are available to all agents. In practice, urban road networks exhibit substantial structural heterogeneity (e.g., lane configurations and phase designs), while sensor failures and incomplete detector coverage often result in missing incoming lane measurements. Such issues can severely impair coordination and destabilize learning. To address these challenges, we propose HALight, a communication-enhanced heterogeneous multi-agent reinforcement learning framework for traffic signal control under lane-level missing observations. HALight supports intersection-specific observation and action modeling to accommodate heterogeneous intersections, and incorporates a lightweight one-hop neighborhood communication module to enrich spatial context when lane-level observations are partially unavailable. Moreover, HALight adopts a sequential policy update scheme to improve coordination among heterogeneous agents. Experimental results on two real-world road networks demonstrate that HALight outperforms representative homogeneous and heterogeneous baselines in terms of traffic efficiency and convergence behavior. The code and datasets used in this work are available at https://github.com/Huchenkaii/HALight.

0.47

Decision Timescales in Max-Pressure Traffic Signal Control: Theory, Methodology, and Validation

eess.SY ·2026-10-23 ·Chaopeng Tan
benchmark similarity 0.69
Abstract · 31 pages, 11 figures

Max-pressure (MP) control adapts signal phases to prevailing traffic states, but its decision timescale is typically fixed a priori. This paper treats the decision timescale as an explicit dimension of MP control. We first characterize the trade-off between responsiveness and switching efficiency by quantifying how an equivalent switching loss reduces service capacity over a finite decision interval. This yields a load-dependent critical timescale determined by switching loss and normalized traffic load, together with a performance-oriented operating range above the stability boundary. Building on this result, we propose Variable-Timescale Max-Pressure (VT-MP), in which each intersection selects its decision interval according to local operating conditions, allowing the control timescale to vary across time and space. We establish strong stability of the resulting asynchronous heterogeneous controller under uniformly admissible local loads and bounded decision intervals, and introduce a downstream receiving-supply correction for finite-storage implementation. Three experiments provide mechanism validation and network-level evaluation. A single-intersection experiment confirms that the preferred decision interval increases systematically with traffic load and follows the derived characteristic timescales. In a store-and-forward network, heterogeneous VT-MP reduces total queueing time by 31.0% relative to the best fixed-interval benchmark. In a microscopic 5x5 SUMO network, it reduces mean trip delay by 24.3%, from 213.3 s to 161.5 s, and peak network accumulation by 12.3%. These findings show that the MP decision timescale should be treated as a traffic-state-dependent control variable rather than a fixed implementation parameter.

0.30

Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks

eess.SY cs.LG ·2026-08-21 ·Giray Onur, Azita Dabiri, Bart De Schutter
similarity 0.69
Abstract

Transportation networks, in particular multi-class transportation networks (i.e., networks with mixed vehicle types), are complex systems that are challenging to control. Recently, Deep Reinforcement Learning (DRL), which learns control policies from interactions with the environment, and Model Predictive Control (MPC), which uses a system model to optimize control inputs, have been increasingly utilized for transportation network control. However, nonlinear system dynamics and high-dimensional state spaces in large-scale networks limit DRL's learning capacity under time-constrained training and increase MPC's computation time, hindering real-time implementation with limited computational resources. Moreover, MPC depends on an accurate network model, which is often unavailable for complex systems such as multi-class transportation networks. This paper proposes a novel DRL-MPC framework for multi-class transportation networks that divides control authority between DRL and MPC, combining DRL's fast online computation and model independence with MPC's built-in optimization and constraint-handling capabilities. In the hierarchical framework, MPC operates at the higher level and determines low-frequency control inputs whose slower update rate accommodates its high computation time, while DRL operates at the lower level and determines high-frequency control inputs using its fast online deployment. The framework is evaluated on a multi-class freeway network against a hierarchical MPC controller and a hybrid state-feedback-MPC controller, including scenarios with model mismatch and noisy traffic demands. Results show that the proposed framework outperforms the hybrid state-feedback-MPC controller, substantially reduces online computation time compared with the hierarchical MPC controller, and provides more effective constraint enforcement under model mismatch.

0.41

A Review of Max-Pressure Traffic Signal Control Algorithms: State of the Art and Research Opportunities

eess.SY math.OC ·2026-11-02 ·Abdullah Al Asif, Chaopeng Tan, Vikash V. Gayah, Hao Liu
survey similarity 0.68
Abstract · 49 pages, 7 figures

Max-Pressure (MP) has emerged as a widely studied decentralized traffic signal control framework, thanks to its desirable theoretical properties and strong control performance. Early research on MP focused on improving control performance by exploring different variables of the traffic state for the definition of pressure. Subsequent studies expanded MP to incorporate the complexity of real-world transportation systems and emerging technologies. This paper provides a comprehensive review of these developments. The review first introduces the theoretical foundations of MP and examines early developments in movement weight definition. It then synthesizes major extensions for multimodal and multiscale network control, followed by recent developments in connected and automated vehicle (CAV) environments and the integration of MP with artificial intelligence (AI) and data-driven methods. Beyond summarizing existing approaches, this review examines their underlying assumptions, data requirements, implementation considerations, and practical limitations. Based on this comprehensive review, key research gaps and future directions are identified, including remaining limitations in the theoretical foundations of MP and the underexplored integration of autonomous vehicle control with the MP framework. Overall, this review provides a structured perspective on the evolution and current state of MP control and highlights opportunities for developing more proactive, robust, and practically deployable MP-based traffic control methods.

0.27

Eco-MP: Ecological Max-Pressure Traffic Signal Control with Ecological Driving

eess.SY ·2026-11-04 ·Haoran Wang, Shuoyuan Li, Chaopeng Tan, Hao Liu, Keshuang Tang +1 more
similarity 0.68
Abstract · 27 pages, 11 figures

Max-pressure (MP) traffic signal control provides a decentralized mechanism for urban networks, but conventional pressure states do not directly capture the propulsion burden generated by signal-induced stop-and-go traffic. Eco-driving, by contrast, typically relies on prespecified signal information and optimizes vehicle trajectories separately from the feedback among vehicle motion, traffic states, and signal decisions. This paper develops Ecological Max-Pressure (Eco-MP), a closed-loop framework that integrates ecological MP signal control with MP-adaptive ecological driving. At the signal layer, connected-vehicle observations are aggregated into a common movement state based on cumulative positive propulsion work, and the resulting upstream--downstream ecological pressure determines phase selection in mixed petrol and electric traffic. At the vehicle layer, a rolling signal interface translates the evolving Eco-MP process into near-term signal information, while each connected and automated vehicle solves a powertrain-independent finite-horizon trajectory problem subject to vehicle-dynamic, car-following, and signal-compliance constraints to reduce unnecessary stopping and acceleration effort. Under full-state information, a finite-storage Lyapunov analysis establishes strong stability for admissible demands satisfying a strict supportability condition and extends the result to the coupled signal--vehicle system. When ecological driving enlarges the admissible-flow sets, the coupled admissible demand region contains the signal-only region. Microscopic experiments on an Amsterdam arterial corridor show that, relative to queue-based MP, Eco-MP reduces mean positive propulsion work by 5.2%, vehicle delay by 20.8%, and stopping time by 39.0%. With only 15% of vehicles directly controlled, ecological driving further reduces positive propulsion work by 4.2% and stopping time by 19.6%......

0.31

Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic

cs.LG cs.AIcs.ROeess.SY ·2026-10-29 ·Hooman Hamze, Benjamin Houshmand, Deepthi Pathare, Erik Börve, Morteza Haghir Chehreghani +1 more
similarity 0.68
Abstract · 13 pages, 2 figures, 8 tables

Autonomous highway driving for heavy-duty trucks is a challenging control problem. The greater mass, dimensions and intricate dynamics increase the requirements for cooperation from surrounding traffic in critical maneuvers, requiring a careful balance between safety constraints and efficient long-horizon decision making. Model Predictive Control (MPC) provides rigorous constraint satisfaction but relies on terminal cost which can limit its effectiveness at short, computationally affordable horizons. Reinforcement Learning (RL) captures long-term driving behavior but offers no constraint guarantees and tends toward overly conservative policies when trained under sparse collision penalties. In this paper, we propose a hybrid MPC and RL framework for highway driving in autonomous trucks, where a finite-horizon MPC, solved via dynamic programming, uses a pre-trained RL model for terminal cost estimation. We evaluate the framework in a simulation environment with reactive surrounding traffic and compare it against pure MPC with zero terminal cost, rollout with terminal cost estimation using a base policy, and pure RL. Results show that the hybrid approach achieves zero collisions with the lowest cost of any configuration.

0.12

Unified Optimization of Traffic Signal and Lane Channelization for an Isolated Intersection Using Reinforcement Learning

cs.LG ·2026-11-10 ·Jiwon Kang, Jiyoung Kim, Hyunseok Jun, Keemin Sohn
similarity 0.68
Abstract

Optimizing traffic signal control has been a long-lasting critical issue in traffic engineering, and a number of robust methodologies have emerged to solve the problem so far. On the other hand, the road design for lane channelization is a static problem that is difficult to solve jointly with the real-time traffic signal control. If traffic volume does not vary, it is possible that both problems can be solved in an integrated manner as a single static optimization problem. Recently, researchers have devised a mixed integer linear problem (MILP) to jointly determine the traffic signal and road width designs. However, this joint optimization problem must be solved repeatedly whenever the traffic flow pattern varies. Solving the MILP may also take impractical computing time as the problem size gets larger. We present a robust methodology for solving the joint optimization problem using reinforcement learning (RL) on a real-time basis. Abrupt changes in lane markings with an incompatible signal phase can jeopardize drivers. To sidestep such a dangerous action change, we train a model with human feedback to filter infeasible action transitions in a rule-based manner. We devised a deep Q-network that can accommodate joint actions for traffic signal and lane marking plans. To ensure the implementation of the proposed operation, a variable sign for lane markings is necessary at each stop line of an intersection. The experimental results confirm that the proposed methodology enhances the intersection performance by increasing throughput.

0.20