CS Paper Radar

Comfort as a Constraint or as a Cost in Reinforcement Learning and Model Predictive Building Control

Joram Liebeskind, Valentin Bhend, Varsha N. Behrunani, Alisa Rupenyan

eess.SY cs.LG ·submitted 2026-11-16 10:58 UTC · v1

Abstract

A building controller that keeps a zone within comfort bounds at minimum energy can treat the bounds as a soft constraint, by adding the weighted violation to its cost, or as a hard constraint. We analyze the effect of the hard and the soft formulation of the same problem and isolate the effect of the formulation from the effect of the algorithm. To this end, we measure the difference in energy use between the two formulations on the BOPTEST bestest_hydronic_heat_pump case, in model predictive control (MPC) and in reinforcement learning (RL). The soft formulations are a weighted MPC and weighted SAC and DDPG learners, each at twelve weights. The hard formulations are an MPC with an exact penalty and the Lagrangian learners SAC-Lagrangian and primal-dual DDPG. We find that the hard formulations use more energy than the soft formulations at the largest weight, but violate the comfort bounds less. For a large weight, the soft formulation approaches the behavior of the hard formulation. The frontiers of the energy-comfort trade-off of the three weighted controllers lie close to each other, and the weight selects an operating point on this frontier. Under the soft formulation, the three algorithms reach very similar operating points, although at somewhat different weights. For this HVAC problem, the formulation and the weight thus affect the energy use more than the choice of algorithm.

Closest papers

Nearest neighbours by meaning across the whole index.

Safe Deep Reinforcement Learning for Energy-Efficient HVAC Control in Multi-Zone Residential Buildings

eess.SY ·2026-08-18 ·Oussama Ziadi, Abdelilah Rochd, Samir Idrissi Kaitouni, Mohamed Oualid Mghazli, Adnane Saoud
published similarity 0.64
Abstract · 6 pages, 5 figures. Accepted to IEEE Conference on Control Technology and Applications (CCTA) 2026

HVAC systems represent a major share of building energy consumption. Traditional control strategies are limited in coordinating energy-comfort tradeoffs across multiple zones simultaneously. Reinforcement learning (RL) offers adaptive, data-driven control that optimizes performance over time. However, deploying learned neural network controllers in safety-critical building systems remains challenging due to lack of formal safety guarantees. We propose a safety-certified deep RL framework for multi-zone residential HVAC control. Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) agents are trained in an EnergyPlus/Sinergym simulation to minimize energy consumption while maintaining thermal comfort. Post-training safety certification is performed on the PPO policy using Lipschitz-based forward invariance analysis, building on existing tools for the computation of Lipschitz constants for neural networks, to guarantee constraint satisfaction. Both agents are evaluated over an annual simulation cycle in an eight-zone variable refrigerant flow (VRF) testbed. The PPO agent achieves 67\% comfort violation reduction compared to rule-based control, while the SAC agent achieves 27.6\% energy savings. The PPO policy satisfies formal safety certification with a margin of $2.003^\circ$C. These results demonstrate the feasibility of combining reinforcement learning with post-training safety verification for multi-zone building control.

0.20

User-specified Kelvin-hour budgets within model predictive control for energy-efficient buildings: Simulation and field demonstration

eess.SY ·2026-09-06 ·Jicheng Shi, Colin N. Jones
similarity 0.63
Abstract

For energy-efficient building control, model predictive control (MPC) has been widely proposed to reduce energy use while maintaining prescribed indoor temperature bounds. In practice, MPC implementations may still produce temperature-bound violations because of model mismatch, weather-forecast errors, and softened constraints introduced to preserve feasibility. Existing soft- constrained MPC usually handles this trade-off through slack-weight tuning, so users cannot prescribe a violation-severity budget before operation. This paper develops a Kelvin-hour (Kh)-budget MPC that allows users to specify a running-average budget for temperature-bound violation severity. During operation, the controller uses the realized Kh budget surplus or overspend to update the temperature bounds, providing step-by-step budget feedback. We evaluate the method in two high-fidelity BOPTEST simulation cases and in an occupied residential deployment. In the first one-zone case, prescribed budgets produce clear running-average Kh violation responses across predictors and disturbance settings. At a prescribed budget of 0.005 Kh/step, the controller reduces energy use by 31.9% and Kh violation by 36.2% relative to the built-in default controller. In the second coupled two-zone case, separate zone budgets produce zone-level responses, with 18.9%-20.5% energy reduction and 9.4%-64.3% zone-level Kh violation reduction. In the occupied residential deployment, the running-average Kh violations remain close to or below their prescribed budgets under real sensing, actuation, weather, and occupancy conditions.

0.27

Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control

cs.LG cs.AI ·2026-08-10 ·Faizan Ahmed, Aniket Dixit, James Brusey
published similarity 0.63
Abstract · 9 pages, 4 figures, Accepted at UKCI 2026, Will be published in Springer

On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.

0.25

Efficient Learning of Differentiable MPC Policies on the GPU via Smoothed ADMM and First-Order Reinforcement Learning

cs.RO cs.AIcs.LG ·2026-10-13 ·Jianghan Zhang, Roland Schwan, Gabriel Bravo-Palacios, John Zhang, Thomas Lew +1 more
code/data similarity 0.58
Abstract

Differentiable model predictive control (MPC) enables structured, constraint-aware policies to be optimized end-to-end. However, learning MPC policies faces two bottlenecks: (1) nonsmooth sensitivities at active-set transitions and (2) expensive policy evaluation, since each requires an optimization solve. We first introduce SmoothedTurboMPC, a GPU-parallel differentiable MPC solver combining sequential quadratic programming (SQP) with a log-barrier-smoothed alternating direction method of multipliers (ADMM) inner solver. Its design enables smooth gradients, exact sensitivities at convergence, and efficient execution on GPUs. Second, to efficiently train MPC policies, we pair our solver with first-order reinforcement learning (FORL) using Short-Horizon Actor-Critic (SHAC), which reuses rollout MPC solutions instead of repeatedly re-evaluating them. Comparisons with state of the art solvers show that the approach achieves up to $4.4\times$ higher end-to-end training throughput. On a vehicle racing task, our method achieves higher returns than MPC PPO with a $46\times$ speedup, training on 3.3 million transitions in only 7 minutes. Finally, we use the framework to train a more reliable MPC racing policy via RL with domain randomization, and validate it on a full-scale race car. We integrate our advances into the open-source TurboMPC solver.

0.20

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC

cs.RO cs.LGeess.SY ·2026-09-01 ·Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter +4 more
similarity 0.57
Abstract · Submitted to IEEE Transactions on Robotics (T-RO). 18 pages, 13 figures

In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context-dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solutions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias-variance tradeoff: Reinforcement Learning (RL) optimizes realized closed-loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low-variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Reinforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve stability and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO's best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments.

0.41

Deep Reinforcement Learning for Heat Pump and Underfloor Heating Control: An End-to-End Case Study

cs.AI eess.SY ·2026-11-17 ·Antonio Manjavacas, David Suárez-Caro, Alejandro Campoy-Nieves, Guillermo Rey, Gonzalo Satrústegui +1 more
similarity 0.56
Abstract · Preprint submited to journal

Efficient control of underfloor heating (UFH) systems coupled with heat pumps (HPs) is challenging due to the high thermal inertia of buildings and the slow dynamic response of UFH systems. Deep Reinforcement Learning (DRL) offers a data-driven control approach capable of simultaneously optimizing energy consumption and thermal comfort. However, the application of DRL-based controllers in occupied buildings under real operating conditions remains limited. This work presents and end-to-end case study (development, evaluation, and real-world deployment) of a DRL-based controller for an integrated UFH-HP system. First, a simulation-based pilot study was conducted to assess different DRL algorithms, control strategies, and configurations. Subsequently, a digital twin of a real residential building was developed and used to train DRL agents under simulated operating conditions. The best-performing agent was then deployed in the physical building without additional online fine-tuning. The results show that the controller successfully maintained indoor temperatures within the prescribed comfort limits while dynamically adjusting the HP supply temperature and UFH valve positions. These findings demonstrate the feasibility of deploying DRL controllers in occupied buildings and highlight their robustness and the practical challenges associated with transferring control policies from digital environments to real heating systems.

0.32

Online Optimization for Controlling HVAC Systems

eess.SY math.OC ·2026-11-05 ·Tong Wu, Dominik Baumann, Angela Fontan, Nicola Bastianello
similarity 0.56
Abstract

Heating, ventilation, and air-conditioning (HVAC) systems account for a significant portion of buildings' energy consumption, motivating the design of energy- and cost-efficient control strategies. However, conventional strategies employ simple, reactive control, without incorporating a model. Therefore, in this paper we adopt an economic model predictive control framework and focus on designing online algorithms to solve the control input design problems with a limited computational budget. In particular, we propose two tailored online algorithms based on projected gradient descent, with a custom projection routine, and the projected primal-dual method. We discuss the pros and cons of both algorithm designs, focusing in particular on computational efficiency. We further compare them through numerical experiments simulating the control of an HVAC system over the course of a day. The results show that computationally efficient online solvers for HVAC systems control are indeed possible. Moreover, the performance closely matches that of a baseline algorithm with unlimited computational budget, in terms of reference temperature tracking and comfort temperature range enforcement.

0.30

Advancing Weighted Partial MaxSAT Solving via Differentiable GPU Guidance

cs.AI ·2026-11-11 ·Mingju Liu, Curie Kim, Steve Dai, Cunxi Yu
benchmark similarity 0.56
Abstract · 13 pages, 6 figures

We present a GPU-CPU hybrid framework for Weighted Partial MaxSAT (WPMAX-SAT) that complements state-of-the-art CPU solvers with differentiable GPU optimization. The GPU stage directly optimizes instance-specific categorical decision parameters without a training dataset or a pretrained predictive model. The central challenge in extending differentiable optimization to WPMAX-SAT is the hard/soft clause asymmetry: hard clauses represent inviolable feasibility constraints while soft clauses carry weights. We introduce a hierarchical differentiable loss that prioritizes hard clause satisfaction before optimizing the weighted soft objective, ensuring that GPU gradient descent leans towards the constraint structure of WPMAX-SAT. Our framework transforms the GPU-optimized variable logits to high-confidence assignments and transfers them to CPU solvers via two complementary modes: partial guidance, which injects the most confident perturbed partial variable assignments as soft clauses into $N$ parallel solver instances, and phase initialization, which guides the CDCL solver's internal branching polarity with the full GPU assignment as a warm start to preserve the problem's originality. We evaluate on all 229 instances of the MSE 2024 (most recent due to cancellation in 2025) anytime weighted benchmark with the top 2 winners as baselines; our hybrid framework improves the average solution score by up to 4.9% over tt-open-wbo-inc-Glucose and 4.8% over SPB-MaxSAT-c-FPS at a 300-second time limit, with the potential to obtain better solutions surpassing the best known solution recorded in the competition evaluation.

0.10