CS Paper Radar

Offline Approximate Value Iteration: A Reinforcement Learning Approach for Satellite Constellation Deployment

Tristan Leuridan, Yiping Fang, Adam Abdin

cs.AI math.OC ·submitted 2026-11-02 09:17 UTC · v1
top venuehot direction

Abstract

This paper introduces a reinforcement learning-based approximate dynamic programming approach to optimize the deployment of Walker-type satellite constellations under uncertainties, including launch failures and satellite malfunctions. We formulate the problem as a Markov Decision Process that incorporates low-injection orbit launches, continuous low-thrust orbit raising, and variable satellite loads. We adopt the offline approximate value iteration solution method, using linear function approximation with basis functions and action pruning strategies to substantially reduce the computational and memory requirements for finding optimal decision policies. Preliminary results demonstrate convergence to policies with low expected costs and deployment times, both of which are lower than those of a rolling-horizon baseline, showcasing a novel application of approximate dynamic programming in space logistics.

Journal ref
3rd Conference on AI in and for Space (SPAICE 2026), European Space Agency (ESA), Oct 2026, Noordwijk, Netherlands

Closest papers

Nearest neighbours by meaning across the whole index.

Reinforcement Learning-Guided Evolutionary Policy Optimization for Preference-Adjustable Heterogeneous Agile Earth Observation Satellite Scheduling

cs.AI ·2026-08-25 ·He Wang, Junyu Wu, Hui Li, Yanjie Song, Witold Pedrycz +1 more
published similarity 0.65
Abstract · Published in IEEE Transactions on Geoscience and Remote Sensing

Heterogeneous agile Earth observation satellite (AEOS) scheduling requires task selection, satellite assignment, and observation sequencing under satellite-dependent visibility windows, attitude maneuvering requirements, energy consumption, and onboard storage constraints. Since satellites differ in orbital access, maneuvering capability, and payload resources, the same task may have different feasible windows, transition costs, and resource-consumption patterns on different platforms, which increases the difficulty of unified modeling and efficient optimization. To address this problem, this paper proposes an evolutionary policy optimization framework for heterogeneous AEOS scheduling with preference-adjustable weighted objectives. In the modeling layer, assignment-based indirect encoding is combined with decoder-based equivalent-cost evaluation to retain satellite-dependent constraints while integrating task gain, energy saving, and load balance into an interpretable scalar utility. In the optimization layer, schedule decoding, population-based search, and online actor-critic operator control are decoupled, so that reinforcement learning selects high-level search operators rather than constructing schedules directly. Based on this framework, a reinforcement-learning-assisted operator-selection memetic evolutionary algorithm (RLOSMEA) is developed to coordinate global exploration, feasibility recovery, and local refinement under a limited function-evaluation budget. Experiments on different heterogeneous AEOS scenarios show that RLOSMEA achieves higher overall weighted utility and more stable convergence than representative metaheuristic baselines. Sensitivity and learning-behavior analyses further confirm the robustness of the proposed method and the effectiveness of reinforcement-learning-guided operator selection.

0.46

Stochastic Multistage Constellation Reconfiguration Problem: Stochastic Dual Dynamic Integer Programming Approach

eess.SY math.OC ·2026-08-10 ·Brycen D. Pearl, Hang Woon Lee
similarity 0.64
Abstract · 34 pages

Observation tasking is critical to satellite operations, enabling observation of planetary phenomena, orbital debris monitoring, and space domain awareness. To improve the effectiveness of satellite operations, orbital maneuverability is introduced as a leading-edge concept of operations within constellation reconfigurability for response to dynamic events. Previous investigations regarding constellation reconfigurability consist of deterministic mission environments, in which the formulations consider a priori knowledge of the environment; however, observation objectives inherently involve uncertainty at any given time. As such, scheduling satellite operations must account for uncertainties to ensure adequate observation tasking. In response, we present a stochastic variant of the Multistage Constellation Reconfiguration Problem (MCRP) which is solved using Stochastic Dual Dynamic Integer Programming (SDDiP). We additionally explore other stochastic problem-solving techniques for solution quality comparison. To demonstrate each solution method, two computational experiments with stochastic target properties are conducted. The first concerns random orbital targets, and the second concerns simulated hurricanes. The results of the experiments demonstrate the effectiveness of the SDDiP solution approach over other stochastic problem-solving methods. Overall, the stochastic MCRP accounts for target stochasticity while obeying visible time windows and maneuver feasibility.

0.10

Risk-Sensitive Mission Planning for Multi-Target On-Orbit Inspection via Distributional Reinforcement Learning

cs.AI math.OC ·2026-10-30 ·Adam Abdin
published similarity 0.64
Abstract

Multi-target on-orbit servicing missions require planning tools that can adapt to evolving operational conditions while accounting for the outcome variability inherent to safety-critical spaceflight. This paper presents an adaptive planning approach for multi-target servicing campaigns in low Earth orbit and applies it to inspection missions flown by a low-thrust servicer. The approach is based on Distributional Reinforcement Learning, which learns the full distribution of mission returns and thereby supports explicit risk-sensitive decision-making at each planning step. Specifically, a Quantile-Regression Deep Q-Network (QR-DQN) agent selects which target to inspect next by jointly considering the expectation and the variance of the predicted return through a mean-variance criterion. The mission environment models circular orbits with a spread in inclination and uses Edelbaum's combined-manoeuvre low-thrust transfer estimates. The observation comprises the orbital elements, the visit history, the observed risk state of each target and the remaining mission-time budget. The risk state of each target changes stochastically during the campaign, making the outcome of an inspection uncertain and creating operationally meaningful distinctions between risk-neutral and risk-averse planning strategies. Because the mission-time budget generally admits only a subset of the candidate targets, the planner must also select which targets to inspect. Numerical experiments on scenarios with five target satellites show that one risk-aversion weight moves the policy across a range of mean return and return variance, over an unchanged environment and reward. Risk-averse agents reduce the variance of the episodic return by approximately one half relative to risk-neutral agents and lower the probability of a mission loss, at a cost in mean return. To validate the learned policies, the efficient frontier, which gives the highest mean return attainable at each level of return variance, is computed exactly by enumerating the admissible visit orders. At equal variance, the learned policies lie close to this frontier in mean return, and the risk-averse policies have a lower probability of a mission loss than the frontier policies.

0.36

Implicit Q-learning-bootstrapped ant colony optimization for maritime moving-target observation scheduling with agile satellites

cs.AI ·2026-08-25 ·He Wang, Junyu Wu, Yeye Liu, Yifan Zhou, Jie Zhang +3 more
published similarity 0.63
Abstract · Revised manuscript incorporating changes made during peer review. Published in Aerospace Science and Technology

Maritime moving-target observation scheduling with agile Earth observation satellites is a dynamic, sequence-dependent combinatorial optimization problem. Sea-surface targets move continuously, causing feasible observation windows to vary with target motion and satellite orbital geometry. The scheduler must jointly determine task selection, satellite assignment, observation-window selection, and observation ordering under time-window, attitude-maneuvering, and onboard-resource constraints. This paper proposes an implicit Q-learning-bootstrapped ant colony optimization method, termed IQACO, for multi-satellite maritime moving-target observation scheduling. Rather than directly learning a task-selection policy, IQACO embeds an offline implicit Q-learning module into constructive ant colony optimization to adaptively adjust the pheromone factor, heuristic factor, and evaporation rate. A compact search-state representation captures pheromone distribution, current and historical-best solution quality, and iteration progress. During online scheduling, ant colony optimization constructs feasible observation sequences, while the learned policy adjusts the search behavior according to the current search state. Experiments on 14 scenarios with different scales and satellite configurations show that IQACO consistently outperforms the compared algorithms, improving the mean objective value over conventional ant colony optimization by 2.86%-9.41%. Further comparative and supplementary experiments demonstrate its effectiveness and robustness across different scheduling conditions and problem settings. These results indicate that offline value learning provides an effective adaptive search-control mechanism for constrained maritime moving-target observation scheduling.

0.40

Towards Optimal Policy Improvement

cs.LG ·2026-10-01 ·Yaniv Oren, Viliam Vadocz, Wiktor Zabka, Thomas Evers, Jan Robine +5 more
similarity 0.63
Abstract

Practical Reinforcement Learning (RL) algorithms learn to solve Markov Decision Processes (MDPs) through iterative policy improvement in the presence of approximate evaluation. We study policy improvement from first principles, defining optimal policy improvement as producing the best policy attainable in a single update under specified constraints. We show that optimal improvement restricted to a set of states is equivalent to solving an induced MDP, characterizing planning with an explicit or implicit model as a path towards optimal policy improvement. Because practical methods commonly solve such induced problems through iterative improvement in the form of greedification, we take steps towards optimal greedification under the central practical constraint of approximate evaluation. We formulate greedification under this constraint as probabilistic decision-making under uncertainty and derive a novel operator that is optimal with respect to the resulting objective. Empirically, the operator and its practical gradient-based approximations improve aggregate performance across GumbelAlphaZero, SAC, ReBRAC and Generalized Policy Iteration, in experiments spanning discrete and continuous actions, model-based and model-free, online and offline RL.

0.21

ASTRA: ADMM-Accelerated Topology Reconfiguration for Dynamic Satellite Constellations

cs.LG ·2026-09-27 ·João Norberto, Ricardo Ferreira, Cláudia Soares
top venue similarity 0.63
Abstract · Published at NeurIPS 2026

Dynamic topology reconfiguration is central to the reliability and efficiency of large satellite constellations, yet many existing approaches rely on idealized assumptions such as full constellation deployment or uniform orbital spacing. We present Adaptive Satellite Topology via Regret-Aware learning (ASTRA), a theoretically grounded framework for dynamic satellite topology reconfiguration that builds on an online learning formulation and makes it computationally practical. ASTRA combines an ADMM-based offline solver with efficient online updates for both online gradient descent and online conditional gradient, yielding markedly cheaper constrained updates than generic optimization pipelines. On the theory side, we show that for a relevant class of entry-wise nonzero utility matrices, the objective is strongly convex, which yields logarithmic static regret for online gradient descent, and we further instantiate known dynamic-regret guarantees under inexact ADMM inner loops. Empirically, ASTRA matches or improves topology quality, presenting a good trade-off with computational time on synthetic constellations, and it remains effective on real Starlink data under partial deployment and non-uniform spacing, where idealized structural assumptions break down. These results position ASTRA as an efficient and theoretically grounded approach to topology reconfiguration in realistic Low Earth Orbit networks.

0.51

From Orbit Matching to Docking: A Hierarchical Reinforcement-Learning Approach to Multiphase Rendezvous Trajectory Design

cs.RO cs.LGeess.SY ·2026-10-12 ·Mariko Storey-Matsutani, Anatoly Zlotnik, Lee Holguin, Zigfried Hampel-Arias, Richard Linares
similarity 0.62
Abstract

As the complexity of space missions increases, autonomous spacecraft guidance must accommodate transitions between mission phases with distinct dynamics, objectives, and constraints. Rendezvous and proximity operations (RPO) are particularly challenging because they span multiple operating regimes, from orbital-scale matching and far-range approach to close-range maneuvering and terminal docking. Existing reinforcement-learning approaches generally address only a subset of this sequence or rely on manually defined switching conditions between controllers. This study presents a hierarchical reinforcement-learning framework for autonomous, three-degree-of-freedom RPO that spans the complete trajectory from orbit matching to docking. Five specialized policies are trained for orbit matching, far-range approach, mid-range approach, close-range approach, and docking. A learned high-level manager policy then selects among these phase-specific controllers based on the evolving spacecraft state. The hierarchical approach is evaluated over the full end-to-end rendezvous sequence and compared with a fixed sequential-invocation heuristic to assess trajectory performance and transition behavior. Training and inference metrics are also analyzed for the individual phase policies and the complete hierarchical system.

0.12

Reinforcement Learning and Operational Research: A Review of Methodological Integration and Research Directions

cs.LG math.OC ·2026-09-21 ·Yahan Lu, Dongyang Xia, Nursen Aydin, Shadi Sharif Azadeh
survey similarity 0.61
Abstract

The growing demand for real-time, data-driven decision-making in complex and dynamic systems is placing increasing pressure on Operational Research (OR) methodologies. Sequential decision-making under uncertainty has long been studied in OR through Markov decision processes, dynamic programming, and their approximations, and reinforcement learning (RL) provides simulation-based methods for learning decision policies in such problems. Recent research shows an increasing interest in integrating RL with OR to address dynamic decision-making problems and enhance heuristic and exact methods for combinatorial optimization. The overarching goal across these efforts is to improve solution quality, computational efficiency, and robustness. Given the diversity of integration approaches and application settings, there is a clear need for a systematic and technically detailed review of how RL empowers OR methods. To address this gap, this paper presents a structured review of two methodological roles of RL in OR. The first concerns sequential decision problems in dynamic environments, where states evolve and decisions are repeated over time. The second concerns static combinatorial optimization problems, where inputs are fixed before the solution process begins and RL either constructs solutions or supports heuristic and exact algorithms. Digital twins provide an optional deployment and evaluation layer for training, testing, and validating these methods. We critically synthesize recent advances across these roles and uses, highlighting their advantages, implementation requirements, limitations, and challenges. Finally, we consolidate guidance on when the reviewed approaches may be considered and outline directions for future research on the methodological and practical integration of RL and OR.

0.26