CS Paper Radar

Reinforcement Twinning via Twin-Calibrated Bayesian Policy Search and Replay-Driven Identification

Miguel A. Mendez, Lorenzo Schena, Romain Poletti, Yannick Lecomte, Sebastiano Randino, Jan van den Berghe

eess.SY ·submitted 2026-11-02 18:03 UTC · v1

Abstract

The feasibility of updating and synchronising digital twins of actuated systems from operational data depends strongly on how the system is controlled and, consequently, on the data generated during operation. This establishes a direct link between control actuation and model identifiability. This work presents a reinforcement-twinning framework that exploits this link through three computational components. A Gaussian-process surrogate of the physical return combines the twin prediction, used as a calibrated trend, with a discrepancy learned from measured returns, allowing Bayesian policy search to exploit the twin without being tied to its predictions. Candidate policies are proposed around the policy that is optimal for the twin, while measured returns can expand this region. The resulting physical episodes are replayed on the twin after observer-based synchronisation with sparse measurements, and its parameters are identified by combining Bayesian optimisation with gradient-based refinement. Selected information-oriented episodes deploy an excitation policy to improve parameter identifiability. The framework is demonstrated on a Burgers-flow analogue of dynamic-induction control in wind farms, with a deliberately misspecified model structure. Despite substantially biased return predictions, the twin provides a useful starting policy and search region, placing the guided search ahead of unguided Bayesian optimisation from its first physical evaluations. This advantage is maintained when the search region is allowed to expand in response to changes in measured returns, whereas searches over-constrained by the twin-defined region are eventually overtaken by the unguided alternatives. The results support reinforcement twinning as a framework for jointly adapting models and control policies when physical interaction is limited and model discrepancy cannot be neglected.

Closest papers

Nearest neighbours by meaning across the whole index.

Bias-Aware Adjoint Calibration and Optimal Control of Stochastic Multiscale Bioprocess Digital Twins

cs.LG stat.ML ·2026-10-07 ·Keilung Choy, Wei Xie
similarity 0.70
Abstract · 35 pages, 8 figures

We develop a bias-aware digital-twin calibration and control framework for multiscale bioprocess models within a biological systems-of-systems (Bio-SoS) paradigm. The digital twin is represented by a stochastic differential equation (SDE) model and calibrated from sparse observations using quasi-likelihood estimation and adjoint sensitivity analysis. SDE generator-based moment expansions characterize truncation-induced parameter bias, while forward-backward adjoints quantify how calibration uncertainty propagates to value functions and policy performance. The resulting parameter-error distribution supports both policy-directed data acquisition and uncertainty-aware policy optimization through a second-order Gaussian-averaged objective. We characterize the asymptotic behavior of the resulting exploration criterion and derive a physical-system performance under the optimized policy. To implement these ideas, we develop an Actor-Simulator algorithm that jointly updates model parameters, selects informative experiments, and optimizes control policies. Numerical studies demonstrate improved calibration accuracy, sample efficiency, and control performance relative to state-of-the-art baselines.

0.29

Decision-Making in Digital Twins: Optimization, Control, and Learning

eess.SY math.OC ·2026-10-12 ·Enlu Zhou
similarity 0.70
Abstract

A digital twin is a virtual representation of a physical system, continuously updated with data from the physical system, enabling dynamic information exchange and decision-making. This bidirectional interaction, which distinguishes digital twins from conventional simulation models, introduces new challenges to simulation-based optimization, control, and learning. This paper reviews recent developments in decision-making methodologies motivated by digital twins. We focus on three interconnected areas: simulation optimization, sequential decision making, and risk-aware decision making. For simulation optimization, we discuss data-driven simulation optimization under input uncertainty, streaming input data, decision-dependent observations, and limited computational and data-collection resources. For sequential decision making, we review the connections among digital twins, stochastic optimal control, and reinforcement learning, with emphasis on Bayesian approaches that update models and policies as new information becomes available. For risk-aware decision making, we discuss distributionally robust and Bayesian risk formulations as alternative approaches for addressing model uncertainty. Throughout the paper, we emphasize the common problem of making decisions while the underlying model is being updated. We conclude by identifying research challenges and opportunities.

0.27

TwinBridge: Digital Twin-Driven Policy and Reward Transfer for Real-World Reinforcement Learning in Laboratory Robotics

cs.RO ·2026-11-04 ·Shu'ang Sun, Yibo Qiu, Haoliang Ye, Zan Huang, Ao Tian +3 more
similarity 0.60
Abstract · 8 pages, 6 figures

Online reinforcement learning offers an effective approach to precise laboratory manipulation, yet costly real-world demonstrations and limited access to dense process rewards hinder its practical application. We present TwinBridge, a digital twin-driven framework that supplies policy initialization and dense visual rewards for real-world online learning. We construct digital twins by combining 3D Gaussian splatting with physics simulation. Here, simulator states provide task events and geometric progress under physical prerequisites and ordering constraints, yielding rewards for policy pretraining with simulated demonstrations and reinforcement learning. We distill this supervision into a visual reward model. During real-world interaction, its predictions are combined with execution history to compute dense rewards for human-in-the-loop policy refinement. Across five real-world laboratory tasks, TwinBridge achieves a final mean autonomous success rate of 98% with at most 25 minutes of real-world interaction per task. Experiments show that dense rewards improve interaction efficiency over sparse rewards, and simulation-based policy initialization further accelerates learning.

0.20

Ruleless Digital Twins: Toward Declarative Decision-Making Through Standardized Frameworks and Technologies

cs.SE ·2026-10-07 ·Ivan Spajić, Volker Stolz
similarity 0.60
Abstract · 47 pages (bibliography included), expanded version of the Spajić and Stolz (2026) DataMod paper

Digital twins (DTs) can be thought of as digital counterparts of physical objects, or more generally, twinning targets (TTs). To enact changes on and optimize for properties of their TTs, DTs use an element of decision-making. Traditionally, many domains, such as home automation, utilize rule-based decision-making models to imperatively define DT actions based on desired TT conditions. With evolving user specifications, rule-based models have become increasingly complex to develop and maintain, particularly in scenarios involving optimizations under dynamically changing systems, such as those affected by weather or dynamic energy pricing. We present an alternative in the form of ruleless digital twins (RDTs) that automatically produce optimal decisions with respect to purely declarative user specifications, similarly to the well-established ruleless approach of model predictive control. They do so through a combination of a semantic knowledge base, logical inference, simulation models, and an autonomic computing architecture, each exclusively based on a widely-used standard or technology. We evaluate our proof of concept through an incubator case study and experiments against a rule-based bang-bang controller in the context of a virtual office room environment. Results show core that RDT functionality performs better than the control in terms of maintaining desired room temperatures and minimizing energy costs under dynamically changing spot pricing. Additionally, due to combinatorial decision tree construction, RDTs require considerable computational power for longer prediction horizons, although their applicability to specific problem domains (and use of shorter horizons) ultimately falls on their users.

0.31

Integrating Machine Learning and Uncertainty Quantification for Real-Time Decision-Making in Digital Twins: A System-Centric Review and Perspective

cs.LG stat.AP ·2026-11-03 ·Yi-Ping Chen, Ying-Kuan Tsai, Seul Lee, Vispi Karkaria, Dex Doksoo Lee +1 more
similarity 0.60
Abstract

Digital Twins are evolving from passive virtual representations into closed-loop systems that continuously synchronize with physical assets and drive real-time decisions. Yet predictive modeling, machine learning (ML), uncertainty quantification (UQ), and decision-making have been studied as separate capabilities, leaving limited understanding of their integration and maintenance during online operation. This narrative review, scoped by the functional requirements of real-time decision-making rather than by Digital Twin terminology alone, examines how such decisions are generated and sustained in engineering Digital Twins. We first examine foundational decision-making architectures and representative engineering applications, followed by methods that address their limitations across the decision loop. We identify optimization-based, policy-based, and knowledge-based approaches as the major pathways from virtual representation to action. Across them, ML plays three roles in decision-making: enabling efficient surrogate modeling, transforming heterogeneous physical feedback into decision-relevant information, and accelerating or directly generating decisions. UQ supports uncertainty-aware decisions and model adaptation, while continual validation, drift detection, and adaptation help maintain reliable operation as the physical system evolves. Together, these findings underscore the need for a shift from a model-centric to a system-centric perspective on Digital Twins, where its performance depends on the interactions with sensing, state estimation, computation, decision-making, and adaptation rather than on their standalone performance. Therefore, future research should shift from isolated algorithm development toward system-level co-design, scalable decision-making, and approaches to design, and data-efficient continual adaptation and validation for AI-enabled autonomous systems.

0.15

NeuronSifter: An Agent-in-Twin for Prioritizing Central Nervous System Interventions with World Action Models

cs.LG q-bio.NC ·2026-09-28 ·Haowei Xu, Wanyi Fu, Hongbin Han, Zhaoheng Xie
code/data similarity 0.59
Abstract · The submission was made without obtaining authorization from all co-authors

Effective central nervous system (CNS) intervention requires more than predicting biological responses: a scientific system must determine which intervention, observation, or additional evidence can improve a decision under uncertainty. However, in partially observed neuronal microenvironments, identical predicted outcomes may correspond to different exposure states, biological mechanisms, and intervention risks. We propose NeuronSifter, a decision-centric Agent-in-Twin framework, where an AI agent operates with an evolving biological digital twin rather than a fixed predictive model. NeuronSifter compiles interventions into state-conditional target-occupancy representations, propagates them through a mechanism-grounded World Action Model (WAM), and selects informative measurements based on their expected reduction in decision uncertainty. A shared probabilistic belief state couples intervention simulation, observation selection, and outcome assimilation, enabling the digital twin to continuously refine its decision-relevant representations through externally verified evidence. We evaluate NeuronSifter in Alzheimer's disease intervention scenarios using mechanistic neuronal microenvironment simulations and retrospective clinical endpoint references. The evaluation framework examines adaptive evidence acquisition, uncertainty-aware intervention planning, and robustness under model discrepancy through controlled environments and independent comparisons. NeuronSifter demonstrates how an AI system can actively identify missing evidence, refine biological hypotheses, and optimize intervention decisions under limited experimental budgets. Code is available at https://anonymous.4open.science/r/NeuronSifter.

0.35

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

cs.LG ·2026-10-22 ·Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman, Samuel Okafor
similarity 0.58
Abstract · arXiv admin note: This submission has been withdrawn by arXiv administrators due to incorrect author affiliation listing

Agentic reinforcement learning rewards a complete episode, yet the policy needs credit for every decision inside it. Counterfactual branching estimates this credit by re-running the episode after an alternative decision, but the branch draws its sampling noise independently of the factual episode, so the outcome difference mixes the effect of the decision with the luck of everything that follows. We observe that in agentic search the retriever, the corpus, and the grader are deterministic, so an episode is a deterministic function of the policy's decisions and its sampling noise. Twin worlds are therefore realizable: the same noise, a different decision. We introduce TWICE (TWIn-world Credit Estimation), which keys the Gumbel noise of every token by turn and line, rolls out counterfactual twins that inherit the factual episode's downstream noise, and turns their paired outcome differences into turn-level advantages for policy-gradient updates on actor-sampled trajectories; only the actor is deployed. Twin credit is unbiased for any coupling quality, its variance is that of an independent counterfactual scaled by $1-ρ$, where $ρ$ is the twins' outcome correlation, and under coupling the number of rollouts needed to resolve an advantage of size $Δ$ falls from $Θ(Δ^{-2})$ to $Θ(Δ^{-1})$. A square-root rule spends the twin budget where outcomes are predicted to disagree per unit cost. On seven question-answering benchmarks, TWICE improves the average exact match of GRPO by 5.3 and 4.5 points on Qwen2.5-3B and 7B, outperforms tree-search and counterfactual-branching baselines, and uses 68% fewer auxiliary tokens than VinePPO.

0.13

Offline Reinforcement Learning for Wind Farm Control: A Wind Tunnel Study under Dynamic Wind Directions

cs.LG ·2026-09-11 ·Yuhan Su, Hongyang Dong, Simone Tamaro, Filippo Campagnolo, Carlo L. Bottasso +1 more
benchmark similarity 0.58
Abstract

This paper addresses the wind farm power maximization problem in the presence of wind direction changes. Specifically, a model-free Modified Twin Delayed Deep Deterministic Policy Gradient with Behavior Cloning (MTD3-BC) algorithm is proposed to tackle this task through yaw control under varying wind direction conditions. MTD3-BC is an offline reinforcement learning (RL) algorithm that aims to infer good behavior from only a precollected offline dataset. Additionally, to ensure smooth and moderate yaw adjustments, a new action consistency term is introduced into the policy optimization objective. Unlike online RL methods, MTD3-BC does not require extensive interactions with a wind farm simulator during training, significantly reducing computational costs and training time. A wind tunnel experiment is conducted to validate the effectiveness of the algorithm under varying wind directions. The results demonstrate that MTD3-BC successfully mitigates wake effects, delivering farm-level power gains of approximately 10\% over the baseline greedy strategy and performance on par with a data-calibrated model-based wake-steering benchmark, while requiring no wake model and only a small fraction of the training cost of online RL. To our knowledge, this is the first time an offline RL wind farm control policy has been validated and demonstrated experimentally.

0.30