CS Paper Radar

On the Optimality of Implicit Feedback in Markov Decision Processes over Imperfect Links

Romain Planchon, Federico Chiariotti, Federico Mason, Andrea Zanella, Deniz Gunduz

eess.SY ·submitted 2026-11-04 11:26 UTC · v1
hot direction

Abstract

Goal-oriented communication protocols are explicitly designed to support the objectives of target applications, such as the monitoring and control of remote processes, while minimizing data traffic. This study considers a transmitter node that observes the environment state and sends it to a remote agent, which then performs control actions. This problem can be modeled as a remote Partially Observable Markov Decision Process (POMDP), a subclass of decentralized POMDPs. Existing studies limit transmissions through communication costs, but neglect the effect of channel impairments and non-ideal or delayed feedback. In this work, we study the optimal joint transmission and control policies over lossy forward channels without explicit acknowledgments. We introduce \emph{revealing} policies, under which the receiver's action, observed directly by the transmitter, discloses the outcome of the forward transmission, keeping the two agents synchronized without a feedback channel. We derive sufficient conditions for the optimality of time-invariant policies in this implicit feedback regime for general POMDPs, along with strictly weaker sufficient conditions in the special case of contextual bandits. Our simulations over a family of remote POMDPs show that these conditions are conservative, and revealing policies are optimal over a substantially wider parameter range than the conditions warrant.

Comment
Submitted to IEEE INFOCOM 2027

Closest papers

Nearest neighbours by meaning across the whole index.

Fundamental Tradeoffs in Sampling, Transmission and Information Freshness: Instantaneous Feedback

cs.IT cs.NIeess.SY ·2026-11-16 ·Subhankar Banerjee, Sennur Ulukus
theory similarity 0.65
Abstract

We study a time-slotted generate-at-will status-update system over an erasure channel with transmission success probability $q$, subject to average sampling- and transmission-rate constraints $f_s$ and $f_r$, respectively. The sampler-transmitter pair jointly decides whether to idle, retransmit a stale packet, or replace it with a fresh sample. We derive a universal lower bound on the optimal average age of information, analyze a class of stationary randomized policies, and obtain an optimal policy within this class that provides a $4$-approximation guarantee. The joint problem reduces to the sampling-constrained problem when $f_s\leq qf_r$ and the transmission-constrained problem when $f_s\geq f_r$. For $qf_r<f_s<f_r$, we formulate a constrained Markov decision process (CMDP) and introduce the class $\barΠ$ of $(W,R)$ policies, where $W$ specifies a waiting threshold for transmission and $R$ specifies a packet-replacement threshold. We prove that, for every choice of Lagrange multipliers, the corresponding Lagrangian-relaxed Markov decision process admits an optimal policy in $\barΠ$. We then formulate a linear program (LP) over distributions of $(W,R)$ policies and prove that choosing a policy according to an optimal LP solution yields an optimal CMDP policy. Moreover, there exists an optimal LP solution assigning positive probability to at most three $(W,R)$ policies. Since $\barΠ$ is countably infinite, for every $ε>0$ we construct a finite subset $\barΠ_ε\subset\barΠ$ and show that the LP restricted to this set yields an $ε$-optimal policy. Finally, we derive a stronger pair of lower bounds by considering systems with only the sampling-rate or transmission-rate constraint. Using these bounds, we construct a policy randomizing among four $(W,R)$ policies with a $2$-approximation guarantee for $qf_r<f_s<f_r$.

0.12

Semantic Freshness Optimal Sampling and Transmission for Gossiping Receivers

cs.IT cs.NIeess.SP ·2026-08-31 ·Irtiza Hasan, Ahmed Arafa
publishedtheory similarity 0.65
Abstract · To appear in IEEE ITW 2026

We study the optimal joint sampling and transmission policy for a transmitter communicating with two gossiping receivers that share information with each other, with the objective of tracking a source under the Version Age of Information (VAoI) metric. The transmitter can observe source-version changes, but it has to pay a sampling cost to get the current source information content. Similarly, it can communicate with a receiver by paying a transmission cost. Gossiping enables local information exchange and is able to reduce costly direct transmissions. With imperfect communication links, we formulate an infinite-horizon average-cost Markov Decision Process (MDP) to jointly minimize receiver VAoI, sampling cost, and transmission cost. Using Relative Value Iteration (RVI), we evaluate the optimal policy and establish several properties of its structure. We prove that sampling has a threshold structure in the transmitter VAoI. Among direct transmissions, it is optimal to serve the older receiver. We further characterize the transmit or idle decision through the receiver VAoI difference. Our analysis shows that link reliability and receiver VAoI imbalance have a significant effect on the optimal policy structure. Numerical results verify the structural properties and demonstrate the performance gains of the optimal policy over multiple baselines.

0.47

Minimizing Age of Incorrect Information in Pull-Based Remote Estimation of a Markov Source

cs.IT cs.NImath.PR ·2026-10-15 ·Ismail Cosandal, Sennur Ulukus, Nail Akar
similarity 0.65
Abstract

In this paper, we consider a discrete-time pull-based remote estimation system involving an information source and a remote monitor where the information source is a finite-state irreducible discrete-time Markov chain (DTMC) process. In this system, the remote monitor sends a pull request towards the source which responds by sampling its process so as to transmit the sampled values towards the monitor through a delay channel, encapsulated in information packets. We consider i) zero delay (ZD) and ii) unit delay (UD) channels, where the packet transmission times are zero and one slot, respectively. For remote estimation, we use the maximum a posteriori (MAP) estimator, and for quantifying its performance, we use the mean age of incorrect information (MAoII) which is a recently proposed freshness and mismatch metric that penalizes an incorrect estimation along with its duration. Our goal is to obtain pull policies so as to minimize MAoII under a long-run sampling rate constraint. For the ZD model, we propose a semi-Markov decision process (SMDP) formulation of the problem, which allows us to find the optimum observation-dependent pull policy. On the other hand, the UD channel prevents the monitor from possessing perfect information on the actual AoII process at any given time, and the SMDP formulation of the ZD channel cannot readily be used for the UD model. For the system with the UD channel, we first provide a numerical algorithm to calculate the MAoII and the average sampling rate for a given observation-dependent pull policy. For the optimization of the underlying remote estimation system, we use the joint distribution of the age and source processes to be obtained from the history of all observations, termed as belief. Subsequently, we propose a belief-dependent policy obtained via the belief-MDP formulation and a constrained deep reinforcement learning (CDRL) algorithm.

0.11

Joint Freshness and Age-Dispersion Control over Finite-State Markov Wireless Channels

eess.SY cs.NIcs.PF ·2026-09-15 ·Aresh Dadlani, Hina Tabassum, Muthukrishnan Senthil Kumar, Masoumeh Moradian
similarity 0.63
Abstract · 6 pages, 5 figures

Age of information (AoI) has become a standard design objective for timely monitoring as it measures the freshness of the latest update at a receiver. AoI alone, however, is insufficient in goal-oriented applications where decisions depend on consecutive observations. Age dispersion complements AoI by measuring the generation-time separation between consecutive updates. In this paper, we study joint freshness and dispersion control for a generate-at-will status update link modeled as a finite-state Markov wireless channel. The objective is to minimize the long-run probability that either AoI or age dispersion exceeds a prescribed threshold under an average transmission rate constraint. For channel-dependent randomized transmission policies, we derive exact matrix expressions for the stationary joint distribution of AoI and dispersion. For reversible channels, mean dispersion is lower bounded by the reciprocal of the delivery throughput, and the difference is an explicit non-negative variance term for which we establish the exact equality condition. We then formulate the adaptive joint-threshold control problem as a constrained Markov decision process and prove that it admits an exact finite-state representation whose size scales linearly with the number of channel states and the threshold levels. Simulation results show that the controller reduces joint threshold violations by 37.3% relative to adaptive AoI-only control under the same transmission budget, while increasing mean AoI by 8.3%.

0.15

Age of Incorrect Information for Pull-Based State Estimation of General Markov Sources

cs.IT cs.NI ·2026-08-13 ·Marco Zanni, Mohamad Assaad, Touraj Soleymani
similarity 0.63
Abstract

We study pull-based remote state estimation of an arbitrary, multi-state Markov source while accounting for both freshness and correctness attributes of information. To that end, we formulate a discounted optimization problem in terms of the age of incorrect information (AoII), and express it as a joint source-AoII belief Markov decision process (MDP) under maximum a posteriori (MAP) estimation. We then exploit the information structure of the model and prove that every reachable belief is represented by the last successfully observed source state and the number of time slots elapsed since that observation. For numerical computation, we truncate the elapsed no-success duration at a finite level and derive an explicit error bound and a criterion for selecting the truncation parameter. For reliable links, we show that an optimal policy can be represented by a look-up table of waiting times. For unreliable links, we propose a persistent policy and derive computable performance bounds. We also show that the MAP estimate stabilizes after a finite number of time slots. To further reduce memory requirements, we introduce a hybrid estimator with an early stationary switch and derive a computable bound on the resulting difference in performance. Finally, we extend the framework to multiple sources, formulate the scheduling problem as a restless multi-armed bandit, establish a sufficient condition for indexability, and develop an approximate Whittle index policy based on interpolation. Our numerical results illustrate the structure of the optimal single-source policy, evaluate the performance of the multi-source policies, and verify that the proposed heuristic policies closely approach the optimal solution while substantially reducing computational efforts.

0.04

Embedding Information in Policy Randomness: Distribution-Preserving Implicit Communication in MDPs

cs.IT cs.AI ·2026-10-11 ·Gongpu Chen, Deniz Gündüz
similarity 0.62
Abstract · 18 pages, 6 figures

This paper investigates a new role of randomness in stochastic policies--as a medium for embedding and communicating information. Specifically, we consider an agent interacting with an environment according to a prescribed stochastic policy. By carefully modulating the random action realizations of the policy, information can be embedded into the agent's behavior without altering the prescribed action distribution. We cast this problem as a channel coding problem, in which the controlled system implicitly serves as a communication channel. We then propose a coding scheme, termed posterior-matching randomness modulation (PRAM), that selects actions according to the message to be transmitted. With the aid of common randomness independent of the control system, PRAM preserves the prescribed action distribution for every state and every message. We further prove that PRAM is asymptotically reliable: the message can be decoded with arbitrarily small error probability as the observed state trajectory becomes sufficiently long. Experiments demonstrate the effectiveness of the proposed framework in applications including multi-bit policy watermarking and trajectory provenance.

0.28

Legitimate Reconstruction Advantage in Wiretapped Semantics-Aware Remote Monitoring

cs.IT ·2026-10-15 ·Bowen Li, Nikolaos Pappas
similarity 0.61
Abstract

This paper investigates joint accuracy and security in semantics-aware remote monitoring, where source updates intended for a legitimate receiver may also be intercepted by an eavesdropper. We define legitimate reconstruction advantage (LRA) as the joint event that the intended receiver correctly reconstructs the current source state using received updates while the eavesdropper reconstructs it incorrectly. Source evolution changes the relevance of received updates at the two receivers, so their marginal reconstruction accuracies do not, in general, determine the probability of this joint event. Under randomized stationary (RS) transmission and zero-order-hold reconstruction at both receivers, we derive an explicit joint stationary distribution and closed-form expressions for LRA probability and persistence for general finite-state ergodic Markov sources. For two-state and uniform-transition sources, we obtain algebraic expressions that enable low-complexity optimization of RS policies. Analytical and numerical results show that maximizing the probability of legitimate advantage need not maximize its persistence. For two-state sources, rare-transmission analysis reveals that, at a fixed sum of transition probabilities, greater transition asymmetry can increase persistence while reducing LRA probability. Moreover, optimizing marginal accuracy and security metrics can select policies that are suboptimal for LRA. Counterintuitively, more frequent transmission or improved legitimate-channel reliability can increase legitimate reconstruction accuracy while reducing LRA probability. Finally, a geofencing application translates prescribed LRA requirements into spatial constraints on eavesdropper locations.

0.21

Fundamental Limits of Dynamic Tracking and Communication: A Stationary Policy Formulation

cs.IT ·2026-09-28 ·Mohammadreza Bakhshizadeh Mohajer, Luca Barletta, Alex Dytso, Giuseppe Caire
similarity 0.61
Abstract · 17 pages, 5 figures, submitted to IEEE Transactions on Information Theory

We study a dynamic integrated sensing-and-communication problem in which a multi-antenna transmitter communicates with a receiver while simultaneously tracking a moving target governed by a stable Gauss--Markov model. We formulate the sensing--communication trade-off as a rate--distortion problem, where the communication rate is defined through the Verdú--Han information spectrum and the sensing distortion is measured by the long-run minimum mean square tracking error in angle and distance. We construct a tractable outer region by lower bounding this distortion through the posterior Cramér--Rao bound (PCRB). The control problem is partially observed because the target state is not directly available to the transmitter. We therefore introduce a belief-augmented information state and show, under the stated regularity assumptions, that stationary randomized Markov policies suffice for the covariance-based scalarized problem used in the outer bound. We then derive, under the stated large-block conditions, a Gaussian covariance-control outer bound by showing that, in the large-block regime, the sensing belief transition becomes asymptotically covariance sufficient; consequently, Gaussian signaling with the same stationary covariance policy preserves the stationary PCRB asymptotically while maximizing the communication reward for each transmit covariance. Numerical experiments compare an extended Kalman filter-based achievable frontier with a PCRB-based Gaussian covariance-control outer frontier.

0.21