CS Paper Radar

Embedding Information in Policy Randomness: Distribution-Preserving Implicit Communication in MDPs

Gongpu Chen, Deniz Gündüz

cs.IT cs.AI ·submitted 2026-10-11 23:20 UTC · v1

Abstract

This paper investigates a new role of randomness in stochastic policies--as a medium for embedding and communicating information. Specifically, we consider an agent interacting with an environment according to a prescribed stochastic policy. By carefully modulating the random action realizations of the policy, information can be embedded into the agent's behavior without altering the prescribed action distribution. We cast this problem as a channel coding problem, in which the controlled system implicitly serves as a communication channel. We then propose a coding scheme, termed posterior-matching randomness modulation (PRAM), that selects actions according to the message to be transmitted. With the aid of common randomness independent of the control system, PRAM preserves the prescribed action distribution for every state and every message. We further prove that PRAM is asymptotically reliable: the message can be decoded with arbitrarily small error probability as the observed state trajectory becomes sufficiently long. Experiments demonstrate the effectiveness of the proposed framework in applications including multi-bit policy watermarking and trajectory provenance.

Comment
18 pages, 6 figures

Closest papers

Nearest neighbours by meaning across the whole index.

On the Optimality of Implicit Feedback in Markov Decision Processes over Imperfect Links

eess.SY ·2026-11-04 ·Romain Planchon, Federico Chiariotti, Federico Mason, Andrea Zanella, Deniz Gunduz
hot direction similarity 0.62
Abstract · Submitted to IEEE INFOCOM 2027

Goal-oriented communication protocols are explicitly designed to support the objectives of target applications, such as the monitoring and control of remote processes, while minimizing data traffic. This study considers a transmitter node that observes the environment state and sends it to a remote agent, which then performs control actions. This problem can be modeled as a remote Partially Observable Markov Decision Process (POMDP), a subclass of decentralized POMDPs. Existing studies limit transmissions through communication costs, but neglect the effect of channel impairments and non-ideal or delayed feedback. In this work, we study the optimal joint transmission and control policies over lossy forward channels without explicit acknowledgments. We introduce \emph{revealing} policies, under which the receiver's action, observed directly by the transmitter, discloses the outcome of the forward transmission, keeping the two agents synchronized without a feedback channel. We derive sufficient conditions for the optimality of time-invariant policies in this implicit feedback regime for general POMDPs, along with strictly weaker sufficient conditions in the special case of contextual bandits. Our simulations over a family of remote POMDPs show that these conditions are conservative, and revealing policies are optimal over a substantially wider parameter range than the conditions warrant.

0.34

Authentication over Arbitrarily Varying Channels with Causal Adversaries

cs.IT ·2026-08-28 ·Mayank Bakshi, Vinod M. Prabhakaran, Bikash Kumar Dey, Oliver Kosut
theory similarity 0.61
Abstract · Extended version of a paper accepted for presentation at the 2026 IEEE Information Theory Workshop (ITW). 22 pages, 2 figures, 2 tables. Inc…

We study authentication over a discrete memoryless arbitrarily varying channel (AVC) in which the adversary selects the channel state causally based on past channel outputs. We prove that the authentication capacity is positive exactly when the channel is not distribution-overwritable under stochastic encoding, and not I-overwritable under deterministic encoding. In the stochastic case, whenever the authentication capacity is positive, it coincides with the no-adversary Shannon capacity. Both converses follow from a novel wait-and-overwrite attack: the adversary tracks the no-adversary posterior over the message via the posterior guessing probability, and once it concentrates past a threshold, it samples a guess and a decoy from the posterior and overwrites the channel to mimic a no-adversary transmission of the decoy. The positivity results are obtained by constructing positive-rate codes with controlled overlap between codewords, together with a martingale concentration argument that handles adaptive adversarial strategies. For stochastic encoding, the full-capacity achievability further combines a capacity-achieving channel code with an authentication tag. These characterizations separate the causal stochastic-code and causal deterministic-code settings from each other and from the oblivious-adversary setting studied by Kosut and Kliewer.

0.37

Inverse Design of Finite-State Channels by Dynamic Measure Matching

cs.IT ·2026-11-02 ·Lav R. Varshney
similarity 0.60
Abstract

Inspired by the optimization approach to biology, we formulate an inverse problem for communication with memory: construct a finite-state channel for which a given signaling behavior supports capacity-achieving codes. Dynamic measure matching is introduced to turn a directed-information converse into conditions on the channel dynamics. These conditions lead to a new notion of dynamic surprise: the log ratio of physical to test output probabilities with a continuation correction for future signaling opportunities. The matching inequalities bound its conditional mean. For fixed architecture, test law, and potential, channel row synthesis under convex physical restrictions is convex. For every signaling constraint represented by a finite irreducible constrained graph of positive entropy, we construct a channel law under which undesired inputs remain available and behaviorally distinct while incurring a strict information penalty. A noisy refractory family has an exact capacity plateau, followed by an interval in which Shannon and zero-error feedback capacities differ. Explicit zero-error codes attain capacity in the graph construction and on the plateau. In this family, a Gibbs choice maximizes the penalty and makes dynamic surprise equal its conditional mean on every transition, yielding an exact trajectory-wise likelihood identity, a finite-block bound on successful decoding with premature requests, and an exponential strong converse. An exact gap decomposition into action residuals and output-law divergence constrains every reliable capacity-approaching code. Within the strict plateau, positive discrimination exponents for unused dynamics require a positive rate loss, while sparse probing permits consistent estimation at asymptotic capacity. A chemical channel illustrates how premature stimulation can alter future responsiveness while leaving the immediate output law unchanged.

0.21

Finite-Memory Control of POMDPs: Fundamental Limits and Efficient Design

cs.LG cs.IT ·2026-11-02 ·Emrecan Kutay, Atilla Eryilmaz, Ness Shroff
top venuetheory similarity 0.59
Abstract · Accepted at NeurIPS 2026

Optimal control of partially observable Markov decision processes (POMDPs), in general, requires a complete history. The belief state, a distribution over hidden states, summarizes this history and provides the classical sufficient statistic to control POMDPs. However, the continuous nature of the belief space generally prevents a finite-memory controller from maintaining it exactly. This raises the question we address in this work: what must the available finite memory preserve to achieve near-optimal control performance? We study this question through an information-theoretic converse that gives a fundamental lower bound on the loss of any finite-memory controller. The analysis introduces the concept of a decision witness that characterizes belief states whose confusion would lead to unavoidable loss. Motivated by this bound, we propose RECAP, a constructive quantization method for designing representative memory states. We prove an upper bound on the optimality gap of any RECAP codebook and use the components of this bound together with the converse analysis to define the codebook objective. Experiments on 11 POMDP benchmarks demonstrate that RECAP improves over finite-memory representation baselines under the same memory budget.

0.32

Revealing After Overwriting: An Exponential POMDP OPE Lower Bound under History-Dependent Logging

cs.LG ·2026-10-04 ·Youyu Luo, Pengzhan Zhou, Zhida Qin, Jia Wang, Zuotao Fu +2 more
theory similarity 0.58
Abstract

Multi-step revealing can make off-policy evaluation tractable under memoryless logging. With history-dependent logging, state decodability and target-relevant evidence can separate. For every horizon $H\ge3$, we construct two exactly realizable POMDPs with four actions, at most four states per layer, a known logger, and a memoryless target. Action overlap, history coverage, and observation-only revealing remain bounded independently of $H$, yet the target values differ by $1/2$ and the KL divergence between the logged laws is $Θ(4^{-(H-1)})$, forcing exponential sample complexity. Logger memory makes states distinguishable, while reset erases the model-distinguishing evidence preserved by the target. A separate construction retains this barrier with common, known observation-only revealing operators. Under action and history coverage, we give a finite-class OPE guarantee using common observable value representations that remain valid at every history. The sample bound depends polynomially on their second-moment cost. In the common-operator construction, the same value direction has constant marginal decoding cost but exponential history-conditioned cost. Finally, on a fixed four-action continuum, we derive matching passive and budgeted readout rates. With one known channel and unit read cost, early reads are optimal. With unknown sensor bias, early reads alone remain exponentially costly. Combining them with post-reset calibration gives sample complexity independent of $H$ when both read types receive fixed positive expected budgets per trajectory.

0.15

Truth Revelation, Information Hiding, or Misinformation: Characterization of Equilibrium Outcomes in Signaling Games

cs.GT cs.ITmath.OC ·2026-08-25 ·Ertan Kazıklı, Sinan Gezici, Serdar Yüksel
similarity 0.58
Abstract · 18 pages and 4 figures

In signaling games where a sender and a receiver have misaligned criteria, equilibrium behavior may lead to fully revealing, quantized, or randomized policies. Notably, the first arises in statistical decision theory and classical communication theoretic problems involving a fully aligned sensor and receiver, the second arises in Nash theoretic simultaneous signaling games, and the last may appear in Stackelberg type (leader-follower) Bayesian signaling games. In this paper, we investigate the Bayesian persuasion problem involving a receiver that tries to estimate the source. We show that for certain payoff structures, the equilibrium solution is such that a source observation is mapped to distinct messages with nonzero probabilities. More specifically, we completely characterize conditions under which the sender requires randomization for the Bayesian persuasion problem involving general sources with finite cardinality. In particular, regardless of whether the equilibrium solution under a deterministic policy restriction is fully revealing, quantized or noninformative, there exists a randomized sender policy that improves the sender's payoff under certain conditions characterized in the paper. Moreover, we provide an algorithmic procedure to obtain the Bayesian persuasion solution, where the algorithm compares the payoffs with finitely many posterior probability combinations. We also consider fully aligned and completely misaligned payoff structures, where the solutions respectively involve a fully revealing sender and a noninformative sender. Then, we unify these results by proving that if the sender's expected payoff with respect to posterior distributions is continuous, then the equilibrium solution involves either a fully revealing sender or a noninformative sender.

0.20

Belief-State Engine: Augmenting LLMs for Principled Planning Under Partial Observability

cs.AI cs.LGcs.RO ·2026-09-09 ·Arnab Chattopadhayay, Debdipta Halder
similarity 0.57
Abstract · Total number of pages: 19, total number of figures: 5

Large language model agents produce fluent action sequences across a wide range of tasks, yet they fail in characteristic ways once the environment becomes partially observable. Ambiguous feedback pushes them into premature commitments. A single informative observation can collapse their uncertainty onto the wrong hypothesis. Policies drift as the history grows. We trace these symptoms to a common structural cause. An LLM agent, as commonly deployed, is a history-conditioned policy with no explicit belief over hidden state. We propose an architectural fix. The Belief-State Engine (BSE) is an inference module placed outside the LLM. It maintains a Bayesian posterior over the latent states of a given POMDP (Partially Observable Markov Decision Process) model, and at each decision step it exposes only that posterior to the LLM. The raw action-observation log is not shown. We set out a minimal four-axiom specification of what a belief-consistent internal state must satisfy, and prove that the LLM paired with the BSE is a sound Markov policy on the belief MDP induced by the underlying POMDP. It therefore inherits the Bellman optimality guarantees of classical POMDP theory, provided the LLM is never exposed to the raw history. We evaluate the architecture on the Tiger POMDP and a red-team attack-graph task, against six baselines: a reactive LLM, Chain-of-Thought, ReAct, a natural-language belief tracker, QMDP, and POMCP. Across both domains, the BSE-augmented agent improves task return, belief calibration, and decision consistency. Ten targeted ablations isolate the contribution of each architectural choice confirms that the effect is not specific to any one model. Code, environment specifications, prompt templates, and seed logs accompany this paper.

0.09

Minimal Recurrent Behavioral Memory for Imitation under Partial Observability

cs.LG cs.ITcs.RO ·2026-09-22 ·Xianyao Li, Fang Xu, Rui Min, Ruitong Tian, Jing Du
code/data similarity 0.57
Abstract · 46 pages, 10 figures. Code: https://github.com/XianyaoLi/DIACRITIC

What is the least recurrent memory needed to reproduce a specified expert under partial observability? The instantaneous requirement is the conditional entropy of the expert's behavioral quotient, but recurrence must also preserve distinctions that future observations will not restore before use. We characterize this minimal recurrent behavioral memory by a compatibility relation: under transitivity its classes attain the exact minimum, while the general case is an entropy minimization over closed compatible state assignments, with exact certificates on finite instances. A sole-carrier measurement protocol separates behavioral sufficiency, excess code rate, and information carried by observations or other memory paths; experimental bit requirements refer to the induced symbolic behavioral model under the stated occupancy. Across manipulation tasks, learned code rates remain near zero- and two-bit requirements as hidden modes grow to $512$, and anticipatory memory follows a $2\to1\to0$ requirement despite zero instantaneous demand during waiting. Learning this representation remains difficult: event-agnostic future-behavior supervision yields $36/40$ sufficient seeds with one frozen configuration and improves the longest-horizon pixel setting from $0/8$ to $6/8$ sufficient held-out seeds (closed-loop success from $0.08$ to $0.57$). On unmodified community benchmarks, the protocol certifies delay-independent requirements, which sufficient codes match at mid-delay. The supervision aids commitment but can induce predictive surplus; annealing it lets imitation and rate training reduce that surplus, separating the information-theoretic target from the ability to learn it.

0.49