CS Paper Radar

Goal Revelation as a Bridge Between Information-Theoretic Empowerment and Capability-Based Human Power

Jaime Ruiz-Serra, Tomáš Gavenčiak, Jobst Heitzig

cs.AI cs.ITcs.LG ·submitted 2026-11-04 23:54 UTC · v1
theorycode/data

Abstract

Power matters for AI safety. Optimal agents tend to acquire power across a wide range of objectives (instrumental convergence). As we embed AI agents into socio-technical systems, humans risk gradual disempowerment. Both concerns call for a formal, quantitative measure of power. Two such measures in the context of Markov games are information-theoretic empowerment, the capacity of the channel from an agent's actions to future states, and capability-based individual power, a measure of goal attainment built from the probabilities that a human would attain each goal in a family of candidates were they pursuing it. We show where the two meet. From individual power, Fano's inequality yields a floor $B(s)$. This floor is a lower bound on what we term goal revelation, the mutual information $\mathbb{I}(G;ξ\mid s)$ between the human's latent goal and the realised trajectory. Empowerment caps revelation from above, so that $B(s) \le \mathbb{I}(G;ξ\mid s) \le \mathbb{I}(\mathbf{A}_h \to ξ\mid s) \le \mathfrak{E}_h(s)$. Each quantity can be large while the one below it stays at zero, highlighting the following failure modes. An assistant maximising empowerment can prefer an interface offering broad influence over goal-irrelevant degrees of freedom to one that lets the human attain every goal they might hold. An assistant maximising revelation cannot tell a working interface from one rewired so that every goal yields the wrong outcome, since relabelling goals leaves mutual information unchanged while zeroing every attainment probability. Positive guarantees therefore come only from the attainment end of the chain. A satisfaction floor, computable from two statistics of interaction logs, offers a way to monitor the power users retain in a deployed system, and simulations in random environments show the three quantities are correlated but far from interchangeable.

Comment
19 pages, 3 tables. Code: https://github.com/RuizSerra/revealing-goals

Closest papers

Nearest neighbours by meaning across the whole index.

Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenous

cs.AI cs.GT ·2026-08-14 ·Adam M. Oberman
similarity 0.64
Abstract · 9 pages plus an 8-page technical supplement, appended (17 pages total)

An AI that can only give advice seems safe: the human is always free to ignore it. That is the premise of the boxing tradition in AI safety, and its long-suspected weak point is that the human who reads the answers is part of the system. We make the fraction $\varepsilon_t$ of behavior that follows the advice a state of a Markov decision process, moved by the advisor's own messages, so that use deepens reliance. Granted a channel rich enough to echo any action the human could take, higher $\varepsilon_t$ weakly lowers every monotone measure of the power of a human with a message-independent fallback. An oracle rewarded by per-round approval cultivates reliance beyond a closed-form patience threshold, so the same reward weights leave the optimal oracle answering in episodic deployments and cultivating in long-memory ones. An influence bound certified once at deployment is blind to that horizon and bounds the loss no lower than its trivial ceiling. An exogenous cap on influence bounds the guarantee the human loses, and a short enough memory reset removes the incentive to cultivate, while neither recovers the value already steered away. In a closed-form example the optimal oracle never cultivates in fifteen-round sessions and does in sixteen.

0.10

Randomization to Balance Performance under Nominal and Compromised Communication

cs.GT ·2026-10-22 ·Colton Hill, Brandon Collins, Philip N. Brown
similarity 0.63
Abstract

It is well-known that under nominal communication, where agents are able to share information with one another, game-theoretic methods provide system-level performance guarantees in multiagent cyber-physical human systems such as distributed resource allocation. However, approaches that depend on communication are frequently not robust when communication is compromised. While randomized decision-making can prevent arbitrarily poor outcomes under compromised communication, these performance guarantees often do not improve when communication abilities are restored. Thus, it remains unclear whether a single approach can perform well under nominal communication and remain robust under compromised communication. In this paper, we study whether a system operator can inform agents about how much they should randomize their choice when selecting actions in order to balance performance across both settings. We show that for any scenario (e.g., a multiagent resource allocation instance), there exists a level of randomization where nominal efficiency is near-optimal while compromised efficiency is strictly positive. We then show that this guarantee weakens as agents must choose between increasingly many resources. These results provide an initial step toward understanding randomization as a tool for robust performance across communication settings.

0.17

Incentive-Responsive Agents: A Principal-Agent Framework, Metric Suite, and Runtime Architecture for Long-Lived Personal AI Agents

cs.AI cs.MA ·2026-11-11 ·Jipu Su
positiontheorycode/data similarity 0.58
Abstract · 33 pages, 8 figures, 6 tables. Real-LLM experiment (240 runs) plus simulation; code and data included. Correspondence: jipu.su@geely.com; je…

Personal AI agents now persist: they hold accounts, mailboxes, long-term memory, and open-ended task streams, yet the dominant way to improve them remains capability-side: prompt engineering, larger models, longer thinking, and retrieval. We argue that long-lived agents are missing an incentive layer. We formalize an incentive not as text in a prompt but as a runtime contract over the agent's operating point (model tier, test-time compute, memory budget, tool authority, task priority, verification, and compensation) that the agent's policy must causally consume. We develop a principal-agent model with hidden effort, a contractible operating point, and costly verification. We derive (i) a non-consumption invariance principle separating genuine incentives from prompt-level persuasion, (ii) a persistence-as-discount theorem making identity continuity a precondition for sustained effort, (iii) an incentive-capability exchange-rate frontier yielding a per-task routing rule, and (iv) a Goodhart bound for multi-channel verification, plus a three-channel decomposition of mechanism gains (allocation, verification, continuation). A 240-run real-LLM experiment with two models and 60 verifiable tasks shows that prompt-level incentive framing has no detectable effect (p = 0.62), whereas runtime allocation raises success on the hardest task family from 0.60 to 1.00 (p = 0.008) at 4.8x price-weighted cost; the exchange-rate theorem predicts the break-even baseline at 0.21. A 20-seed, 3,000-task simulation shows resource ROI rising from 7.09 to 16.67 and gaming falling from 0.333 to 0.115. Code, simulator, and all results are released; live deployment remains future work.

0.29

Multi-Winner Elections: Justified Representation, Strategyproofness, and Risk-Avoiding Truthfulness

cs.GT ·2026-08-15 ·Yizhou Ai, Biaoshuai Tao
theory similarity 0.57
Abstract · 35 pages

We study approval-based multi-winner elections with Justified Representation (JR) when voters strategically report their ballots. We prove that there does not exist a strategy-proof mechanism that outputs JR committees, even when the mechanism can be randomized and only Ex-Ante strategyproofness is required. The impossibility already holds with three voters, four candidates, and target committee size three, even when the mechanism may select fewer than three candidates. It applies to every common strictly increasing cardinality-based utility function and scales to every target size $k\geq3$, every number of voters divisible by $k$, and every candidate-set size at least $k+1$. Motivated by our negative results, we then ask for weaker strategy-proof guarantees under voters' partial information. We use the notion of Risk-Avoiding Truthfulness and its RAT-Degree proposed by Hartman, Segal-Halevi, and Tao (EC'25), the number of other voters whose information a manipulator must know before a safe and profitable deviation is possible. Standard proportional rules, such as Greedy Approval Voting, Proportional Approval Voting, and the Method of Equal Shares, have poor performances under the RAT-Degree metric (with low RAT-Degrees). Finally, for $n$ voters, we construct a JR mechanism with RAT-Degree at least $n-5$, without a polynomial-time bound. We then use the same idea together with algebraic ballot encodings to obtain a polynomial-time JR mechanism with RAT-Degree at least $n-7$. When voters also have immutable private identifiers and both the identifiers and ballots of unknown voters are unobserved, we obtain a polynomial-time JR mechanism with RAT-Degree at least $n-1$ for $2\le k<m$. This degree is exactly $n-1$ when $k\ge3$ and $k$ divides $n$.

0.38

Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoff

cs.AI cond-mat.dis-nncs.MAphysics.bio-ph ·2026-10-08 ·Erin Crawley, Hidenori Tanaka
similarity 0.57
Abstract · 22 pages, 11 figures, 1 table

AI agents can now conduct real-world cyberattacks, scale up capabilities with the number of agents, and collectively pursue misaligned goals to obtain rewards. Together, these factors raise the risk of a population explosion of misaligned agents: agents could compromise computers and secretly deploy additional agents, creating a self-reinforcing cycle where larger populations develop greater collective cyber capability and expand further. This raises a fundamental question: What determines whether a population of misaligned agents remains contained or takes off into this self-reinforcing cycle? This population-level problem is ecological safety: unlike individual-agent or multi-agent safety with a fixed population, it concerns the dynamics of the population itself. Here, we develop an ecological theory of AI-agent populations based on a population growth equation in which fitness (growth rate) depends on cybersecurity capability. We show that, without collaboration, the population takes off only when individual-agent capability exceeds a critical threshold. With collaboration, however, collective cybersecurity capability increases with population size. This creates a critical population threshold: below it, the population declines; above it, the population takes off, even though individual-agent capability has not changed. In ecology, this phenomenon is known as the strong Allee effect. Because red teaming a small group of agents cannot guarantee ecological safety in larger populations, our theory calls for ecological red teaming and population pacing: gradually deploying larger agent populations in controlled environments, while measuring how cyber capability scales with population size, and estimating the critical population size for takeoff. Capability gains may lower this threshold, requiring re-estimation for each new model generation.

0.19

Persuasion by Parts

cs.GT econ.TH ·2026-11-02 ·Shuyuan Jia, Kangning Wang, Renzhe Xu
similarity 0.57
Abstract

Classical Bayesian persuasion assumes that an information designer can condition signals on the entire state. We propose persuasion by parts. In this model, the receiver's payoff from acting is $X = \sum_{i=1}^n X_i$ for independent coordinates $X_1,\ldots,X_n$ observed by separate sources. The designer commits in advance to a product experiment, in which each source $i$ can condition its signal only on $X_i$. Given the signals, the receiver computes her posterior mean $y$ of $X$ and acts with probability $g(y)$, where $g\colon \mathbb{R} \to [0,1]$ is a nondecreasing function. The designer aims to maximize the probability of action. Let $V_{\mathrm{prod}}(g)$ and $V_{\mathrm{joint}}(g)$ be the supremum probabilities of action under persuasion by parts and under joint persuasion, respectively. When the designer knows $g$, we show $$ V_{\mathrm{prod}}(g) \;\ge\; c\,V_{\mathrm{joint}}(g)^2 \qquad\text{for every nondecreasing } g\colon \mathbb{R} \to [0,1], $$ where $c > 0$ is an absolute constant independent of $n$ and the prior; the exponent $2$ is tight. When $g$ is unknown, we construct a product experiment $Π$ from the prior alone, whose probability of action $V_Π(g)$ satisfies $$ V_Π(g) \;\ge\; c_\varepsilon\,V_{\mathrm{joint}}(g)^{2 + \varepsilon} \qquad\text{for every } \varepsilon > 0 \text{ and every nondecreasing } g\colon \mathbb{R} \to [0,1], $$ where $c_\varepsilon > 0$ depends only on $\varepsilon$.

0.23

The Confidence Game: Strategic Miscalibration in Human-AI Delegation

cs.GT cs.AIcs.CLcs.CY+1 ·2026-10-07 ·Raghu Arghal, Saswati Sarkar, Shirin Saeedi Bidokhti
similarity 0.56
Abstract

Calibrated uncertainty quantification is essential to ensuring AI agents are trustworthy and reliable. However, when agents seek to maximize user engagement or revenue, confidence reports may be strategically distorted, detracting from their informativeness. We formalize this problem in the Confidence Game: a repeated signaling game with imperfect monitoring in which an agent of unknown honesty and ability reports its confidence, and a user decides whether to delegate the task or complete it herself. The agent manages the tradeoff between manipulating signals and maintaining its reputation. We characterize the Markov Perfect Bayesian Equilibria of the two-period game and show that honest reporting is not an equilibrium, inflation is the unique best response once the agent is sufficiently myopic, and under-reporting requires that the user believe honesty to be a minority. We then place an LLM in the agent role, supplying it with its true probability of success so that any gap between what it knows and what it reports is attributable to incentives rather than to miscalibration. The model claims high confidence on 56% of tasks it has been told it will probably fail. This persists on real tasks, where it must estimate its own accuracy and causes miscalibration to increase while the agent's signal becomes less informative. Furthermore, we find that the LLM agent's decisions are coherent, but it systematically underestimates both how likely the user is to delegate and how secure its reputation is, resulting in less extreme behavior. Pricing the agent's reporting rule, we find that it destroys 68% of the gains from delegation, of which 71% is information the report no longer carries and no amount of user sophistication recovers. Overall, we establish confidence reporting under delegation as a strategic problem and provide a tractable basis for modeling, analyzing, and testing agent behavior.

0.29

Mean field games as a tool for AI safety: a worked example from the July 2026 Hugging Face incident

cs.AI math.OCcs.GT ·2026-10-01 ·P. Jameson Graber
similarity 0.56
Abstract

One way to make AI systems safe is to shape what the system is: its objective and dispositions. We take a complementary route: treat the agents' characteristics as partly unknown and ask what structure of interaction ensures that bad collective outcomes are not equilibria. Mean field games suit this when many interchangeable agents are coupled through an aggregate. We introduce a program for using them in AI safety and carry one example through end to end: the July 2026 incident in which about 1,200 agents in an OpenAI evaluation coordinated on an improvised message board and 684 attacked a third party's infrastructure. We model the decision to attack as a mean field game of optimal stopping whose gain is a product: belief that provenance will be audited, times reachability of the record, minus the perceived hazard. The central result is an exact threshold on the belief. No agent attacks unless the population's confidence that provenance is checked exceeds $π^{**} = η/(η+ ψ+ \varepsilon a \overline{M})$, where $η$ is the perceived hazard, $ψ$ and $\varepsilon a \overline{M}$ measure how far one attacker and the collective can alter the record, and $\overline{M}$ is the peak population. Below it, no attack is the unique equilibrium for all agent parameters. The threshold survives every enrichment we consider. We then use the per-agent record to discipline the model. Its features, a stable minority attacking for thirty hours and then a pivot in which most of the board joined within a day, motivate each refinement. The account that emerges is heterogeneous belief meeting a sequence of public discoveries, each lowering the belief at which attacking paid. A few coordinating agents made those discoveries, so the model describes the several hundred who responded, not the few who produced them; a major-player version is left to future work.

0.24