CS Paper Radar

Why Finite Intelligence Works in an Infinite World: A Mathematical Theory

Mojtaba Soltanalian

cs.IT cs.LGstat.ML ·submitted 2026-11-13 18:47 UTC · v1

Abstract

An information system often has to learn before it knows what it will be asked. A finite representation can succeed without recording an infinite environment in full when those asks share collective structure, rather than demand unrelated facts scattered arbitrarily across it. We formalize this phenomenon through future-task information, the minimum retained information needed to preserve decision value relative to the available evidence. A joint decision-regret state records posterior comparisons among actions for every permitted task. Under compactness assumptions, every retained representation admits a calibrated replacement based on this state, preserving all task regrets without increasing information; the resulting information-regret curve equals an operational coding rate. Two exact boundaries govern finite preservation: weak-star closure for a fixed action count, and the singular mass of the source-evidence law when all finite action menus are admitted. Collective structure determines information requirements beyond individual-question difficulty. Families with identical source laws, pairwise task metrics, and individual-task rates can have different high-resolution exponents. Covariance-identical coordinate and parity families have different exact rate-distortion curves, with optimal parity retention characterized by a sharp Fourier-entropy inequality. For balanced nested task families, refinement sizes and weighted reach determine the rate within an additive constant, attained by prefix coding under explicit decay conditions. Nonlinear learning and computational separations distinguish preservation from efficient discovery and use. The proposed mathematical theory identifies which shared decision requirements allow finite information to remain useful for infinitely many possible purposes.

Comment
56 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond Task Success: A Theory of Representation Reuse

cs.LG ·2026-10-13 ·Zibo Meng, Andrew C. M. Austin, Alin Achim
similarity 0.66
Abstract · 42 pages, 10 figures

A learned representation is useful beyond its source task only if later tasks can make use of what the encoder has already learned. Source-task performance, however, gives no guarantee that the representation itself will remain useful once the original head is removed. We study representation reuse with respect to a specified family of future tasks. For such a family, we define the task-relevant state as the coarsest deterministic state sufficient for all of its tasks. A larger task family can require a finer state, and losing a required distinction makes at least one future task unrecoverable on a target support containing that conflict. Exact reuse also depends on how the retained information is presented to the downstream rule. We separate this into structural accessibility, which concerns whether the required distinctions remain recoverable, and coordinate consistency, which concerns whether those distinctions can be interpreted by the same downstream rule across samples. Both failure modes appear separately in the experiments. Reuse improves as task-relevant distinctions become more accessible. Reversible sample-dependent coordinate changes can sharply reduce reuse even when the representation remains recoverable, and inverting the applicable coordinate change restores performance. The same accessibility-reuse ordering appears in prospectively specified CIFAR-100 experiments with both ResNet-18 and DenseNet-121. Finally, preserving a candidate state that contains the distinctions needed by the anticipated task family improves reuse on compatible held-out tasks, with the clearest benefit when target supervision is limited. Representation reuse is therefore relative to the tasks that follow source learning: a useful representation must retain the distinctions those tasks need and make them available in a form that a shared downstream rule can use.

0.19

Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis

cs.LG cs.IT ·2026-10-21 ·Nicholas Stiles, Gregory W. Wornell
similarity 0.65
Abstract · 9 pages main text; 39 pages total. 4 figures main text; 15 figures total

Which pretraining objective yields the best representation for downstream tasks under an information budget? We formalize this problem as a multitask information bottleneck, using a mutual information constraint to model incomplete information retention. For Gaussian inputs and tasks, we characterize optimal representations and show that a weakly task-aligned specialist can outperform a reconstruction generalist at low rates but lose at high rates. We also construct task families for which every fixed Gaussian prediction objective incurs arbitrarily large relative regret, driven by reversals in optimal allocation priorities. Conversely, the average task information matrix defines an objective with a uniform relative-regret bound controlled by maximum task SNR. CelebA and dSprites experiments illustrate that compression can change which pretraining objective performs best, and even task-aware training can reverse which downstream task is better preserved as rate increases.

0.29

When Does Less Information Pay? An Exact Finite Atlas of Representation, Payoff, and Cost

cs.AI ·2026-11-12 ·Aleksandr M. Shtraukh-Adamsky, Vladimir M. Adamsky, Irina G. Voloshina, Artemiy D. Spitsyn
similarity 0.63
Abstract · 40 pages, 5 figures; supplementary material included. Code, data and Russian translation: https://doi.org/10.5281/zenodo.23868864

A representation that retains one bit about a four-state world can solve a task perfectly, while another that retains 1.5 bits on average reaches only 75% accuracy. The difference lies in which distinctions each representation preserves. We examine this relationship in a uniform four-state world by enumerating all admissible representations and binary action rules. Constructions are compared both by replacing one component while holding the others fixed and by choosing the best action rule separately for each module. We consider three tasks, their mixtures, and four separate hypothetical cost tariffs. In some comparisons, the representation retaining less world information has the higher payoff; in others, the more informative representation does. There are also pairs with different information amounts but equal payoff--that is, equal task accuracy. If a representation preserves all existing distinctions and adds new ones, its best zero-cost payoff cannot decrease. Lower-information advantages therefore depend on which distinctions are retained. Equal counts of opposing advantages under representation replacement follow from coding symmetry. In broader comparisons, the shares depend on how repeated realizations are counted. A different task set or resource price can change which member of a fixed pair performs better. These results are exact, complete censuses of the specified finite sets of constructions and comparisons. The atlas provides a reproducible worked model for examining how action constraints, counting rules, and cost assumptions affect comparisons. It illustrates established distinctions between information quantity and decision value rather than proposing a new general law of their relation. $I(W;R)$ measures retained world information, and payoff measures task accuracy.

0.39

Task-Sufficient Contraction: Source Selection for Machine Information Interfaces

cs.IT cs.LG ·2026-10-06 ·Joss Armstrong
similarity 0.61
Abstract · 19 pages, 1 figure, 1 table. An earlier version was first posted on Zenodo in September 2026 (v1: doi:10.5281/zenodo.22834532; all versions:…

A declared task can sometimes certify a reduced source before a downstream encoder, codebook, rate, distortion target, or optimizer is chosen. This paper studies when one such reduction preserves the complete downstream problem family, a property termed Task-Sufficient Contraction. The reduced source is fixed by the task before the later operating point is selected. An exact contraction allows the later problem to be solved on that source with the same result as if the full source had been retained. For a machine with a fixed set of possible actions and a fixed loss, the paper identifies a consumer-specific source by merging states only when every available action has the same regret in both. For finite action sets, replacing the richer source by this reduced source preserves the complete one-step rate-regret curve, even though the reduction is fixed before the distortion target is chosen. A second result gives an exact characterization for quadratic loss on affine feasible-action sets: the canonical reduced source is the projection onto the directions in which feasible actions can differ. Under a fixed energy budget, this becomes centered load, while retaining only the optimal water-filled action is too coarse. Earlier Information Bottleneck, semantic rate-distortion, and goal-oriented quantization results are then used to distinguish exact, architecture-conditioned, approximate, failed, and corrected contractions. The framework suggests a way for heterogeneous machines to exchange what a receiving task needs without first aligning their full internal representations.

0.32

Hilbert States for Decision-Making under Partial Observability

cs.AI cs.LG ·2026-11-16 ·Zibo Meng, Andrew C. M. Austin, Alin Achim
theory similarity 0.60
Abstract

Decision-making under partial observability can be fundamentally limited by information, even with known dynamics: distinct physical states can induce the same observation history while favoring different actions. We study three questions: when is full-state optimality unattainable, what information exactly preserves the best observation-based decision, and how does finite-data learning affect reliability? First, we characterize decision ambiguity through conditional action preferences. We give a zero-regret criterion, a positive lower bound for conflicting uniquely optimal actions on positive-probability regions with uniformly positive cost gaps, and an exact binary-action regret formula. This also shows why physical-state ambiguity alone need not cause decision loss. Second, for two actions and squared terminal-goal costs in the plane, we derive an exact inner-product representation of conditional expected cost differences using three conditional moment contrasts and a goal-dependent feature. This Hilbert state preserves observation-optimal actions without hidden-state reconstruction. Under nondegenerate history and goal variation, its three-dimensional form is minimal among exact bilinear cost-contrast representations. Third, we relate finite-data state errors to excess decision regret via goal-projected errors, action margins, and goal geometry. Physics-based simulations verify information-limited regret, exact action preservation, and the role of second moments. Held-out real-SAR experiments demonstrate feasible goal-conditioned decisions from masked images, while finite-data evaluations reveal goal-dependent reliability. Together, these results separate irreducible information loss, exact representation sufficiency, and learning-induced excess regret.

0.23

Same Behavior, Different Mechanism: What Finite Experiments Can Certify About Integrated Information

cs.LG cs.AIq-bio.NC ·2026-11-05 ·Linqian Wu, Diaoxuan He
theory similarity 0.60
Abstract · 30 pages, 7 figures. Code available as ancillary files

Interpretability, AI auditing and assessments of machine consciousness increasingly rest on mechanism-level quantities: functionals of a system's causal transition law that describe how it computes, not what it outputs. Experiments see that law only through finite, partial and noisy interventions. We ask when such a quantity is identifiable from an experiment's access, continuous in the mechanism and certifiable from finite data, and answer exactly for a deliberately demanding case - the integrated information $\varphi$ of integrated information theory - in stochastic networks of four and five units with known ground truth. Behavior can be silent: a recurrent and a feed-forward network with identical input-output laws at their response delay carry 0.28 and 0 bits, yet internal recordings separate them within 4-53 trials. Partial access can be just as blind: in our examples, units left out of a one-step recording impose precision floors of 0.30-1.02 bits that no sample size lowers, and in all-but-self parity networks, recordings of any length see several hidden units only through the product of their fidelities - a symmetry that one intervention broke in exact checks. The measure has sharp edges: because $\varphi$ selects a partition, and the cause and effect it specifies, by one criterion and reports another, infinitesimal changes of mechanism that cross an edge change it by 25-100% at 99% of the edges we found in random networks, and about 1% of surrogate models that fit interventional data almost perfectly misreport it by more than 20%. One-sided claims survive: the same data certify a lower bound on both sides of an edge, and we prove $\varphi>1.004$ bits throughout a 64-dimensional region of mechanisms and certify it from about $10^7$ controlled transitions with false-certificate probability at most $α$. Measurability is thus a design question with computable answers.

0.27

Beyond Accuracy: Information-Theoretic Decoder Evaluation Under Latent Intent

cs.LG cs.RO ·2026-11-03 ·Tasha Kim, Benjamin Ballyk, Anson Lei, Ingmar Posner, Oiwi Parker Jones
similarity 0.60
Abstract

A central challenge in assistive brain-to-robot control is decoding a user's latent intent from complex and noisy brain signals. Prior approaches evaluate decoders as single-trial classifiers with accuracy-based metrics, but this neglects the inherently sequential nature of assistive control. Since a planner must integrate intermittent, noisy decoder outputs over time, single-step accuracy can only be a proxy when predicting downstream task performance. We formalize brain-to-robot control as a partially observable Markov decision process (POMDP), and prove that there is a sharp phase transition: regret is uniformly bounded across the horizon as long as pairs of goals are all decoder-distinguishable, but grows linearly otherwise. We show that a single channel statistic, the worst-pair Chernoff information $Γ_{\min}$ sets this boundary. Then, we derive a closed-form regret predictor $U_{\mathrm{id}}(C)$, that can be computed from any confusion matrix in $O(K^2)$ time. Across two domains spanning orthogonal information channels, we show that $U_{\mathrm{id}}$ predicts cumulative planning regret beyond what accuracy can explain. We then highlight a pathological accuracy trap, in which two decoders have identical validation accuracy 35.1% on real EEG data (BNCI2014001 subject 5) but result in a difference of $9.6\times$ in held-out regret, which $U_{\mathrm{id}}$ predicts. Same-signal control confirms $U_{\mathrm{id}}$'s advantage with decoders that span independent feature channels. Ultimately, our work provides a drop-in diagnostic to evaluate and select sensors and decoders in sequential assistive control.

0.20

Memory Is Communication: The Frontier Between Remembering and Signaling

cs.AI cs.CLcs.ITcs.MA ·2026-08-17 ·Yashar Talebirad, Eden Redman, Ali Parsaee, Osmar R. Zaiane
similarity 0.59
Abstract

A bounded agent may obtain information for a decision from its own past, from peers, or from both sources. Retaining task-relevant history can reduce later communication, while a peer message can supply what memory lacks. Under limits on both resources, how should an agent allocate its information budget? Given a fixed task and decision rule, the memory and message rate pairs attaining a performance threshold form an achievable region under specified rules for using history and peer observations. We call its efficient boundary the remembering--signaling frontier. Across conditions where history permits the same maximum reduction in task loss, we hypothesize that a bounded agent will need less peer communication when it obtains a larger loss reduction from history. In preliminary referential games, target repetition coincided with shorter successful messages, while predictability from a hidden cyclic rule did not shorten them. Experiments varying memory and message rates can estimate the frontier and test this prediction across cooperative tasks.

0.10