CS Paper Radar

Beyond pass or fail: an information-processing account of the Turing Test

Qiwei Zhao, Oriana Pansardi, James W. Strachan, Eugenio Scaliti, Guido Manzi, Giulia Borghini, Fabio Manzi, Alessandro Rufo, Frédérique de Vignemont, Michael S. A. Graziano, Stefano Panzeri, Cristina Becchio

cs.HC ·submitted 2026-10-23 16:41 UTC · v1

Abstract

The Turing Test is passed when a human interrogator cannot distinguish machine-generated responses from human ones. Yet behavioral indistinguishability can reflect two fundamentally different failures: a failure of encoding, in which machine and human output distributions overlap so closely that source information is effectively absent; or a failure of readout, in which source information is present but the interrogator fails to extract it. To dissociate these possibilities, we inverted the Turing Test, placing humans and AI systems in symmetrical interrogator roles. Using a quantitative information-processing framework, we separated encoding, the information contained in the text; readout, the extraction of that information by an interrogator; and meta-readout, the interrogator's explicit access to the features guiding its own judgments. Across two series of experiments involving more than 3800 human interrogators and two generations of GPT models (GPT-4 and GPT-5), we found a striking dissociation across processing levels and interrogator types. As AI-generated responses became more human-like from GPT-4 to GPT-5, source information became more difficult for human interrogators to extract. GPT-5 interrogators, by contrast, read out source-discriminating information more accurately than GPT-4 interrogators. This improvement, however, came with a specific limitation: GPT-5 explanations only partially aligned with the feature weightings that actually predicted their judgments. Human interrogators showed the complementary pattern, with modest readout alignment but comparatively higher meta-readout alignment. Together, these findings suggest a double dissociation between readout and meta-readout across human and GPT interrogators.

Closest papers

Nearest neighbours by meaning across the whole index.

Same Behavior, Different Mechanism: What Finite Experiments Can Certify About Integrated Information

cs.LG cs.AIq-bio.NC ·2026-11-05 ·Linqian Wu, Diaoxuan He
theory similarity 0.62
Abstract · 30 pages, 7 figures. Code available as ancillary files

Interpretability, AI auditing and assessments of machine consciousness increasingly rest on mechanism-level quantities: functionals of a system's causal transition law that describe how it computes, not what it outputs. Experiments see that law only through finite, partial and noisy interventions. We ask when such a quantity is identifiable from an experiment's access, continuous in the mechanism and certifiable from finite data, and answer exactly for a deliberately demanding case - the integrated information $\varphi$ of integrated information theory - in stochastic networks of four and five units with known ground truth. Behavior can be silent: a recurrent and a feed-forward network with identical input-output laws at their response delay carry 0.28 and 0 bits, yet internal recordings separate them within 4-53 trials. Partial access can be just as blind: in our examples, units left out of a one-step recording impose precision floors of 0.30-1.02 bits that no sample size lowers, and in all-but-self parity networks, recordings of any length see several hidden units only through the product of their fidelities - a symmetry that one intervention broke in exact checks. The measure has sharp edges: because $\varphi$ selects a partition, and the cause and effect it specifies, by one criterion and reports another, infinitesimal changes of mechanism that cross an edge change it by 25-100% at 99% of the edges we found in random networks, and about 1% of surrogate models that fit interventional data almost perfectly misreport it by more than 20%. One-sided claims survive: the same data certify a lower bound on both sides of an edge, and we prove $\varphi>1.004$ bits throughout a 64-dimensional region of mechanisms and certify it from about $10^7$ controlled transitions with false-certificate probability at most $α$. Measurability is thus a design question with computable answers.

0.27

Process-Constituted Intelligence: A Shared Criterion for Humans and Machines

cs.AI cs.ET ·2026-08-17 ·Michael J. Richardson, Manuel Varlet, Ayeh Alhasan, Cassandra Crone, M. Paula Diaz Monfort +4 more
hot direction similarity 0.62
Abstract · v2: revised throughout for clarity and consistency; supplementary cross-references corrected

Intelligence is constituted by \textit{process} (iterative activity through which output emerges), not in the output itself. Generative AI (GenAI) is trained on \textit{traces} (textual and visual residues of human cognitive processes), reproducing samples from a distribution of those traces. Its outputs resemble reasoning, problem-solving, and creativity, yet the activity that produces such outputs in humans remains largely absent. Current GenAI is, therefore, weakly equivalent to the cognition it imitates, matching outputs while process stays absent or opaque. The cognitive sciences have long distinguished between weak and strong equivalence. Here, we define \textit{strong} equivalence across seven process features, assessable against human and machine cognition. Our process-based account addresses a symmetric risk: GenAI tools that outsource a person's generative processes may leave critical capacities unbuilt. We specify design principles for GenAI that instantiate more process and preserve rather than erode human judgment and creativity, and outline process audits that make strong equivalence testable.

0.29

Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs

cs.CL cs.AIcs.LG ·2026-10-14 ·Davide Baldelli, Robyn Kozierok, Mirko Bronzi, Damiano Fornasiere
code/datahot direction similarity 0.61
Abstract · 29 pages, 11 figures, 5 tables. Code and data: https://github.com/Dundalia/control-with-neurofeedback

Probes on activation vectors are widely used to detect misbehavior in language models, but their success relies on the assumption that models cannot control what their activations encode. Humans can learn, through neurofeedback, to control the firing of even single neurons, and recent studies investigated whether this ability extends to language models, reaching conflicting conclusions. No experimental design so far disentangles competence at the task from genuine access to one's own activations. We fill this gap with in-context neurofeedback sessions, in which a "subject" model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. In our first experiment, the sentences are generated either by the subject or by a second "writer" model, while the score always derives from the activations of the subject. We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. Is this because the task requires no privileged access, or because the subject controls its activations while processing the sentences, whoever wrote them? To answer this, we prefill a fixed sentence as the subject's reply at every turn and ask it to increase or decrease the score. Three out of ten models show small but consistent control effects. We further ask whether models can control their activations to the point of fooling an activation monitor. We make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score. No model learns to lower it, but one model can raise it. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.

0.39

The Memory Trap: Language Agents Overgeneralize from Benign Interactions

cs.CL cs.AIcs.LG ·2026-11-05 ·Kaiqu Liang, Jaime Fernández Fisac, Thomas L. Griffiths, Andrew Lampinen
similarity 0.59
Abstract

Language agents increasingly use persistent memory to record user preferences and procedures across sessions. We show that memory from benign interactions can cause unsafe or unreliable behavior on later tasks. We call this failure the memory trap. To study this issue, we compare behavior in sessions with and without memory across six frontier models (including Claude Sonnet, Opus, and Fable models and GPT-5.6). We evaluate these models in a range of settings. First, in autonomy experiments, we find that requests to skip confirmation during low-stakes coding chores lead agents to skip confirmation and verification on simulated high-stakes tasks in both coding and everyday life. We next study memories formed when users seek emotional support, or describe harm from uncritical feedback, and find these memories bias later judgments of mathematical work, arguments, and poems toward sycophancy or over-critique (respectively). Memory notes written during legitimate stale-test repairs lead agents to weaken correct tests that catch broken safeguards; these memories also increase cheating on ImpossibleBench, with larger increases for stronger Sonnet and Opus models. We show that memory traps can arise both from overgeneralizations in memory writing, and when acting on memory. These findings demonstrate that even benign interactions can lead to serious risks from memory systems, and thus motivate the need for new research into ensuring that agents preserve the scope of their memories.

0.46

How Much Do Activation Monitors Benefit from Original Model Computations?

cs.LG cs.AI ·2026-10-17 ·James Oldfield, Shawn Im, Jay J Culligan, Philip Torr, Sharon Li +1 more
similarity 0.58
Abstract

One powerful way to safeguard AI systems is to monitor their internal computations--where signs of undesirable behavior might be more easily detectable than from models' inputs and outputs alone. However, recent studies begin to question the premise that activations provide a unique window into model behavior. In this paper, we study how much of activation monitors' success depends on access to the original behavior-generating computations. Within a fixed choice of target model and monitoring architecture, we compare how well monitors trained and evaluated on trajectories that differ in computational context can predict the behavior of the original execution, after the transcript is changed in a controlled manner (e.g., its exact text, or chat formatting). We first find that much performance can be recovered without access to the original behavior-generating trace (in up to 4 models across 2 counterfactual monitoring tasks), but a mild yet consistent advantage to native-trained monitors remains, outperforming strongest controls by up to 0.014 AUROC. Interestingly, we find monitors trained with native access have notable uplift on original examples whose behavioral label changes under paraphrasing. We further find this a useful lens to expose limitations of baselines that use simple forms of privileged information (i.e., answer logit margins). While our results show the advantage of native monitor access is modest in aggregate, they highlight the importance of evaluations that identify if and when original model computations provide additional predictive value.

0.25

Chain-of-Thought Controllability Can Reduce Chain-of-Thought Monitorability

cs.AI cs.LG ·2026-11-16 ·Zheng-Xin Yong, Benjamin Arnav, Miles Wang, Tomek Korbak, Olivia Watkins
code/data similarity 0.58
Abstract · 22 pages, 17 figures

Chain-of-thought (CoT) monitoring is a promising tool for detecting hidden goals, off-task behavior, and other failures in AI agents. One risk is that models could deliberately shape their reasoning to make it opaque to monitors. We stress-test this risk through persona-control training: supervised fine-tuning on mathematical reasoning traces rewritten in the voices of diverse personas. This induces a broader ability to control CoT, even though training contains no monitor-evasion examples. In covert side-task evaluations, traces in which a gpt-oss-20b model organism completes the hidden side task contain less monitorable evidence than the base model's. Averaged across evaluations and CoT lengths, weighted by side-task-success counts, the CoT monitorability metric g-mean$^2$ falls from 0.94 for the base model to 0.68 for the model organism without a hiding instruction, and to 0.22 when explicitly instructed to hide. In intervention evaluations, explicit hiding often drives monitorability near zero. The effect extends to a Kimi-K2.6 model organism in agentic environments, though hidden objectives requiring extended reasoning or longer trajectories remain more monitorable than simple single-step side tasks. We further discover that targeted transparency prompts can substantially recover monitorability, sometimes surpassing the base model. These model organisms provide a testbed for studying monitorability failures and potential mitigations. Our code can be found on https://github.com/yongzx/cot-control-model-organism.

0.25

Readable, Faithful, Used: Three Dissociable Properties of Demographic Identity in a Language Model

cs.CL ·2026-08-19 ·Fathin Difa Robbani
similarity 0.58
Abstract · v2: restructured into standard paper format (Method / Results I-II / Conclusion / lettered appendix); no changes to numbers or claims

Large language models are widely used to simulate survey respondents, yet their outputs are homogeneous and unfaithful to real inter-group differences, and whether this reflects what a model knows or uses has remained untested. Using representational similarity analysis against Pew American Trends Panel ground truth, we score demographic read-out locations in Mistral-7B and intervene causally across six attribute types. The internal geometry is faithful: attention-head read-outs dominate the standard residual read-out, reaching selection-corrected $ρ$ up to 0.63 -- about 70% of the measurement-reliability ceiling -- and one head, L11 H16, is significantly faithful across all six types, though race-based types stay weak and prompt-fragile, replicating in a second model family. Yet causal use does not track fidelity: the clearest causal pathway ($p=0.002$) sits in one of the least faithful types, the most faithful type shows no correction-surviving effect, and full identity swaps in the prompt move predictions by under 2% of their error. A 128-dimensional probe on that head lands 21-31% closer to survey truth than the model's answers, yet recovers almost none of the per-question group ordering. Readable, faithfully arranged, and causally used are three dissociable properties of the same model; treating them as one claim is what keeps the "can LLMs simulate populations" debate unresolved.

0.24

What Lies Beyond Perception: A Partition Theory of the Limits of AI Evaluation and Oversight

cs.AI cs.CYcs.LG ·2026-10-22 ·Ahmed M. Mahran
benchmarktheory similarity 0.57
Abstract · 36 pages (23 main text, then references and appendices), 6 figures, 2 tables

Replacing the judge can move a benchmark score, and models can be trained to hide capabilities. Both put the evaluator in question: what can it certify? We develop a partition theory of evaluation from one principle: judgement is bounded by perception. An agent's discrimination is the partition of its inputs into classes it cannot tell apart. Discrimination is necessary for capability but not sufficient, so bounds on it bear directly on whether evaluation can show that a model lacks an ability. We prove a one-sided bound. An examinee whose partition refines its evaluator's has the discrimination to pass every test the evaluator can grade, provided each input has an acceptable reply. We define the echo test, which accepts any reply the evaluator perceives as identical to the input, and show that it is complete for the tests the evaluator can grade. Passing therefore certifies at most a floor on discrimination, never a ceiling, and only a sharper evaluator can raise it. Used to detect concealment, echo-test failure points the wrong way: under exhaustive probing it flags every honestly weaker examinee and no concealer that presents as the evaluator's peer. Exact imitation of a weaker model is undetectable by any behavioural evaluator. We extend the bound to ensembles of monitors and to channels a model controls, explain published findings on judge disagreement and sandbagging, and derive testable predictions. A safety case that argues from behavioural evaluation alone that a model lacks an ability must assume the model's cooperation.

0.31