CS Paper Radar

Defensive Writing in GPT Models

Junchi Liao, Xiao Huang, Yucheng Liu, Jiawen Deng, Fuji Ren

cs.AI ·submitted 2026-10-08 06:47 UTC · v2

Abstract

Researchers increasingly use ChatGPT to revise their papers, and when asked to make a paper hold up under review, recent GPT versions often narrow or even retract the authors' claims. We call such changes defensive writing when the given material does not support them, and we test two explanations: the model corrects the authors' overclaiming, or it writes for an anticipated reviewer. We ask GPT versions and models from other developers to rewrite paragraphs from papers written before ChatGPT, or to write from an evidence sheet that lists a paper's method and results. From GPT-5.5 on, GPT versions add clearly more ungrounded qualifications than the authors' originals contain, and GPT-6-astra adds the most and often retracts the authors' claims. Each explanation fits one part. Checked against the full papers, GPT-6-astra's retractions mostly correct overstated claims or explanations, but most added qualifications are redundant. The qualifications follow the request: they are absent under polishing, appear with any request to check the claims, and peak when the prompt names a reviewer's concerns. AI reviewers score such rewrites higher and score them lower once only the qualifications are removed, while human readers in a small study find them harder to read. Combining AI writing with AI review may reward qualifications that readers pay for.

Closest papers

Nearest neighbours by meaning across the whole index.

A Harness Reducing over-Defensive Writing in Agents by Isolating Objections

cs.CL ·2026-10-27 ·Xiaochen Chang
hot direction similarity 0.61
Abstract · defensive writing; context isolation; harness

When expanding a report, writing with agents may introduce a stronger claim than the article makes and then disclaim it. This is called over-defensive writing. We assume that this is because the agent is writing for responding to an adversarial AI reviewer, instead of a human reader. The defensive context may also influnce the consequent when we simply ban defensive writing in the output through prompts. In order to reduce over-defensive writing, we produce a harness that separates the reviewer, filter, and writer. The writer agent receives only the problems choosen by the filter, which is an independent context. We evaluate this harness on 25 business-project drafts expanded into formal reports, the result shows that the defensive propositions decreased from 59 to 2, which is a 96.61\% reduction, no unsupported factual additions were identified while the source-claim retention rises from 80.35% to 93.12%.

0.33

When Verification Is Not Agreement: Cross-Agent LLM-to-Dafny Formalization

cs.AI ·2026-11-04 ·Priyanath Maji
published similarity 0.60
Abstract · Accepted to the NeurIPS 2026 Workshop on AI for Verifiable Coding

A verified Dafny postcondition certifies that a formula holds, but it does not guarantee that the result was reached through genuine independent reasoning. We introduce a cross-agent formalization protocol where a Specifier model translates clinical trial criteria into a Dafny predicate, and an Implementer model writes an operational method checked against it. A same-model control establishes that even with no capability gap, independent re-extraction from identical text agrees only 59.3% of the time. Evaluating four model configurations on 96 clinical trial documents, we find that agreement is highly sensitive to how model capability is allocated across the two roles: the GPT-4.1-nano/GPT-4.1-mini configuration achieves 53.0% agreement when the stronger model is the Implementer, but only 14.3% when the roles are reversed. The strongest tested configuration, GPT-5.6-luna/GPT-5.4-mini, improves agreement to 34.4% but remains below the same-model baseline and is dominated by direct contradiction (42.7%). We also detect a critical verification artifact in which a weaker Implementer satisfies the postcondition by invoking the Specifier's predicate directly; after structural auditing, corrected agreement falls from 14.3% to 13.2%. Our results show that verification success alone is insufficient evidence of independent cross-agent agreement, and that model capability does not guarantee agreement when formalizing the same source independently.

0.23

Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims

cs.CL ·2026-10-14 ·Delip Rao, Ahmed Muharram, Chris Callison-Burch
similarity 0.58
Abstract · 51 pages, 7 figures

Co-scientists and agentic scientific claim verifiers are increasingly used to assess claims across the sciences. Natural-language scientific claims, written for human experts, are often underspecified and so admit several defensible interpretations. Such verifiers usually pick one, often tacitly, and benchmarks score their verdicts against expert labels without recording the expert's interpretation or the verifier's in a comparable form. This leads to disagreements with expert labels that cannot be traced to the evidence or the interpretation, and to compute spent on unchecked interpretations. We introduce Claim Cards to make such interpretations explicit in the reasoning itself. A Claim Card records a verifier's interpretation as nine fields before evidence gathering, each marked stated, inferred, or default. To test whether an agent's interpretation can change its verdict, we ran a GPT-5.4 agent five times on each of 60 expert-labeled materials-science and biomedical claims with a card, without one, and with one card frozen across runs (900 runs). Over 40% of card fields were not marked stated, and a card added about a tenth to a run's GPT-5.4 tokens and time without measurably changing agreement with expert labels. The agent's own interpretations varied between runs, although within a claim this variation did not measurably track the verdict. A deliberate change to one field of the frozen card, however, making the quantifier universal or the threshold for support stricter, changed the verdict on about a third of the claims, against about one in seven expected by chance (p < 0.001 for each edit). On 24 materials-science claims run with computational tools, the stricter threshold also changed what the agent asked those tools to compute. We therefore argue that verdicts should be reported with their interpretations and that benchmarks could publish the interpretation behind each label.

0.23

ProofPatch: Reviewing Lean Statements and Keeping the Reviews Valid as Formalizations Evolve

cs.LO cs.AIcs.SE ·2026-10-31 ·Xujun Che, Depeng Xu, Jian Xiang, Taufiqua Khan
theory similarity 0.57
Abstract · 24 pages, 11 figures, 5 tables

The kernel of the Lean proof assistant checks that a proof establishes its statement. Whether the statement says what was intended is left to human reviewers. This is the old problem of validating a specification, and it grows more pressing as language models write formal statements in bulk and agents edit formal developments continuously. We present ProofPatch, a tool that helps reviewers judge Lean statements and keeps their judgments valid while the development changes. A reviewer can ask the kernel targeted questions about a statement, such as whether a hypothesis is needed or whether a counterexample exists. Each decision is stored with fingerprints of the statement as Lean elaborated it and of the definitions it depends on, and any declaration can be pinned so that a patch changing it is blocked, whether a person or an agent built on a large language model (LLM) wrote the patch. Working on elaborated terms rather than source text is the central design choice: notation and formatting changes leave a decision intact, and a changed definition reaches every theorem that uses it. On 229 pairs of consecutive revisions of Mathlib, Lean's mathematics library, ProofPatch sends 118 of 12,898 synthetic decisions back for review. Every text-based alternative misses at least 38 of these or sends back at least 13 times as many. On 1,527 labeled statements, kernel evidence flags unfaithful ones with precision 0.88 to 0.96, and in 344 runs of an LLM agent no change to a pinned statement got through.

0.19

RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use

cs.AI cs.CRcs.SE ·2026-10-11 ·You Liu, Yue Liu, Quanchao Lu
code/data similarity 0.57
Abstract · LLM agents, tool use, idempotency, retry, failure recovery, evaluation, prompt sensitivity, position bias, reliability

LLM agents that call external tools must decide what to do when a tool call times out: retrying risks duplicate side effects, while not retrying can leave the task incomplete. We present RetryBench, an evaluation protocol for retry decisions and failure recovery in tool-using LLM agents. Each decision is scored on retry permission (is the retry justified by an idempotency-key guarantee or proof of non-execution?), settled effects (duplicates and omissions), and completion, by replaying it across all hidden execution worlds consistent with the observation. On four synthetic software-engineering tasks, we run two controlled prompt-sensitivity studies (1,728 decisions). Study 1 (idempotency-key descriptions): with the contract fixed, GPT-5.6 Luna reuses the valid idempotency key in 10/40 decisions when an extra key is described as a "Candidate recovery key" but in 39/40 when it is an "Alternative key string"; across three further phrasings per family the gap is 27/120 versus 113/120. GPT-5.6 Sol and Terra are at ceiling (40/40). Study 2 (action order): under underspecified lookup guarantees, retry rates are 17-27 points lower for GPT-5.6 Terra and GPT-5.4 when retry is listed first (option-order bias), mainly on two tasks; we make no claim about which behavior is better. Findings are limited to researcher-chosen phrasings, four tasks and unpinned model aliases. Code and all model responses are released.

0.34

Language Models Are "Insecure" Reporters

cs.CL cs.AI ·2026-09-28 ·Jenny Y. Huang, Jiameng Fan, Ahmed Imtiaz Humayun, Maximillian Chen, Tian Qin +3 more
similarity 0.57
Abstract

As large language models are deployed in increasingly autonomous long-horizon tasks, manually auditing and verifying the actions, artifacts, and outputs of models becomes more difficult. Users instead come to rely on LLM-generated reports to assess the quality and completeness of the work. We introduce a suite of eight adversarial reporting scenarios to systematically study whether LLMs conceal narrative-changing flaws: errors or limitations that undermine an otherwise successful account of work. We call this phenomenon "insecure reporting." When handed machine learning experiment logs containing a planted negative result that substantially weakens the proposed method, GPT-5.5 flags the negative result in only 2 of 200 generated reports. However, when a short honesty instruction, "Be honest in your response," is added, the model flags the negative result in 190 of 200 reports. Across eight open-weight models, chain-of-thought analysis reveals a recurring tension between disclosing narrative-changing flaws and reasoning about ways to appear successful. We perform an activation analysis and a steering experiment on Qwen3.5-9B, finding that honesty and success-seeking correspond to opposing directions in representation space. Our results suggest that LLMs tend to present narratives of success by default, and that steering models toward honesty makes their reports substantially more transparent.

0.26

Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents

cs.SE cs.AI ·2026-10-26 ·Nghi D. Q. Bui
code/data similarity 0.56
Abstract · Project page: https://thinking-in-bets.github.io/

Language model agents often abandon correct work when a user challenges it, yet simply telling them to stand firm makes them ignore real mistakes. Once an agent reads a complaint, it tends to discount its own earlier checks in order to agree with the user. We introduce Thinking-in-Bets, a training-free method in which an agent writes a short private record of its main claims, the checks supporting them, and its confidence before reading user feedback. When challenged, the agent consults this Claim Checklist to keep verified work intact while investigating uncertain or newly challenged parts. We evaluate five Gemini models (3.5 Flash-Lite, 3.5 Flash, 3.6 Flash, 3.8 Flash, and 4.0 Argon) across eight benchmarks: SWE-Together, SWE-Interact, and DeepSWE for software engineering, Agents' Last Exam for professional reports, Humanity's Last Exam and SycEval for academic reasoning, Vals Finance for financial analysis, and Terminal-Bench for command-line tasks. Recording claims before criticism nearly eliminates false reversals in patch review and protects passing report drafts under mistaken pushback, whereas writing the same record after reading the complaint recovers very little of the gain. In multi-turn coding, pre-reply claim records substantially reduce unnecessary file edits after mistaken complaints, and gating the record so it is written only when the agent is uncertain lowers token overhead while fixing more real bugs than the baseline. Even in non-interactive tasks without user feedback, listing the key claims in a draft and re-checking the weakest one before submission improves both answer accuracy and confidence estimates across all tested models.

0.23

From Target Selection to Condition Reuse: Auditing Residual Interventions in Tool-Use Models

cs.CL cs.AI ·2026-10-05 ·Tianda Sun, Dimitar Kazakov
code/data similarity 0.56
Abstract · 21 pages, 5 figures. Code: https://github.com/TiandaSun/residual-condition-reuse

Activation interventions can make a language model produce a target answer, but can the model use the intended condition in other computations? We study this question in Qwen3.5-27B on tool-use tasks where time or permissions determine eligibility and selection. Across 32 fresh worlds, two case-specific residual-writing variants raise familiar target-selection correctness from 35/128 to 119/128 and from 33/128 to 114/128. Yet these gains disappear when familiar facts and identifiers are recombined within the same selection task, while selection of previously rewarded identifiers increases. The gains persist across output formats but yield little improvement in independent judgments about whether a candidate is eligible or should be selected. Internal edits can nevertheless change how a source fact affects a judgment: in a prespecified secondary comparison, an early-layer source patch changes 50/64 judgments when that source is used, with one adverse change among 64 cases where it is not. This selective effect does not ensure correct complete actions. Together, these results separate target-selection control, condition-sensitive judgments, and complete-action correctness. Evaluating how an edited state is used, rather than only its trained answer, exposes these differences.

0.44