CS Paper Radar

Provenance-Aware Belief Updating in Human-AI Feedback Loops

Tianyi Huang, Eric Liu, Jay Chong, Emily Yin, Wanqi Zhao

cs.AI ·submitted 2026-10-09 05:09 UTC · v2
publishedbenchmark

Abstract

In repeated human-AI interaction, a single observation can be quoted, summarized, and returned to a model as multiple reports, making repetition look like independent corroboration. We study whether language models can use supplied report lineage to update according to evidence provenance rather than report count. We introduce Latent Quotient Evidence Training (LQET), a relational objective that combines sparse Bayesian supervision with constraints that encourage stable predictions when reports add no new evidence and consistent updates to the same new evidence across relay histories. On a controlled binary benchmark, LQET reduces new-evidence update RMSE by 41-44\% and log-odds drift after eight relay steps by 89-91\% relative to task-only training across Qwen3-4B and Phi-4-mini, with three paired training seeds per backbone. The gains extend to a closed-loop simulation in which model confidence controls how many relayed reports re-enter the model's context in later rounds. By grounding belief updates in supplied provenance, LQET offers a route toward dynamic human-AI systems in which influence follows new evidence, not the number of times the same evidence returns.

Comment
Accepted Oral at the NeurIPS 2026 Workshop on Dynamic Alignment in Human-AI Coupled Systems

Closest papers

Nearest neighbours by meaning across the whole index.

GraphEcho: Structural Redundancy and Evidence Provenance in LLM Graph Agents

cs.AI cs.CL ·2026-09-15 ·Sikun Wang, Yixi Zhou, Lei Fan, Fan Zhang
benchmark similarity 0.67
Abstract · 13 pages, including figures and tables

A large language model (LLM) agent can follow more graph paths without acquiring more independent evidence. GraphEcho tests whether agents mistake these repeated encounters for additional corroboration. The benchmark varies path counts and evidential origins while holding evidence content fixed, and evaluates both judgments and active exploration. Controlled synthetic experiments reveal model-dependent judgment shifts, but redundant supporting paths increase the share of repeated walks across all evaluated frozen agents. Provenance-aware post-training (PAPT) reduces revisits and improves synthetic accuracy, yet covers fewer distinct sources. On scientific claims, it continues to reduce repetition while accuracy declines. These findings expose a gap between efficient exploration and effective evidence use: an agent can learn to stop repeating itself while overlooking information it needs. GraphEcho provides a controlled way to evaluate both what graph agents conclude and whether their exploration reaches distinct evidential sources.

0.36

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

cs.CL cs.AIcs.IR ·2026-09-08 ·Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang, Mark Stevenson
top venue similarity 0.67
Abstract · CIKM'26

Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.

0.54

A Benchmark and Layered Evaluation Protocol for Evidence-Grounded Shared Belief Updating in Connected Robotics

cs.RO cs.AIcs.CLcs.MA ·2026-11-09 ·Ranulfo Bezerra, Kimitoshi Yamazaki
benchmarkpositioncode/data similarity 0.66
Abstract · 15 pages (10 main + 5 supplementary), 6 figures, 4 tables; supplement adds 1 figure and 7 tables. Project page: https://creg-bench.github.io

Connected physical systems fuse observations from robots, IoT sensors, and edge-AI modules into a shared account of the world, yet most evaluations score only the final answer or action and cannot tell whether a method attributed the evidence, updated its belief, handled contradictions, and justified its decision. We argue that shared reasoning in Connected Robotics should be evaluated as an evidence-to-belief trajectory, and introduce the Connected Robotics Evidence Graph (CREG) Benchmark to make that trajectory measurable. CREG comprises 350 deterministically generated scenarios across seven regimes (justified revision, belief preservation, relation association, contradiction onset and resolution, stale/delayed evidence, and source reliability), each exposing a normative posterior reference at every evidence prefix, validated by an independent log-odds checker and screened by a blinded expert audit. Nine representation conditions separate relation content from serialization, ordering, and metadata. Evaluating four open-weight instruction models and six transparent fusion baselines yields three findings. First, entity grounding accounts for a substantial part of the weak-model deficit. Second, deleting semantic relations costs 15.3 points of strict belief-state accuracy on relation-dependent scenarios (Holm-corrected p<0.001), but an information-matched structured list outperforms the full graph and shuffling the graph does not hurt: relation content matters where required, while graph serialization confers no consistent advantage. Third, belief accuracy is strongly non-monotonic along the evidence sequence, with mid-trajectory failures that endpoint-only evaluation cannot detect. All scenarios, references, prompts, metrics, and analysis code are released for offline, hardware-free reproduction.

0.43

CAMR: Compatibility-Aware Memory Replay for Continual Learning in Large Language Models

cs.CL ·2026-10-27 ·Jingpu Yang, Tianhao Li, Fengxian Ji, Yufeng Wang, Shuxin Zheng +1 more
similarity 0.66
Abstract

Continual learning requires large language models to acquire new knowledge and capabilities while retaining what they have learned. Fine-tuning and replay support acquisition and retention, but deciding whether incoming examples need supervision and which historical examples to revisit remains a challenge. We propose Compatibility-Aware Memory Replay (CAMR), an adaptive framework that jointly controls current-example supervision and historical replay. CAMR combines compatibility checks with a learned memory gate to guide an online policy over three actions: learning incoming examples with general replay, jointly reinforcing compatible related examples, and skipping direct supervision of verified mastered duplicates. The policy learns from acquisition, retention, and training-cost feedback under a bounded per-arrival replay budget. We evaluate CAMR with Qwen3-8B-Base on continual factual updating and cross-task and cross-domain learning. On WikiBigEdit, CAMR consistently leads evaluated methods in Original Match from 3K to 100K facts, reaching 93.66% after just 3K facts and surpassing Full-history Replay (LoRA) after 100K facts. It also achieves the highest final scores on all four selected TRACE tasks and maintains or improves earlier-task performance relative to post-acquisition levels in a sequence spanning financial question answering, mathematical reasoning, code generation, and summarization.

0.15

Same Answer, Different Future: Dissociating Belief Content and Evidence Weight in Language Models

cs.LG ·2026-11-05 ·Weihan Li, Junhao Wu, Xinlei Chen, Tianshi Zheng
position similarity 0.64
Abstract · 37 pages, 18 figures, 17 tables

Memory summaries, prompts and steering vectors set a language model's beliefs and are judged by the prediction they produce. A prediction does not say how firmly it is held. We separate a belief's content, what it predicts now, from its weight, the number of new observations it is worth: for an ideal observer, the amount of evidence behind it. In Bernoulli inference we set beliefs in nine open-weight models of 8 to 397B parameters and follow each belief with the same new observations. We argue that these models hold an answer without its evidence and recover its weight from the text they re-read. Weight therefore follows the form of the evidence and the question asked more than its amount: matched on their current estimate, a record of observations and a summary of its counts usually move apart under the same new observations, in either direction. Scale does not close the gap. Across the nine models 64 observations are worth 20 to 53 as a record and 9 to 151 as a summary, no closer to 64 at 397B than at 8B. Where no text stands behind a belief there is little to re-read: a steering vector can install a record's answer, and sixteen observations undo it where a neutral passage leaves it in place. Where reasoning first writes the counts into the context the weight returns, and the same 64 observations are worth 58 to 67 in either form. A matched answer is not a matched belief, and methods that set beliefs should be judged by the futures they produce.

0.17

JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias

cs.AI ·2026-11-02 ·Fengxu Yang, Minghui Liwang, Yang Lu, Bo Han, Xianbin Wang +1 more
similarity 0.64
Abstract

High-quality annotations are essential for training and evaluating artificial intelligence (AI) systems. Quality control (QC) often acquires additional human or AI judgments before accepting a label. Nevertheless, shared biases can cause correlated judgments, leading conventional methods to mistake repeated agreement for independent evidence of correctness. To address this challenge, we propose JABER (Joint Alias-Preserving Belief for Evidence Routing), a bias-aware sequential QC framework that maintains a joint Bayesian posterior over label correctness and shared biases to guide evidence acquisition and termination decisions. Building on this posterior, we develop a finite-horizon algorithm that selects evidence-acquisition and termination actions to minimize expected QC loss under resource constraints. Our theoretical analysis shows that conditional-independence updating can become arbitrarily confident in an incorrect label under persistent shared bias. We further identify when repeated inspections cannot distinguish a correct label from an incorrect label supported by shared bias, when obtaining a new label can resolve this ambiguity, and why modeling the dependence between label correctness and shared bias can be necessary for QC decisions. Our experiments show that JABER reduces expected QC loss by an average of 4.65% compared with a method maintaining separate label correctness and bias probabilities when shared bias makes repeated inspections unreliable. Further, JABER's finite-horizon planning, which evaluates outcomes of successive QC actions, reduces expected QC loss by an average of 10.05% compared with a one-step lookahead method. In real-API experiments on FEVER, JABER achieves zero labeling errors while reducing QC costs by 2.67% and 16.99% compared with marginal-based belief modeling and one-step lookahead methods, respectively.

0.14

EVISKILL: Grounding Skill Evolution in Replayable Evidence

cs.AI ·2026-10-04 ·Yan Zhou, Yili Wang, Yiwei Dai, Qinggang Zhang, Xin Wang
highly cited similarity 0.64
Abstract

Continual skill evolution enables LLM agents to accumulate and refine reusable procedural knowledge from interaction experience without updating model parameters. Its effectiveness depends on determining not only what to change, but also why a change is justified and when it should become persistent guidance. However, existing experience-driven methods can lose the behavioral evidence and task contexts supporting edits. Moreover, a global validation outcome provides an incomplete judgment of its constituent changes: locally supported corrections may be discarded with a rejected revision, while evidence may require further experience to inform useful updates. To this end, we introduce EVISKILL, an evidence-driven framework that organizes execution observations into Replayable Evidence Cards and synthesizes edits with explicit links to their supporting contexts. Targeted replay verifies these edits through re-execution and provides feedback for correction. Across epochs, EVISKILL preserves evidence and provisionally retains supported edits for further refinement, while global validation governs their incorporation into the final skill. Experiments on three interactive benchmarks across six LLM backbones demonstrate the effectiveness of this approach.

0.46

Read More, Reason Deeper: Latent Workspaces for Long-Sequence Recommendation

cs.IR ·2026-10-29 ·Renzhi Wu, Li Sheng, Zikun Cui, Junjie Yang, Tai Guo +6 more
similarity 0.64
Abstract

Long-sequence recommendation can improve along two axes: \emph{evidence scaling} exposes longer histories, while \emph{reasoning scaling} adds layers that interpret that evidence for a candidate. Existing models couple them: sequence encoders propagate every added event through depth, whereas one-shot selection prevents later layers from revisiting omitted evidence. We introduce the Latent Workspace Ranker (LWR), which keeps typed evidence in persistent banks and propagates a fixed-size candidate-local workspace. Each layer rereads the banks from its evolving workspace and archives compact post-interaction state, allowing history length and reasoning depth to grow independently. We compare four architectures across various history--depth configurations using evidence scaling at fixed depth, reasoning scaling at fixed 10K history, and their joint compute-efficient frontier. LWR leads all families at long histories, continues improving with reasoning compute, and converts additional training compute into quality gains at the fastest fitted rate on the joint frontier. At 10K history, it improves NE by 0.19% over HSTU while using 69.6% fewer forward GFLOP/request. It also outperforms existing baselines on two billion-scale ranking surfaces. These results show that separating addressable evidence from propagated state makes reading more and reasoning deeper independently controllable and jointly useful.

0.22