CS Paper Radar

Novice Reliance Calibration in AI-Assisted Decision Making: The Role of Explanations and Self-Assessment

Eun Jeong Kang, Peter, Duan, Swati Mishra

cs.HC cs.AI ·submitted 2026-10-06 05:48 UTC · v1

Abstract

Artificial Intelligence (AI) tools are widely used to support decision making in tasks and domains where no immediate performance feedback is available. In these settings, users cannot learn to adjust their reliance behavior over time through trial and error. However, little is known about how novice users calibrate reliance on AI when external feedback is unavailable, or whether AI explanations can support calibration in its absence. We introduce reliance calibration as an organizing construct for studying how novice users dynamically adjust reliance behavior, and examine how AI explanations and meta-cognitive self-assessment shape it. Through a between-subjects study with 110 participants completing a clinical entity extraction task with AI assistance and limited performance feedback, we observe that novice users exhibit systematic drift toward over-reliance in the presence of explanations, while higher self-reported task understanding is associated with more selective reliance behavior. These results extend reliance calibration research into human-AI collaboration contexts without real-time performance signals and present actionable guidelines on designing AI tools that must support appropriate reliance in these settings.

Comment
under review

Closest papers

Nearest neighbours by meaning across the whole index.

Think Once, Reflect Twice: How Feedback Shapes Human-AI Collaboration

cs.HC ·2026-10-14 ·Joshua Holstein, Philipp Spitzer, Sebastian Schäfer, Maximilian Förster, Gerhard Satzger
hot direction similarity 0.65
Abstract

As artificial intelligence (AI) increasingly supports human decision-making in high-stakes domains, designing effective human-AI collaboration becomes critical. Successful collaboration requires humans to reflect on decision-making processes and adjust their strategies accordingly. Current research typically investigates interventions, such as explainable AI, that target reflection during the decision-making process (reflection-in-action). Less attention has been given to retrospective examination of decisions (reflection-on-action). This work explores reflection-on-action by studying the impact of feedback on reliance behavior and human-AI team performance. Specifically, we investigate feedback interventions targeting overrides of AI advice through a behavioral study (N=244). We find that such feedback influences reflection-on-action, while unidirectional feedback risks reducing subsequent reflection-in-action. Furthermore, we identify opposing shifts in reliance behavior that leave overall human-AI team performance unchanged but show context-dependent effects. Balanced feedback did not show these risks, preserving both concurrent and retrospective reflection, offering insights for designing for appropriate reliance on AI.

0.29

Confident, Not Wiser: The Dunning-Kruger Effect in Human-AI Interaction

cs.HC ·2026-09-25 ·Daniela Fernandes, Michelle Rausch, Agnes Mercedes Kloft, Daniel Buschek, Robin Welsch
similarity 0.64
Abstract · 37 pages, 10 figures, 12 tables

AI assistance can improve performance without improving self-assessment. We report a study (N=366) comparing Human alone and Human+AI performance on reasoning tasks, for which the AI model is benchmarked on the same items. Participants estimated global and block performance and rated confidence in their answers. Human+AI achieved higher scores, but self-estimates tracked performance weakly. Average overestimation was similar across groups, covering individual errors. Across tasks, confidence distinguished correct from incorrect answers less accurately in the Human+AI group, while within-task differences remained uncertain. The Dunning-Kruger pattern was found in both groups, with a larger observed contrast in Human+AI. Controls for score noise reduced but did not eliminate the pattern, with the controlled group difference remaining inconclusive. An extended computational account describes global and block estimates. Our findings distinguish performance augmentation from metacognitive augmentation and motivate interfaces that support verification, communicate task-specific AI model performance, and help users evaluate the quality of their joint work rather than produce answers.

0.14

What Human-AI Interaction Leaves Behind: Residual Human States -- A Trajectory-Based Framework and Multi-Study Analysis

cs.HC ·2026-10-13 ·David Herrera-Poyatos, Carlos Peláez-González, Cristina Zuheros, Francisco Herrera
code/data similarity 0.63
Abstract · 29 pages, 13 tables. Code: https://github.com/ari-dasci/S-ResidualHumanStates

Human-AI Interaction is commonly evaluated by what an artificial agent enables during use, yet interaction can also alter the human who acts next. We introduce Residual Human States, a trajectory-based framework for representing the potentially multidimensional human state carried forward after interaction with AI, relative to a relevant alternative trajectory. The framework distinguishes assisted output from human-state transition and derives four empirical propositions: multidimensionality, interaction-policy dependence, horizon dissociation, and temporal persistence. Its central claim is that assisted performance does not by itself determine the human state carried forward. We examine these propositions through four empirical cases with different inferential roles. Published medical-training evidence illustrates how one AI interaction history can be associated with both slightly higher observed accuracy on some later unaided cases and transmission of a systematic decision bias on others. A second diagnostic-learning study shows that later unaided behaviour differs across AI-support policies, while also revealing that aggregate accuracy cannot by itself distinguish improved discrimination from a shifted response criterion. In a classroom experiment, generative-AI tutoring substantially improved assisted practice performance without a corresponding detectable gain on a subsequent unaided exam. A longitudinal reanalysis of personalised AI dialogue further shows that residual contrasts can evolve across post-interaction horizons and that such change may arise from the comparison trajectory as well as from the focal intervention. Evaluating interactive AI systems therefore requires attention not only to contemporaneous performance, but also to which human states alternative interaction trajectories produce, relative to which counterfactuals, and how durably.

0.46

Structured pre-generation elicitation versus single-shot prompting in AI-assisted enterprise decision-making: a randomised online experiment

cs.HC cs.AI ·2026-10-07 ·William Scott-Jackson
similarity 0.63
Abstract · 12 pages

Generative AI speeds, and mostly improves, professional work, but there is concern that users who delegate both the production and the evaluation of an answer may accept weak output and engage less with the underlying reasoning (cognitive surrender). Interventions proposed so far, such as unassisted practice or slowing adoption, sit outside the working task. We tested a different approach: an interactive metacognitive scaffolding layer (Cognistance, a prototype developed at the Oxford Centre for Impact Research (OCIR) that asks users to clarify context, choose a strategic direction and explain their reasoning before the AI generates a deliverable). Mean composite quality was 32% higher with the scaffold, with the same direction for every rater. Gains were largest for trade-off articulation and strategic coherence and absent for technical specificity. A large part of the aggregate effect reflected rescue of weak prompts: floor-scored (off-task) deliverables fell from 34% to 5%. Among participants whose own prompt already stated the data-localisation problem, the advantage was 21%. Treatment participants reported greater involvement and took about 2.4 minutes longer on average (10.46 minutes). Immediate recall scores were higher, which tentatively suggests better retention, but in this limited experiment, was not robust to sensitivity analyses. Self-ratings of quality did not track rated quality in either condition. Structured elicitation before generation improved the rated quality and task relevance of AI-assisted strategy documents at modest cost in time. Delayed retention, error detection and effects in live organisations are the priorities for the next stage of research.

0.18

Are Concept Bottleneck Models Effective as Decision-Support Systems?

cs.HC cs.AI ·2026-08-26 ·Alessandro Bogani, Nicola Debole, Emanuele Marconato, Andrea Pugnana, Katya Tentori +1 more
similarity 0.63
Abstract

Concept Bottleneck Models (CBMs) are interpretable-by-design neural networks that detect human-understandable concepts from the input and use them to generate predictions. By allowing users to inspect the concepts underlying a prediction and explore how predictions change under alternative concept configurations, CBMs have emerged as one of the most prominent approaches to supporting human-AI collaboration. However, user studies investigating their actual effectiveness as decision-support systems remain limited. We present two large-scale user studies (N participants = 705, N observations = 6,959) evaluating how concept-based explanations and user interventions on the model's concepts affect the performance of the human-AI team in two distinct binary classification tasks. Our results show that CBMs, and particularly their interactive component, can improve human-AI team accuracy relative to both unaided human performance and performance with non-interpretable AI support. However, these benefits emerge only under certain conditions: classification tasks perceived as difficult, easily identifiable concepts, and active interaction with the model. We also discuss how inaccurate concept detection may undermine users' trust in the model. Overall, this work provides practical guidance for the deployment of CBMs as effective decision-support tools.

0.43

Human-AI Conversational Behaviors Predict Unassisted Task Performance

cs.HC ·2026-10-07 ·Li Siyan, Federico Bianchi, James Zou, Kaitlyn Zhou
similarity 0.63
Abstract

Assistance from AI tools has supported and improved human performance across domains. However, recent research suggests that these immediate benefits may entail future costs, including diminished performance when AI assistance is no longer available. We study how human-AI interaction behaviors correlate with immediate and future unassisted task performance across two game-based problem-solving user studies ($n=139$ and $n=111$), using a \textit{dialogue act} framework adopted from a tutor-student dialogue taxonomy. In our studies, verbalizing thought processes correlates with higher unassisted outcomes, whereas directly requesting solutions correlates negatively. Similar to these participant-side patterns, assistant explanations of the current problem state are associated with better subsequent unassisted performance, whereas directly providing the next action is associated with worse performance. Qualitative and subtype analyses further show that ostensibly similar reasoning turns can elicit different assistance. Our findings suggest that preserving users' cognitive participation in problem-solving may support performance beyond AI-assisted interaction.

0.08

AI as Teammate: Rethinking Task Distribution in Medical Training

cs.HC cs.AI ·2026-08-28 ·Fendi Tsim, Alina Gutoreva, Anthony Weiss, Nicole Dubosh
similarity 0.62
Abstract

Integrating Artificial Intelligence (AI), particularly generative AI, into medical training has prompted concerns about learner over-reliance, misuse, and erosion of foundational clinical competencies. We propose a conceptual reframing at the decision level: the problem is not misuse but misclassification - a mechanistic failure of real-time metacognitive evaluation in selecting a subzone-inappropriate AI interaction mode. Drawing on "SCAN" (Substitute, Complement, Aid, Non-Negotiable), a human-centric decision-making framework for generative AI task allocation grounded in Vygotsky's Zone of Proximal Development and metacognition, we advance the emerging social-constructivist conversation around AI in medical education by offering a testable account of AI's role in clinical reasoning development. This framework yields testable predictions for how misclassification can be detected, mitigated, and, more importantly, prevented in the clinical learning environment. Regarding clinical reasoning development, we show how trajectories of skill acquisition (upskilling) and failure (the triad of skill failure: de-skilling, never-skilling, and mis-skilling) operate at the individual task level in ways that fixed-phase, cohort-wide treatments fail to capture. We further identify passive engagement within correctly classified AI-scaffolded tasks as a particularly insidious, detection-resistant pathway to mis-skilling - one requiring subzone re-identification from AI assistance to expert assistance, with human experts serving as epistemic auditors. The paper operationalizes SCAN for clinical curriculum design, supervision, and assessment, and opens an empirical research agenda grounded in cognitive science. This paradigm shift from misuse to misclassification is not semantic: it offers educators a clear perspective on what to look for, what to assess, and what to intervene on.

0.27

AutoAnalysis: An automated, self-updating meta-analysis of human-AI collaboration

cs.HC cs.AIcs.CYcs.DL ·2026-10-30 ·Michelle Vaccaro, Abdullah Almaatouq, Thomas W. Malone
similarity 0.62
Abstract · Please visit auto-analysis.dev to explore the data and view the most recent results

As generative artificial intelligence (AI) becomes more capable and widely used, understanding when and how humans should work with these systems is becoming increasingly important. However, the empirical evidence in this space offers conflicting recommendations, which leaves individuals and organizations without clear guidance on when to combine human and AI capabilities. Existing meta-analyses can help reconcile these findings, but they require substantial manual effort and cannot readily incorporate new evidence as it emerges. To address this challenge, we introduce AutoAnalysis: an automated, self-updating meta-analysis of when humans and AI work better together, and when they work better alone. AutoAnalysis' most recent run synthesizes 1,955 effect sizes from 420 papers. Across this body of work, we found that, on average, human-AI combinations performed significantly better than humans alone, but they performed significantly worse than the better of humans or AI alone. These findings highlight the important distinction between improving human performance and achieving human-AI synergy and underscore the need to evaluate collaboration against both human and AI performance alone. As new studies emerge, AutoAnalysis will integrate their findings into the meta-analysis, providing cumulative evidence about when humans and AI perform better together and how human-AI performance varies across AI agents, tasks, and processes.

0.13