CS Paper Radar

Supporting or Replacing Students' Reasoning? Cognitive Offloading in AI-Assisted Proof Learning

Eason Chen, Sophia Judicke, Kayla Beigh, Xinyi Tang, Isabel Wang, Nina Yuan, Zimo Xiao, Chuangji Li, Shizhuo Li, Reed Luttmer, Shreya Singh, Maria Yampolsky, Naman Parikh, Yvonne Zhao, Meiyi Chen, Scarlett Huang, Anishka Mohanty, Gregory Johnson, John Mackey, Jionghao Lin, Ken Koedinger

cs.HC ·submitted 2026-10-22 16:06 UTC · v1

Abstract

We studied 153 students in an undergraduate proof-based mathematics course with staggered-access to an AI support system. Students were randomly assigned to receive an AI chatbot and a homework-feedback interface earlier or later in the course. During the period when only the early-access group had access, their homework scores were 2.56 points higher, but exam performance did not differ significantly between groups, suggesting that improved AI-supported homework performance did not translate into detectable gains on a subsequent exam. After all students received access, every additional 100 messages sent to the AI chatbot was associated with a 3.02-point lower score on the subsequent exam. A mediation analysis showed a significant indirect association linking lower baseline self-efficacy to greater chatbot use and, in turn, lower exam scores: a one-point decrease in self-efficacy corresponded to 0.62 fewer exam points through chatbot use. These findings raise concerns that students may use the chatbot to offload their cognitive reasoning. Use of the homework-feedback interface, which required students to submit their proof attempts before receiving feedback, was not negatively associated with exam performance. We recommend against using a chatbot for AI tutoring and suggest exploring a tutor embedded in the course activities.

Comment
14 pages. This manuscript presents a substantially revised and extended analysis of the classroom study reported in arXiv:2602.18807 and the earlier preprint arXiv:2509.16778

Closest papers

Nearest neighbours by meaning across the whole index.

Instructor-designed AI improves learning in a randomized trial across 19 universities

cs.AI cs.CYcs.ETcs.HC ·2026-11-14 ·Jun Li, Robert Bray, Sébastien Martin, Nils Van den Steen, Andrew Wu +47 more
similarity 0.69
Abstract

Universities are alarmed by recent evidence that artificial intelligence (AI) can undermine learning, with students prescribed generic chatbots or centrally designed AI tools performing worse. In a pre-registered randomized controlled trial across 7,200 students at 19 universities over three semesters, we tested whether AI designed by frontline instructors can improve learning. We encouraged instructors to design AI solutions they believed would best help their students. Drawing on their unique knowledge of their classes and students, the instructors created a wide array of innovations. We show that these instructor-designed AI innovations improved student learning, measured by AI-proof assessments (proctored, in-person examinations). We found no change in student engagement, measured by course activity records, or satisfaction, measured by course evaluations. Our findings demonstrate that AI can improve student learning when its adoption is frontline-driven, with instructors empowered to design and refine AI solutions for their own classrooms.

0.28

LLM Pedagogical Behavior in AI Tutoring Interactions

cs.CL cs.HC ·2026-08-24 ·Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin
top venue similarity 0.69
Abstract

Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.

0.46

HelpCoach: Scaffolding Targeted AI Help-Seeking During Problem-Solving

cs.HC ·2026-09-24 ·Hyoungwook Jin, Weirui Peng, Jieun Han, Q. Vera Liao, Xu Wang
similarity 0.68
Abstract

Students increasingly turn to AI for help with problem-solving, yet too much AI support can undermine learning itself. To benefit from AI, students need to specify the necessary knowledge and scaffold type in their questions. However, they struggle to formulate such targeted questions because they lack metacognitive skills to recognize and select effective help options. We developed HelpCoach, an add-on for chat interfaces that helps students formulate knowledge- and scaffold-specific questions and receive targeted help during problem solving. HelpCoach continuously assesses students' help-seeking performance and prompts students to improve through an adaptive revision template. Whereas prior work has largely taught help-seeking skills apart from learning tasks, HelpCoach's in situ scaffold enables concrete practice on metacognitive skills and immediate revisions to help-seeking behavior. In a study with 40 college students learning web programming, HelpCoach led to more specific questions during chatbot interactions and greater knowledge retention than pre-task help-seeking training alone.

0.17

Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course

cs.AI econ.GN ·2026-10-05 ·Canh Thien Dang, An Nguyen
position similarity 0.65
Abstract

Experimental studies of generative AI in education mostly report average effects, yet field evidence shows that AI can narrow attainment gaps or widen them. We argue that the direction depends on the judgement burden, the expertise a learner must supply to screen AI output before learning from it, and that expert verification before release moves this burden from students to an accountable tutor. We test the argument in a two-cohort difference-in-differences design in which one half of a compulsory firstyear university economics course received AI-generated podcasts, FAQs and quiz-based study guides, produced with a source-grounded model and checked by a named graduate teaching assistant (170 students; 340 examination marks). Access was associated with a 2.34-mark advantage on a 50-mark component. The share of marks below the upper-second classification boundary fell by 24.7 percentage points relative to the counterfactual, effects were significant at every threshold from 23 to 31 marks and at none above, and roughly three-quarters of the average originated in the bottom quintile. The threshold estimate is robust to removing the lowest-scoring students from the pre-intervention cohort; the average effect is not. Interviews and feedback from 36 students indicate that the verification label gave students a reason to engage with AI-generated material without ending their scrutiny of it. Evaluations of AI learning resources that report only mean effects cannot detect whether the students the resources are meant to help are the ones who gain.

0.15

Human-AI Conversational Behaviors Predict Unassisted Task Performance

cs.HC ·2026-10-07 ·Li Siyan, Federico Bianchi, James Zou, Kaitlyn Zhou
similarity 0.65
Abstract

Assistance from AI tools has supported and improved human performance across domains. However, recent research suggests that these immediate benefits may entail future costs, including diminished performance when AI assistance is no longer available. We study how human-AI interaction behaviors correlate with immediate and future unassisted task performance across two game-based problem-solving user studies ($n=139$ and $n=111$), using a \textit{dialogue act} framework adopted from a tutor-student dialogue taxonomy. In our studies, verbalizing thought processes correlates with higher unassisted outcomes, whereas directly requesting solutions correlates negatively. Similar to these participant-side patterns, assistant explanations of the current problem state are associated with better subsequent unassisted performance, whereas directly providing the next action is associated with worse performance. Qualitative and subtype analyses further show that ostensibly similar reasoning turns can elicit different assistance. Our findings suggest that preserving users' cognitive participation in problem-solving may support performance beyond AI-assisted interaction.

0.08

The Uneven Impact of Generative AI on Student Learning: Examining the Roles of Reliance, Evaluation Literacy, and Course Policy in AI-related Courses

cs.AI cs.ETcs.LG ·2026-09-16 ·Lydia Manikonda, Mei Si, Sirajam Munira, Oshani Seneviratne, Kristin Bennett
hot direction similarity 0.65
Abstract · Paper under review

Generative artificial intelligence (GenAI) is changing how students learn, yet the roles of course context, cognitive reliance, evaluation literacy, and early reliance remain underexplored. Using survey responses from 118 students across 12 AI-related courses at our institution, we examined differences in GenAI use and perceived learning experiences. We identified four user clusters: high-use students reporting many benefits, light users reporting less reliance and fewer benefits, and two moderate-use groups reporting different levels of benefit. We also found significant differences between free- and premium-version users, single- and multiple-tool users, and students experiencing different instructor policies. In multivariable regression models, academic benefit was associated with early reliance and academic task support; positive impact was associated with cognitive reliance, academic task support, confidence in GenAI reliability, and instructor policy; and negative impact was associated with early reliance and attitudinal change. The association between early reliance and negative impact became stronger as evaluation literacy increased. Finally, perceptions of GenAI-enhanced learning appear to reflect cognitive, performance, and self-efficacy benefits, while concerns about stress and diminished critical thinking are associated with lower perceived learning benefits. These findings suggest that institutions need better policies to address such inequities so that institutions can enable students to benefit from increasingly capable AI systems.

0.35

Feeling Easier, Working Harder: Cognitive Load in AI-Assisted Code Comprehension

cs.SE cs.AI ·2026-11-09 ·Maryangela White, Maximilian Rose, Matthew Buckman, Amy McDonnell, Francisco Ortega +1 more
top venue similarity 0.65
Abstract · Accepted to ICSE 2027

Software developers spend significant time reading and understanding code, increasingly supplemented by generative AI. While these tools aim to reduce the difficulty of reading code, interacting with them introduces a dual-task environment that may impose extraneous cognitive load that is challenging to capture. This study investigates how access to generative AI affects developers' real-time cognitive workload, code comprehension performance, and self-perceived code comprehension strategies. We conducted a 3x2 within-subjects study with 20 novice CS students reviewing Java code snippets of varying lengths with or without AI assistance, followed by an AI-unassisted comprehension quiz. Cognitive workload was continuously captured using a Detection-Response Task (tDRT) during both reading and quiz phases, and compared to post-quiz NASA-TLX and SEQ scores. AI assistance significantly increased objective cognitive workload, raising tDRT reaction times by 11.2%, yet participants' perceived workload remained unchanged. Code comprehension scores were high under both conditions, with no evidence that AI assistance improved comprehension at the difficulty levels evaluated; a ceiling effect limited power to detect benefits on harder tasks. Qualitatively, participants reported trusting the AI conditionally -verifying its output against their own reading for more complex snippets, but directly trusting it for simpler code- and, while many felt their reading process was unchanged, a notable subset shifted toward skimming or reading the AI's explanations in place of the code itself. These findings reveal that developers do not accurately perceive the cognitive resources they expend when managing AI tools. This study provides evidence of added cognitive effort associated with using generative AI, pointing towards a need to account for this dual-task penalty in software engineering tools and education.

0.61

"Can a Robot Really Care About Me?" How Perceiving a Tutor as Human or AI Affects Computer Science Students' Attitudes and Experiences

cs.HC ·2026-11-16 ·Tymon Vu, Wenxuan "Wendy" Shi, Silas Hsu
similarity 0.65
Abstract · 30 pages, 3 figures

Conversational artificial intelligence can reproduce many instructional and social behaviors of human tutors, but students may interpret those behaviors differently depending on who they believe is providing them. We examined how perceived tutor identity shapes students' evaluations of instructional feedback, satisfaction of psychological needs according to Self-Determination Theory, and their communication strategies with tutors. In a within-subjects Wizard-of-Oz study, 27 introductory computer science students completed two text-based tutoring sessions involving Python code-explanation problems. Participants were told one tutor was human and the other was AI, although the same human provided feedback in both conditions. We found participants evaluated both tutors favorably and reported few consistent differences in the instructional value of the feedback. However, participants frequently questioned whether AI could genuinely care about them, which corresponded with the human tutor having higher scores of relatedness need satisfaction. Human presence offered connection and accountability for some participants but introduced anxiety in others. Participants also reported communicating more casually with the human tutor and more explicitly with the AI tutor. These findings suggest complementary roles for human and AI tutoring and highlight the importance of considering psychological needs alongside instructional quality.

0.36