CS Paper Radar

Impacts of generative AI skill development on academic adaptation and learning outcomes among university students

Lei Fan, Fangxue Liu

cs.HC cs.AI ·submitted 2026-10-22 01:58 UTC · v1

Abstract

Understanding how university students perceive the development of their generative artificial intelligence (GenAI) skills over time can clarify how GenAI use relates to learning and academic outcomes. This study examined perceived skill-development trajectories, changes in learning practices, and associated academic gains using a retrospective questionnaire with 313 analyzed responses and interviews with 18 students across subject areas. Descriptive and comparative analyses and exploratory structural equation modeling (SEM) were integrated with thematic analysis. Perceived developmental trajectories were heterogeneous: 34.2% of students reported rapid early improvement followed by a plateau, 28.4% steady improvement, and 21.1% delayed acceleration. Development was also uneven across skills, with improvement in evaluating GenAI outputs perceived as more challenging; assignments were the most frequently selected driver of skill growth. Interviews further indicated increasingly specific prompting, tailored verification, and selective delegation, although some students reported stable routines, dependence, skipped reasoning, or improvements attributable partly to better GenAI tools. Academic gains were reported more often for assignments than for examinations. The SEM indicated a positive indirect association between perceived skill growth and academic gains through adaptation of assignment processes, whereas academic gains did not clearly differ across developmental trajectory groups after multiplicity correction. The findings suggest that perceived GenAI skill development varies in both trajectory and skill domain and is associated with changes in learning practices and differentiated academic benefits.

Closest papers

Nearest neighbours by meaning across the whole index.

The Uneven Impact of Generative AI on Student Learning: Examining the Roles of Reliance, Evaluation Literacy, and Course Policy in AI-related Courses

cs.AI cs.ETcs.LG ·2026-09-16 ·Lydia Manikonda, Mei Si, Sirajam Munira, Oshani Seneviratne, Kristin Bennett
hot direction similarity 0.70
Abstract · Paper under review

Generative artificial intelligence (GenAI) is changing how students learn, yet the roles of course context, cognitive reliance, evaluation literacy, and early reliance remain underexplored. Using survey responses from 118 students across 12 AI-related courses at our institution, we examined differences in GenAI use and perceived learning experiences. We identified four user clusters: high-use students reporting many benefits, light users reporting less reliance and fewer benefits, and two moderate-use groups reporting different levels of benefit. We also found significant differences between free- and premium-version users, single- and multiple-tool users, and students experiencing different instructor policies. In multivariable regression models, academic benefit was associated with early reliance and academic task support; positive impact was associated with cognitive reliance, academic task support, confidence in GenAI reliability, and instructor policy; and negative impact was associated with early reliance and attitudinal change. The association between early reliance and negative impact became stronger as evaluation literacy increased. Finally, perceptions of GenAI-enhanced learning appear to reflect cognitive, performance, and self-efficacy benefits, while concerns about stress and diminished critical thinking are associated with lower perceived learning benefits. These findings suggest that institutions need better policies to address such inequities so that institutions can enable students to benefit from increasingly capable AI systems.

0.35

Generative artificial intelligence prompting strategies among Chinese university students across academic tasks

cs.HC cs.AI ·2026-10-29 ·Lei Fan, Fangxue Liu
similarity 0.65
Abstract

Generative artificial intelligence (GenAI) is increasingly embedded in university learning, yet effective human-GenAI interaction depends not only on access to these tools but also on students' ability to formulate, evaluate, and refine prompts for academic tasks. This study examined university students' perceived GenAI prompting capability and reported prompting strategies. Questionnaire data from 234 students were analyzed using descriptive statistics, measurement modelling, scenario comparisons, structural associations, and qualitative coding of optional comments. Students reported using GenAI across diverse academic activities and generally expressed confidence in their prompting capabilities, particularly in providing relevant context and refining prompts following initial outputs. However, prompting strategies showed limited sensitivity to task types: specifying output format was more strongly endorsed in a creative workshop preparation scenario than in a research-oriented assignment, whereas other matched strategies showed no adjusted differences. Students also commonly endorsed both concise instructions and providing all relevant information at the outset, suggesting that effective brevity may depend more on selecting and organizing task-relevant information than on prompt length alone. By contrast, fixed prompt templates received lower endorsement than other prompting strategies. In the adopted factor model, perceived cognition showed the clearest unique association with perceived prompting capability, while open-ended responses highlighted prompt-intent alignment as a common challenge.

0.10

Measuring undergraduate students' reliance on generative AI during problem-solving: Scale development and validation

cs.HC ·2026-10-21 ·Chenyu Hou, Gaoxia Zhu, Vidya Sudarshan, Fun Siong Lim, Yew Soon Ong
publishedhighly cited similarity 0.63
Abstract · If you are interested in accessing the full scale and instructions, please fill in this form: https://forms.gle/xbvg9CF84o3AW1SW7

Reliance on AI describes the behavioral patterns of when and how individuals depend on AI suggestions, and appropriate reliance patterns are necessary to achieve effective human-AI collaboration. Traditional measures often link reliance to decision-making outcomes, which may not be suitable for complex problem-solving tasks where outcomes are not binary (i.e., correct or incorrect) or immediately clear. Therefore, this study aims to develop a scale to measure undergraduate students' behaviors of using Generative AI during problem-solving tasks without directly linking them to specific outcomes. We conducted an exploratory factor analysis on 800 responses collected after students finished one problem-solving activity, which revealed four distinct factors: reflective use, cautious use, thoughtless use, and collaborative use. The overall scale has reached sufficient internal reliability (Cronbach's alpha = .84). Two confirmatory factor analyses (CFAs) were conducted to validate the factors using the remaining 730 responses from this activity and 1,173 responses from another problem-solving activity. CFA indices showed adequate model fit for data from both problem-solving tasks, suggesting that the scale can be applied to various human-AI problem-solving tasks. This study offers a validated scale to measure students' reliance behaviors in different human-AI problem-solving activities and provides implications for educators to responsively integrate Generative AI in higher education.

0.77

Generative AI Expands the Intellectual Reach of Course Based Undergraduate Research Experiences (CUREs)

cs.AI cs.HC ·2026-08-27 ·Aditi Babar, Kristin J. Davin, Alex Dornburg
similarity 0.59
Abstract

Course-based undergraduate research experiences (CUREs) broaden access to authentic scientific inquiry through responsive instructor support as research problems become increasingly complex. Generative artificial intelligence (GenAI) may extend this support by providing individualized assistance that can adapt as student needs change. However, how embedding GenAI within a CURE to provide support across the research process impacts student inquiry, collaboration, and scientific reasoning remains unresolved. Here we use longitudinal qualitative data collected across three semesters of a bioinformatics and genomics CURE to show that GenAI expanded the intellectual reach of the research experience in three distinct ways. First, personalized, on-demand scaffolding allowed students to move beyond the boundaries of instructor expertise and transform their own interests into researchable inquiry, with all teams developing distinct self-directed projects rather than selecting instructor-provided topics. Second, GenAI became part of the distributed cognitive system of research teams, helping novice researchers communicate and coordinate across differentiated expertise without eliminating specialization. Third, expanded capability did not replace the need for disciplinary judgment. Students increasingly validated, revised, or rejected AI-generated contributions, such that research independence emerged through retained intellectual responsibility. Together, these findings suggest that GenAI can extend the reach of CUREs by expanding what novice researchers can investigate, how they can collaborate, and the level of responsibility they can assume while preserving human judgment central to authentic scientific inquiry.

0.20

Critical Thinking with Generative AI: A Constraint-First Design Pilot of a Thinking-Partner Intervention

cs.HC ·2026-09-29 ·Fatima Tuz Zahra, Jiangen He, David M. Bowers, Wei Wang
similarity 0.59
Abstract · 53 pages, 5 figures

Generative AI (GenAI) tools entered higher education classrooms faster than the field was able to study their effects on learning. One concern is that GenAI may displace the critical thinking and AI literacy that students will need after graduation. This paper reports a Design-Based Research pilot of a GenAI-assisted critical thinking framework, in which ChatGPT was used as a thinking partner in an undergraduate research methods and statistics course during Spring 2025 (N = 14). The mixed-methods design combined pre- and post-intervention measures of statistical learning (AASCDM), AI literacy (MAILS), and critical thinking (WGCTA) with instructor field notes, student artifacts, and student-AI interaction logs. Pre-post tests showed gains on every AASCDM dimension and on eight of nine MAILS dimensions, while WGCTA percentiles did not change. Qualitative analysis identified four themes: the ways students positioned the LLM (as answer generator, validator, or co-thinker); the depth of student engagement (procedural vs. conceptual); occasional humanizing of the tool; and the role of curriculum design in shaping each of the prior three. Read together, the findings indicate that one semester of GenAI-assisted instruction can move domain learning and self-reported AI literacy but does not move standardized critical thinking, and that the modal student-LLM relationship is one of validation instead of dialogue. We end with design principles for the next iteration of the framework and implications for research on adaptive and personalized learning.

0.20

Who Benefits, and When? Trajectories of Academic Achievement and Interest in AI-Supported Learning

cs.HC ·2026-11-07 ·Hyejeong Lee
similarity 0.58
Abstract

AI-supported learning environments are increasingly used, yet relatively little is known about how students' achievement and interest develop over time or whether these trajectories vary across learners. This study examined longitudinal trajectories of mathematics achievement and interest in a semester-long AI-supported mathematics learning. Using a mixed-methods longitudinal design, repeated measures of mathematics achievement were collected at four time points and mathematics interest at three time points. Linear mixed models (LMMs) were used to examine overall changes over time, while linear quantile mixed models (LQMMs) were used to examine differences in trajectories across the lower, median, and upper portions of the outcome distributions. Mathematics achievement increased substantially during the intervention, with the largest gains emerging between the second and third measurement occasions. However, the trajectory differed across the achievement distribution. Mathematics interest increased significantly after the first month, but this increase was not sustained through the end of the semester. Qualitative findings further indicated that students experienced the AI platform differently. The study highlights the importance of examining both cognitive and motivational trajectories.

0.24

Beyond Correctness: How Novice Software Engineering Students Exercise Oversight of Generative AI

cs.SE cs.HC ·2026-10-23 ·Ali Shakiba, Thomas Chaffey
similarity 0.57
Abstract · 10 pages, 3 figures, 2 tables

Generative AI (GenAI) is shifting software engineering students' work from producing artefacts towards evaluating and supervising AI-generated outputs. However, little is known about what characteristics students identify in outputs when reporting interventions, or how they exercise oversight during authentic software engineering work. We analysed 962 student-AI interaction turns documented by 145 first-year software engineering students through structured AI Journals, including 172 student-declared interventions. Using a multi-dimensional coding framework, we characterised the output characteristics identified in these reported interventions, students' reasons for intervening, the actions they took, and how those actions were enacted. Fitness problems were identified in 74.4% of interventions, compared with 16.9% involving correctness problems, indicating that oversight more often concerned whether plausible AI outputs were suitable for the engineering task than whether they contained explicit errors. Among classifiable interventions, students most frequently modified generated outputs (51.0%) or refined their prompts (28.2%), with both response types commonly associated with engineering fit. Within-student analysis further showed that outputs perceived as containing minor flaws were associated with lower evaluation confidence than outputs perceived as perfect. Taken together, these findings characterise novice AI oversight as a process that extends beyond error detection to encompass both judgement of fitness for purpose and active intervention in AI-assisted engineering work.

0.49

AI literacy over tool design: a mixed-methods study of scaffolded versus unrestricted generative AI in programming education

cs.CY ·2026-09-15 ·Sepinoud Azimi
similarity 0.57
Abstract

Generative AI has become a routine resource in programming education, and most institutional responses to it are attempts at control, either by restricting access or by offering students a controlled version of the technology. This paper reports a seven-week mixed-methods pilot study in a master's-level data analytics course, in which 33 students were randomly assigned either to a scaffolded AI Study Coach embedded in the notebook-based laboratory sessions or to unrestricted use of AI tools of their own choosing. The Coach offered stepwise hints, did not generate code, limited the number of hints per session, and required a short reflection at the end of each session. The design assumed, in line with scaffolding theory and recent experimental evidence, that guided and limited support would build confidence and reduce over-reliance, and that the scaffolded group would learn more. Assignment performance did not differ between the conditions. Students in the Coach condition reported higher confidence but managed the hint budget poorly, while students in the unrestricted condition were satisfied with their tools and uneasy about how much they depended on them. In interviews, students in both conditions identified awareness of their own reliance on AI as the most valuable outcome of the course. Students who had formulated their own rules for when to use AI performed better in both conditions, and those with the best understanding of how the models work, in every case self-taught, used the tools most deliberately and achieved the highest scores. The design of the tool mattered less than the students' capacity to govern their own use of it, a capacity that is at present acquired by chance. The paper argues that the appropriate response is structural: assessment that grades the reasoning behind AI-assisted work, and AI literacy taught explicitly as a core skill.

0.27