CS Paper Radar

Fluency Without Evidence: Constraint-First Design and the Limits of Self-Report in AI-Assisted Learning

Fatima T. Zahra, Wei Wang, Frances Harper, Jiangen He

cs.HC ·submitted 2026-09-29 15:53 UTC · v1

Abstract

A generative AI teaching partner should support reasoning over supplying conclusions; however, this has not been tested against learning in an authentic course. Drawing on design-based research, we specify the position as a conjecture map and report a first design cycle in two graduate-level research methods courses. Students used an AI teaching partner employing a constraint-first sequence requiring them to state and justify positions before receiving questions. Pre- and post-measures of AI literacy, critical thinking, and metacognitive awareness were collected alongside interaction records. AI literacy increased, concentrating in understanding AI, whereas critical thinking, awareness, and knowledge did not change. Since changes were limited to self-report measures, they may reflect growth in confidence instead of capacity. Interaction records, meanwhile, showed brief exchanges, uneven enactment of the constraint-first sequence, and missing records. These findings show why AI-supported learning requires interaction records to provide a more defensible basis for AI-supported designs than self-reports.

Comment
37 pages, 1 figure

Closest papers

Nearest neighbours by meaning across the whole index.

Critical Thinking with Generative AI: A Constraint-First Design Pilot of a Thinking-Partner Intervention

cs.HC ·2026-09-29 ·Fatima Tuz Zahra, Jiangen He, David M. Bowers, Wei Wang
similarity 0.72
Abstract · 53 pages, 5 figures

Generative AI (GenAI) tools entered higher education classrooms faster than the field was able to study their effects on learning. One concern is that GenAI may displace the critical thinking and AI literacy that students will need after graduation. This paper reports a Design-Based Research pilot of a GenAI-assisted critical thinking framework, in which ChatGPT was used as a thinking partner in an undergraduate research methods and statistics course during Spring 2025 (N = 14). The mixed-methods design combined pre- and post-intervention measures of statistical learning (AASCDM), AI literacy (MAILS), and critical thinking (WGCTA) with instructor field notes, student artifacts, and student-AI interaction logs. Pre-post tests showed gains on every AASCDM dimension and on eight of nine MAILS dimensions, while WGCTA percentiles did not change. Qualitative analysis identified four themes: the ways students positioned the LLM (as answer generator, validator, or co-thinker); the depth of student engagement (procedural vs. conceptual); occasional humanizing of the tool; and the role of curriculum design in shaping each of the prior three. Read together, the findings indicate that one semester of GenAI-assisted instruction can move domain learning and self-reported AI literacy but does not move standardized critical thinking, and that the modal student-LLM relationship is one of validation instead of dialogue. We end with design principles for the next iteration of the framework and implications for research on adaptive and personalized learning.

0.20

Outcome Over Learning? Evidence from Student Use of AI Code Assistants for Unit Test Writing

cs.SE cs.AI ·2026-11-10 ·Daniel Hinbest, Jeremy S. Bradbury, Michael A. Miljanovic
similarity 0.65
Abstract · 8 pages

Artificial intelligence (AI) code assistants are rapidly being adopted in programming education, yet little is known about how students naturally use these tools when given unrestricted access. In a user study investigating AI-assisted unit test writing, we observed that a majority of students adopted a one-shot interaction strategy, asking the AI to generate a complete unit test directly from the initial requirements. While this approach is capable of efficiently producing a solution, it shifts students' focus from learning how to write effective unit tests to simply completing the task. Moreover, because students rarely engaged in critically evaluating, refining, or iterating on AI-generated output, they may be missing opportunities to develop the critical thinking skills that are essential for effective and responsible use of AI. In an effort to shift student focus from outcome to learning we also discuss how training, accountability and improved learning task design can better support a more learning-centric responsible use of AI.

0.37

Three Pathways of Student-AI Interaction: Constraint-First Design for Higher-Order Thinking

cs.HC ·2026-09-29 ·Fatima T. Zahra
similarity 0.65
Abstract · 26 Pages, 2 figures

How students interact with artificial intelligence (AI) systems in educational settings may determine whether that interaction supports or displaces critical thinking. This paper introduces two contributions. The first is the Three Paths of Student-AI Interaction, a typological framework identifying three qualitatively distinct modes of student-AI engagement: Passive Review, Direct Question, and Strategic Dialogue. The second is the Next Level Teaching Blueprint (NLTB), a three-stage instructional design system intended to make Strategic Dialogue more likely. Qualitative content analysis of 50 randomly sampled student-AI interaction messages from an undergraduate research methods course was used to examine the typology. Two human coders achieved 68% path-level agreement ($κ$ = .48), with 80% agreement on Strategic Dialogue identification specifically. GPT-5, used as a third coder, produced a similar overall distribution and introduced a coding category absent from the human scheme. Path 1 (Passive Review) accounted for 46% of exchanges in the primary researcher's classifications, Path 2 (Direct Question) for 18%, and Path 3 (Strategic Dialogue) for 36%. A second, descriptively examined dataset contained predominantly Strategic Dialogue content, offering a preliminary indication that instructional framing may influence which path students take. Together, the Three Paths framework and the NLTB contribute a language for describing student-AI interaction and a design approach for supporting higher-order engagement.

0.28

What does it mean to use AI critically? Unpacking critical AI literacy through students' evaluation of AI-generated content

cs.HC ·2026-10-07 ·Hyejeong Lee, Wonjin Yu
similarity 0.62
Abstract

Although critical AI literacy has emerged as an important educational goal, the construct remains conceptually broad and insufficiently specified for guiding students' day-to-day interactions with AI. This study examines how students critically evaluated AI-generated content. Drawing on an analysis of students' chatbot interactions and written reflections, we first identified seven stages of AI-supported academic task completion and five functional roles assumed by AI. More importantly, we identified eight evaluative lenses that students used to assess AI-generated responses (accuracy, completeness, task alignment/relevance, personalization, practicality, creativity, ethics, and bias). The findings suggest that critical AI literacy extends beyond verifying the factual accuracy of AI-generated information. Rather, critically engaging with AI involves situated and multidimensional judgment about the epistemic quality, contextual fit, practical feasibility, creative value, and ethical implications of AI-generated content. The framework offers both a conceptual contribution to the emerging literature on critical AI literacy and a practical tool for educators seeking to help learners move from passive acceptance of AI toward deliberate, context-sensitive, and responsible human judgment.

0.08

AI literacy over tool design: a mixed-methods study of scaffolded versus unrestricted generative AI in programming education

cs.CY ·2026-09-15 ·Sepinoud Azimi
similarity 0.62
Abstract

Generative AI has become a routine resource in programming education, and most institutional responses to it are attempts at control, either by restricting access or by offering students a controlled version of the technology. This paper reports a seven-week mixed-methods pilot study in a master's-level data analytics course, in which 33 students were randomly assigned either to a scaffolded AI Study Coach embedded in the notebook-based laboratory sessions or to unrestricted use of AI tools of their own choosing. The Coach offered stepwise hints, did not generate code, limited the number of hints per session, and required a short reflection at the end of each session. The design assumed, in line with scaffolding theory and recent experimental evidence, that guided and limited support would build confidence and reduce over-reliance, and that the scaffolded group would learn more. Assignment performance did not differ between the conditions. Students in the Coach condition reported higher confidence but managed the hint budget poorly, while students in the unrestricted condition were satisfied with their tools and uneasy about how much they depended on them. In interviews, students in both conditions identified awareness of their own reliance on AI as the most valuable outcome of the course. Students who had formulated their own rules for when to use AI performed better in both conditions, and those with the best understanding of how the models work, in every case self-taught, used the tools most deliberately and achieved the highest scores. The design of the tool mattered less than the students' capacity to govern their own use of it, a capacity that is at present acquired by chance. The paper argues that the appropriate response is structural: assessment that grades the reasoning behind AI-assisted work, and AI literacy taught explicitly as a core skill.

0.27

Configuring AI for Exploratory Learning: Educator Perspectives on Inference, Persistence, and Disclosure

cs.HC ·2026-09-17 ·Si Chen, Xinyue Chen, Artur Mullagaliyev, Alexander Nwanganga, Shifu Hou +3 more
similarity 0.61
Abstract · 32 pages

Generative AI (GenAI) can support open-ended learning through generation, personalization, and learner modeling, yet educators need ways to shape these capabilities around educational goals. Through interviews and design activities with 15 U.S. educators, we examined educator configuration of GenAI using K-12 career exploration as an exploratory context. Educators configured not only AI-generated experiences, but also when student activity became an inference, whether learner information persisted, who could access it, and how it informed subsequent human action. They also faced challenges translating teaching needs into configurations: recognizing possibilities for control beyond familiar uses of GenAI, decomposing general-purpose AI into understandable functions and responsibilities, and identifying useful information through intended teaching actions. We discuss how GenAI systems can support educators in expressing and testing configurations, while establishing boundaries around personalization, inference, persistence, disclosure, and action to keep AI-supported learning aligned with evolving learner needs.

0.20

How assigned AI use before class shapes active student engagement in class

cs.HC cs.AI ·2026-10-07 ·Dan J. Wang, Neelam Modi Jain, Vanessa Burbano, Jorge Guzman, Daniel Keum +3 more
similarity 0.61
Abstract

AI learning tools are rapidly entering classrooms, but evidence about whether they help students learn is mixed and rests mostly on test scores. Comparatively less research addresses whether the use of AI changes students' live learning behaviors in class. Here, we report the results of a preregistered field experiment with 759 MBA students enrolled in ten sections of a course, in which each student was randomly assigned two of ten class sessions to prepare for with a purpose-built voice-based AI discussion partner. After two uses of the AI discussion partner, students made about 31% more voluntary contributions in each later class session. Students who used the AI discussion partner more also reported greater comfort speaking up and greater perceived learning, but not greater focus or motivation. These findings suggest that repeated practice with a voice-based AI partner can meaningfully increase students' engagement in class discussion, enhancing a critical intermediate learning outcome.

0.44

Exploring K-12 Teachers' Perceptions of Students' Relationships with AI Companions: Boundaries, Intervention Strategies, and Design Implications

cs.HC cs.CY ·2026-09-11 ·Qing Xiao, Wenhan Xie, Ziyu Deng, Ruiwei Xiao, Ziyue Feng +5 more
similarity 0.61
Abstract · 17 pages

K-12 students increasingly form relationships with AI companions. Schools face growing expectations to teach AI literacy, yet existing frameworks treat AI as a tool rather than a relationship, and little is known about how teachers understand and act on students' relational use of AI. We conducted scenario-based interviews with 33 US K-12 teachers. Teachers welcomed academic companions but worried that intimate companions remove the developmental friction through which students learn to sustain human relationships. Teachers drew the boundaries of their jurisdiction by setting and observable wellbeing: within it they taught, talked, and watched; beyond it they positioned themselves as the adults best placed to notice and connect students with support. They envisioned AI companion literacy as shared work across the jurisdictions of counselors, parents, platforms, and policymakers, spiraling across grade levels. We introduce AI companion literacy as an extension of AI literacy and discuss implications for K-12 AI education.

0.37