CS Paper Radar

Share your Screen and Talk to Gemini: How Young Learners Interact with Multimodal GenAI While Solving Scratch Tasks

Sven Jacobs, Lukas Berning, Natalie Kiesler

cs.AI cs.CYcs.HC ·submitted 2026-11-16 16:05 UTC · v1
published

Abstract

Children using Scratch to learn programming can struggle with text-heavy, lengthy explanations and hints during debugging. Verbal interactions based on screen sharing or camera data may be an interesting alternative for supporting learners. We therefore explore the quality, use, and perceptions of fifth- and sixth-graders' verbal interactions with a GenAI tutor. The goal is to investigate the quality and use of the multimodal GenAI tutor in the given context. First of all, a pilot study of eight tasks conducted without students helped inform the process of the tutor's classroom evaluation. The classroom study was realized after fifth- and sixth-graders had participated in six Scratch classroom lessons. Then 23 students were introduced to the tutor, worked on the eight tasks with the GenAI tutor, and completed a questionnaire. The analysis of the pilot and classroom study shows that the tutor's responses were intelligible and actionable, and no single turn revealed a complete solution. However, the tutor missed existing errors in 61.5% of 603 error-related classroom turns, compared to 15.7% in the pilot study. Children's voice prompts were brief and mostly responded to the tutor. Although a video had explained that responses could be incorrect, 39.1% of the children trusted the tutor's answers to always be correct. These findings have implications for educators and learners using multimodal GenAI tutors, as they need to be aware that an intelligible and human-like voice does not ensure correct support. For developers, our results suggest that screen sharing alone does not ensure responses that match the program state.

Comment
accepted for: The 21th WiPSCE Conference on Primary and Secondary Computing Education Research (WiPSCE 27)

Closest papers

Nearest neighbours by meaning across the whole index.

Exploring Visual Access and Proactive AI Tutoring for Handwritten Mathematics

cs.CY cs.HC ·2026-10-19 ·Nachiket Kotalwar, Conrad Borchers, Shubham Gandhi, Partha Sarathi Purkayastha, David Yaron +1 more
similarity 0.66
Abstract · 24 pages, 7 figures, 13 tables

Handwritten work captures mathematical reasoning that learners may not express in typed or voice-dictated messages. We built a multimodal AI tutoring platform in which a phone camera gives the tutor a live view of learners' work on paper. In a randomized online study, 117 adults solved calculus problems with Text-Reactive, Visual-Reactive, or Visual-Proactive assistance. Analysis of 2,140 tutor messages showed that visual access shifted assistance toward checking ongoing work, while proactivity enhanced error identification and directive hints. Learners valued feedback grounded in their writing; those who received proactive help rated it as supportive, and about a third wanted it more often. These findings inform when tutors should intervene and what they should address. We release the platform, prompts, and instruments, together with 255,984 timestamped images of handwritten work linked to tutoring conversations, intervention decisions, and learning outcomes, to support multimodal tutoring systems that preserve the affordances of handwritten work while providing adaptive feedback.

0.18

LLM Pedagogical Behavior in AI Tutoring Interactions

cs.CL cs.HC ·2026-08-24 ·Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin
top venue similarity 0.62
Abstract

Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.

0.45

"Can a Robot Really Care About Me?" How Perceiving a Tutor as Human or AI Affects Computer Science Students' Attitudes and Experiences

cs.HC ·2026-11-16 ·Tymon Vu, Wenxuan "Wendy" Shi, Silas Hsu
similarity 0.61
Abstract · 30 pages, 3 figures

Conversational artificial intelligence can reproduce many instructional and social behaviors of human tutors, but students may interpret those behaviors differently depending on who they believe is providing them. We examined how perceived tutor identity shapes students' evaluations of instructional feedback, satisfaction of psychological needs according to Self-Determination Theory, and their communication strategies with tutors. In a within-subjects Wizard-of-Oz study, 27 introductory computer science students completed two text-based tutoring sessions involving Python code-explanation problems. Participants were told one tutor was human and the other was AI, although the same human provided feedback in both conditions. We found participants evaluated both tutors favorably and reported few consistent differences in the instructional value of the feedback. However, participants frequently questioned whether AI could genuinely care about them, which corresponded with the human tutor having higher scores of relatedness need satisfaction. Human presence offered connection and accountability for some participants but introduced anxiety in others. Participants also reported communicating more casually with the human tutor and more explicitly with the AI tutor. These findings suggest complementary roles for human and AI tutoring and highlight the importance of considering psychological needs alongside instructional quality.

0.10

Peers4U: Designing Learner-Centered Virtual Peers for Children's Multimodal Mathematics Learning

cs.HC ·2026-10-27 ·Zhi Zheng, Binglin Liu, Weihao Chen, Lan Luo, Yu Mei +3 more
similarity 0.60
Abstract

In early mathematics, children learn abstract concepts through concrete representations, such as manipulating objects, drawing, and speaking. Learning with peers can enrich this process by exposing children to diverse strategies and enabling them to exchange ideas through interaction. We realized this potential with virtual peers in Peers4U, a multimodal learning system to enhance children's mathematical conceptual understanding. Virtual peers driven by LLMs are personalized to the learner's cognitive level and assigned multimodal solutions with inspectable processes. During learning, peers engage in mixed-initiative interactions by responding, initiating topics, and advancing discussions along a pedagogical progression. A user study with 22 children and two experts showed our learner-centered topic orchestration significantly outperformed a persona-centered baseline, and children spontaneously authored new artifacts to demonstrate inspired ideas or share their own strategies, producing significantly more solutions when grouped with more Leading Peers. We contribute design insights for multimodal, learner-centered virtual peer systems.

0.29

Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop

cs.CL ·2026-10-11 ·Sayantan Adak, Rima Hazra, Koyel Mukherjee, Digbalay Bose, Somnath Banerjee
position similarity 0.58
Abstract

Large language models can produce fluent tu- toring dialogue, but whether they actually help students learn remains unclear. We present a process-centered synthesis of LLM tutoring that follows the path from learner-state infer- ence to pedagogical action, tutor training, eval- uation, and student simulation. This view ex- poses a central gap: implementing a tutoring component does not establish that it works or that its outputs meaningfully affect later stages. A structured study-level audit shows that peda- gogical evaluation is common, while evidence becomes much thinner for learner-conditioned intervention, human learning gains, and espe- cially transfer or delayed retention. Human studies nevertheless provide a conditional posi- tive result: LLM tutors can improve learning, but benefits depend on pedagogical structure, learner engagement, and how learning is mea- sured. We argue that progress requires validat- ing the interfaces between tutoring stages, not only improving individual components.

0.22

Examining Variation in How Guided AI Tutors Resolve Student Impasses

cs.AI ·2026-09-29 ·Bakhtawar Ahtisham, Kirk Vanacore, Alessandra Napoli, Josh Arens, Ksenia Ionova +3 more
similarity 0.58
Abstract

When a student is stuck, a tutor faces the assistance dilemma: help given too early can hinder productive struggle, while help withheld too long leaves the student in a frustrating, persistent impasse (i.e., wheel spinning). Generative AI tutors increasingly use guardrails restricting answer-giving, yet little is known about how such tutors behave once an impasse persists. We analyze 20,462 student turns from 1,260 authentic sessions with a guided LLM chemistry tutor, identifying 6,630 impasse turns of three major types: conceptual errors, expressed uncertainty, or help-seeking. We then used these impasses to simulate three tutoring conditions to study variation in AI tutor guidance through impasses: baseline, no-direct-answer, and guided tutor. For a sample of 150 impasses, prompt specificity changed pedagogy: a baseline tutor provided the answer directly in 50.7% of responses, a no-direct-answer tutor asked a follow-up question every time, and the guided tutor responded in a wide variety of ways depending on the context. We then analyzed impasse trajectories in authentic interactions, finding that each additional impasse turn lowered the odds of next-turn recovery by 12.7% (AOR = 0.873, p < .001), and early dropouts were caught in recursive concept elicitation before reaching execution. The benefit of questioning decayed as impasses persisted (scripted question x depth AOR = 0.78; follow-up x depth AOR = 0.83), whereas addressing the student's error grew more beneficial (AOR = 1.14); after a failed scripted question, repeating it was followed by recovery in 28.1% of cases, compared with 39.8% when the tutor addressed the error instead. For learning analytics, these findings identify impasse depth and type as observable, turn-level dialogue signals that analytics can use to trigger graduated, state-sensitive assistance in real time.

0.13

Adapting for AI: How elementary teachers adjust their practices for an AI-integrated curriculum

cs.HC cs.AI ·2026-09-25 ·Fasika Melese, Ruiyang Wu, Xinyue Cui, Joanna Perkins, Xiaoyi Tian +2 more
similarity 0.58
Abstract

Conversational AI tools are entering children's everyday experiences, and schools are interested in adopting them. However, successful classroom integration depends not only on the technology but also on the work teachers do to make it usable and appropriate for their students and classroom context. There is little known about how elementary teachers work as they implement conversational AI tools in real classrooms. In this study, we examine three teachers' experiences implementing an AI literacy and English Language Arts (ELA) curriculum built around ToyTalk, a conversational AI toy development platform, over 13 instructional days, a three-week summer camp. Drawing on daily individual reflections, group reflections, and post-camp interviews, we find that teachers' adaptive practices of repair, differentiation, translation, and balancing sit at the intersection of three tensions (technology, learner, and instruction). Teachers' understanding of AI and their role evolved over the camp experiences. From these findings, we contribute design implications and considerations for deploying conversational AI within elementary classrooms.

0.23

For You, Not for Everyone: Bystander-Aware Delivery of Critical Feedback in Robot Tutoring

cs.HC cs.RO ·2026-11-03 ·Atikkhan Faridkhan Nilgar, Manuel Dietrich, Kristof Van Laerhoven
similarity 0.57
Abstract · Submitted for Review

Robots in shared settings may expose information intended for an individual user to nearby bystanders. Robot tutoring makes this concern particularly salient: critical feedback is directed at an individual learner, yet bystanders may overhear or see it. To address this, we compared context-invariant and context-adaptive robot behaviors in a between-subjects study with 40 participants. Following an initial phase without a bystander, a bystander entered the setting. Participants then received two scripted critical feedback messages concerning their pronunciation and handwriting, independent of their actual performance. With the context-invariant behavior, the robot tutor delivered both messages aloud. With the context-adaptive behavior, the robot tutor scanned for the bystander, turned its body and display away after detecting them, gave a spoken attention cue, and presented the same feedback messages silently on its display. Participants rated the robot tutor higher in perceived privacy awareness and perceived social competence in the context-adaptive condition than in the context-invariant condition, while evidence for reduced experienced embarrassment was inconclusive. These findings inform the design and evaluation of robots that respond to bystander presence by adapting how they communicate potentially sensitive information.

0.27