CS Paper Radar

Will It Teach as Intended? How Teachers Configure Educational AI Chatbots

Bahare Riahi, Deniz Ozturk, Alice Guth, Jiayu Li, Daksh Pratap Singh, Xiaoyi Tian, Jennifer Chiu, Nicholas Lytle, Tiffany Barnes, Veronica Catete

cs.HC ·submitted 2026-09-24 15:42 UTC · v2

Abstract

Teachers are increasingly using generative AI to support instruction, yet it remains unclear how pedagogical intentions are translated into chatbot configurations and reflected in chatbot behavior. We studied a teacher-facing chatbot authoring tool in professional development workshops with 27 middle school teachers, analyzing focus-group interviews alongside configuration and interaction logs. Teachers envisioned chatbots as instructional scaffolds that could provide differentiated support, extend access to assistance, and preserve student thinking within teacher-defined boundaries. Configuration analysis showed that Purpose primarily captured instructional goals and content focus, whereas Rules more often specified pedagogical behavior, guardrails, and learner-specific adaptations. Log-based evaluation showed stronger alignment for responsiveness (88.9%) and persona (81.5%) than for rules (70.4%) and purpose (59.3%). These findings show that configurable controls alone do not ensure pedagogical fidelity and highlight the need for authoring tools that help teachers express, test, and refine intended chatbot behavior.

Comment
17 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Adapting for AI: How elementary teachers adjust their practices for an AI-integrated curriculum

cs.HC cs.AI ·2026-09-25 ·Fasika Melese, Ruiyang Wu, Xinyue Cui, Joanna Perkins, Xiaoyi Tian +2 more
similarity 0.71
Abstract

Conversational AI tools are entering children's everyday experiences, and schools are interested in adopting them. However, successful classroom integration depends not only on the technology but also on the work teachers do to make it usable and appropriate for their students and classroom context. There is little known about how elementary teachers work as they implement conversational AI tools in real classrooms. In this study, we examine three teachers' experiences implementing an AI literacy and English Language Arts (ELA) curriculum built around ToyTalk, a conversational AI toy development platform, over 13 instructional days, a three-week summer camp. Drawing on daily individual reflections, group reflections, and post-camp interviews, we find that teachers' adaptive practices of repair, differentiation, translation, and balancing sit at the intersection of three tensions (technology, learner, and instruction). Teachers' understanding of AI and their role evolved over the camp experiences. From these findings, we contribute design implications and considerations for deploying conversational AI within elementary classrooms.

0.23

EduWild: How Teacher-Configured Chatbots Fail in Real-World Use and How to Fix Them

cs.CL ·2026-10-16 ·Ziqi Liu, Ruiwei Xiao, Yuxuan Li, John Stamper
dataset similarity 0.70
Abstract

Teachers increasingly configure educational chatbots through natural-language instructions, but how these bots fail in real-world use remains poorly understood. We introduce EduWild, a dataset of 1,726 conversations with 217 teacher-configured chatbots, each paired with the instructions and configuration behind it, including 900 conversations from the intended end users of 35 deployed chatbots. We develop a 29-code taxonomy covering factual, workflow, diagnostic, and pedagogical failures and apply it using a human-calibrated LLM judge. The most frequent failures are lapses in following teacher intent --- bots skip required workflow steps (53.8% of end-user conversations), omit expectation setup (33.4%), and overload learners with information or questions (37.1%). Guided by these findings, we design a prompt-based harness targeting the six most frequent failure modes. In a study with 200 college students across ten bots, the harness reduces the mean number of target failure codes per conversation from 1.406 to 0.489, a 65.2% relative reduction with bots weighted equally. These findings show how real-world failure analysis can guide practical improvements to educational chatbots.

0.26

Thinking Is Not Writing: Designing AI Chatbots to Connect Reasoning and Revision

cs.HC ·2026-11-05 ·Lucile Favero, Tanja Käser, Juan Antonio Pérez-Ortiz, Nuria Oliver
similarity 0.69
Abstract · 21 pages, 11 figures, 2 tables

Supporting students' argumentative writing with AI requires connecting reflection to their revision processes. We present Maike, a Socratic chatbot supporting writing through argument diagnosis, targeted questioning, and a student model across Think and Write modes. In a classroom study, 65 secondary-school students used Maike or a control chatbot sharing the interface, language model, and Think/Write modes. Approximately 40% of students improved their essay scores. However, Maike users engaged more in idea-level dialogue (56% vs. 44%) and moved iteratively between discussion and editing. Transitions from ideas discussed in dialogue to revisions were positively associated with score gains. These findings suggest a design target for AI writing support: coordinating reflection with revision. We contribute an architecture for supporting argumentative reasoning without generating essay text and an approach linking interaction patterns to writing processes and outcomes. As AI becomes embedded in classrooms, understanding these connections is essential to assessing (and designing for) its educational value.

0.12

How a Chatbot's Response Style Shapes a Classroom: A Multi-Agent Simulation of Students Consulting AI

cs.HC cs.AIcs.CYcs.MA ·2026-09-04 ·Rin Tamai, Yuya Dan
similarity 0.67
Abstract · 61 pages

Chatbots built on large language models (LLMs) are increasingly used as confidants. Tuned to satisfy users, they may answer with excessive empathy and affirmation that fosters dependence, and how the states and relationships of many users co-evolve under repeated consultation is hard to observe in real settings. We build a virtual classroom of 20 student agents who interact through rule-based chats, quarrels and consultations with friends and, when stressed, may instead consult a counselor AI (Gemini 2.5 Flash) under one of six style prompts: affirming, listening, solution-oriented, reality-redirecting, inciting and blaming. A second LLM call turns each exchange into updates of five state variables (stress, happiness, self-reliance, sociability, AI dependence) without seeing the prompt. We compare the seven conditions, including a no-AI control, over 15 and 50 days and under a lower consultation threshold, and test the robustness of the 50-day comparison with a pre-specified protocol: the same block in ten independent classrooms, repeated LLM realizations of one classroom with its event stream fixed, and evaluator updates scaled by 0.3 and 0.1. In every classroom the affirming and inciting prompts ended with lower self-reliance and higher AI dependence than the control, and the listening, reality-redirecting, inciting and blaming prompts with higher stress, lower happiness and more non-attendance; the solution-oriented prompt did not differ consistently from the control. The robust self-reliance and AI-dependence differences kept their signs at the 0.3 scale with highly similar rankings (Spearman 0.89, 0.93); the stress and happiness rankings did not, and the affirming prompt's lower stress reversed its sign. All quantities are simulation state variables, not effects on users. We specify the agent dynamics completely and discuss the limits of an LLM as generator of state updates.

0.26

How Children Design and Reason about Trustworthy AI Chatbots

cs.HC cs.AI ·2026-09-21 ·Deniz Ozturk, Jiayu Li, Daksh Pratap Singh, Yasitha Rajapaksha, Fasika Melese +7 more
similarity 0.66
Abstract

Children increasingly interact with AI chatbots, making trust calibration essential to AI literacy. Prior research has examined children's trust in AI mainly as users evaluating systems built by others, rather than as designers of their own chatbots. We developed a chatbot-building environment with adjustable trust-relevant traits (e.g., confidence, transparency, formality, assertiveness), rules, and persona. We conducted mixed-methods study with 115 learners (ages 8-18) who made 119 chatbots. We examined how children configured their chatbots, reasoned about trustworthiness, and how closely chatbot behavior aligned with their designs. Younger students (age 10-13) set significantly higher confidence than older students (age 14-18), and some deliberately built chatbots that gave wrong answers on purpose, yet still called them trustworthy, arguing that a chatbot does what it was built to do. Younger students equated trust with purpose-fulfillment, while older students linked it to transparent, calibrated design. Students also calibrated academic chatbots to be more transparent and formal than hobby chatbots. We identify seven design dimensions describing what children believe makes a chatbot trustworthy, and discuss implications for AI literacy tools.

0.15

From Instructional Intent to Classroom-Ready Lessons: Investigating Teacher-AI Collaboration in Lesson Authoring for Secondary Computing Education

cs.HC ·2026-10-09 ·Tahreem Yasir, Wenbo Li, Deniz Ozturk, Yasitha Rajapaksha, Dmitri Droujkov +2 more
similarity 0.65
Abstract

Generative AI is increasingly studied as a tool for supporting lesson authoring; the creation and adaptation of classroom materials, yet little is known about how it should fit into secondary computing teachers' subject-specific practices. We conducted two complementary qualitative studies with eight secondary computing teachers: focus groups examining existing authoring practices and a think-aloud study using LessonBuilder, a structured AI-supported lesson authoring system, as a research probe. We found that teachers' instructional intentions developed through creating, inspecting, and adapting materials, while teachers selectively delegated tasks to AI and retained responsibility for judging classroom readiness. The think-aloud sessions further showed how examining AI-generated materials prompted teachers to clarify and refine their intentions, making authoring an ongoing negotiation between what teachers intended and what the AI produced. In the context of computing lessons, classroom readiness also involved considering how programming materials and supporting guidance matched students' understanding and intended learning goals. We synthesize these findings into a conceptual model explaining how teachers coordinate instructional intent, AI assistance, and judgments of classroom readiness during lesson authoring. We derive design recommendations for supporting this coordination while preserving teacher control and addressing computing-specific authoring needs.

0.11

HelpCoach: Scaffolding Targeted AI Help-Seeking During Problem-Solving

cs.HC ·2026-09-24 ·Hyoungwook Jin, Weirui Peng, Jieun Han, Q. Vera Liao, Xu Wang
similarity 0.65
Abstract

Students increasingly turn to AI for help with problem-solving, yet too much AI support can undermine learning itself. To benefit from AI, students need to specify the necessary knowledge and scaffold type in their questions. However, they struggle to formulate such targeted questions because they lack metacognitive skills to recognize and select effective help options. We developed HelpCoach, an add-on for chat interfaces that helps students formulate knowledge- and scaffold-specific questions and receive targeted help during problem solving. HelpCoach continuously assesses students' help-seeking performance and prompts students to improve through an adaptive revision template. Whereas prior work has largely taught help-seeking skills apart from learning tasks, HelpCoach's in situ scaffold enables concrete practice on metacognitive skills and immediate revisions to help-seeking behavior. In a study with 40 college students learning web programming, HelpCoach led to more specific questions during chatbot interactions and greater knowledge retention than pre-task help-seeking training alone.

0.17

Instructor-designed AI improves learning in a randomized trial across 19 universities

cs.AI cs.CYcs.ETcs.HC ·2026-11-14 ·Jun Li, Robert Bray, Sébastien Martin, Nils Van den Steen, Andrew Wu +47 more
similarity 0.62
Abstract

Universities are alarmed by recent evidence that artificial intelligence (AI) can undermine learning, with students prescribed generic chatbots or centrally designed AI tools performing worse. In a pre-registered randomized controlled trial across 7,200 students at 19 universities over three semesters, we tested whether AI designed by frontline instructors can improve learning. We encouraged instructors to design AI solutions they believed would best help their students. Drawing on their unique knowledge of their classes and students, the instructors created a wide array of innovations. We show that these instructor-designed AI innovations improved student learning, measured by AI-proof assessments (proctored, in-person examinations). We found no change in student engagement, measured by course activity records, or satisfaction, measured by course evaluations. Our findings demonstrate that AI can improve student learning when its adoption is frontline-driven, with instructors empowered to design and refine AI solutions for their own classrooms.

0.22