CS Paper Radar

Interaction Effects Between Learner Characteristics and Dialogue Format in LLM- and TTS-Generated Dialogue-Based Lessons

Fumie Watanabe, Gendo Kumoi, Tota Suko, Takashi Ishida, Yuko Kuma, Manabu Kobayashi, Shigeichi Hirasawa

cs.CY ·submitted 2026-08-21 07:43 UTC · v2

Abstract

Large language models and text-to-speech technology enable low-cost production of dialogue-based lessons: videos in which two speakers converse. The speakers' roles can vary, but little is known about whether all learners respond alike to such differences. In a repeated-measures design, 222 first-year high school students viewed lessons in three formats: a teacher and a student (TS), two students (SS), and two teachers (TT). Within the aptitude-treatment interaction framework, linear mixed-effects models tested whether associations between learner characteristics and outcomes differed by format. The learner characteristics were experiential learning style - seeking challenging, new experiences (CE factor) and learning by reflecting on experience (RCE factor) - and critical thinking disposition; the outcomes were motivation based on the ARCS model, self-reported learning outcomes, and overall evaluation. For motivation, both factors interacted with format: the CE factor's association differed between TS and TT, and the RCE factor's slope was significantly positive in TS and SS and significantly larger in TS than TT. For learning outcomes, no interaction was detected, but critical thinking disposition was positively associated with them. For overall evaluation, again no interaction was detected, but TT was rated lowest. Given small-to-medium interaction effects, these findings are preliminary evidence to inform personalized learning design.

Comment
Revised version of a manuscript submitted to Education Sciences (MDPI); not yet peer-reviewed. The title has been changed from v1. 23 pages, 1 figure

Closest papers

Nearest neighbours by meaning across the whole index.

Bias and Reliability in Learning-Style Inference from Student--AI Dialogues: An Evaluation of GPT-4o and Gemini 2.0 Flash

cs.HC ·2026-10-20 ·Samaneh Zamanifard, Sajad Goudarzi, Andrew Robb
similarity 0.64
Abstract

Large Language Models (LLMs) are increasingly integrated into educational technologies for personalized support, creating opportunities for real-time learner modeling from dialogue. This study evaluates whether GPT-4o and Gemini 2.0 Flash can infer three Felder-Silverman Learning Style Model dimensions (Active-Reflective, Sensing-Intuitive, and Sequential-Global) from short, task-focused student-AI problem-solving dialogues. Using questionnaire-derived Index of Learning Styles labels from 153 undergraduate participants, we compared deterministic zero-shot and few-shot prompting under five-fold cross-validation. Although accuracy and support-weighted F1 were moderate for Sensing-Intuitive and Sequential-Global, majority-class baselines, macro-F1, balanced accuracy, and per-class recall showed that performance was often driven by dominant-label prediction rather than reliable recognition of both classes. The models substantially under-identified Intuitive and Global learners; few-shot prompting produced mixed effects and sometimes amplified majority-class bias. Active-Reflective classification remained unstable across models. These findings indicate that LLM-generated learner-style labels should be treated as uncertain, bias-prone signals rather than stable learner profiles, and that dialogue-based personalization systems require bias-aware evaluation and uncertainty-sensitive design.

0.13

From Written Response to Dialogue with AI: How Activity Format, Interaction Modality, and Language Impact Student Learning and Engagement

cs.HC cs.CY ·2026-10-05 ·Deepak Varuvel Dennison, Connie Zhang, Rene Kizilcec, Aditya Vashistha
similarity 0.59
Abstract

The widespread availability of LLMs is challenging written learning activities, as students can increasingly generate responses without necessarily engaging with the learning content. Conversational AI creates an opportunity to redesign these activities as dialogue, while multilingual capabilities may make such dialogue more accessible to students learning through a non-native language. We conducted a field study with 305 native Kannada-speaking undergraduate students at English-medium institutions in Karnataka, India. We compared written response activities with text- and voice-based dialogic activities with AI, each conducted in English-only or bilingual Kannada-English settings. Students who completed dialogic activities spent more time on the activities, contributed more, and reported greater interest and self-efficacy than those completing written responses, although fewer students completed the dialogic activities overall. Knowledge increased across all conditions, with no reliable differences in gains between activity formats or languages. Language shaped participation differently across modalities: bilingual interaction was particularly beneficial in voice dialogue, where it reduced articulation difficulties, increased turns demonstrating understanding, and reduced conversation abandonment. However, students also valued English because of its connection to their academic and professional aspirations. These findings show that designing dialogic learning with AI requires more than choosing between writing and dialogue, voice and text, or English and students' native languages. We highlight opportunities to give learners greater control over modality, information, and pace, and to use native languages as translanguaging support rather than as a replacement for English.

0.24

A Prompt-Engineering Approach to Develop Scalable, Flexible, and Real-Time Hybrid Micro-Level Personalization in a General Purpose AI Teaching Assistant

cs.AI ·2026-09-03 ·Saptarshi Basu, Sandeep Kakar, Ashok Goel
similarity 0.59
Abstract · 7 pages, 9 figures, IAAI27 conference

Artificial intelligence (AI) teaching assistants powered by large language models (LLMs) offer scalable educational support but often provide limited personalization. This study presents a prompt-engineering-based framework for personalizing general-purpose LLM/RAG-based AI teaching assistants such as Jill Watson across academic disciplines and courses. The framework adapts responses using six learner-specific dimensions: self-assessment, abstraction preference, verbosity preference, perceptual orientation, information processing style, and level of understanding, yielding 96 distinct learner profiles. Student queries are additionally analyzed using Bloom's Taxonomy to estimate cognitive complexity at the interaction level. Learner attributes and cognitive assessments are encoded in structured prompts that condition the LLM without requiring model retraining. The framework is evaluated through experiments using NLP metrics and a human study with five participants. Results show perceived differences in response style and structure across personalization conditions, with statistical analyses identifying learner attributes associated with measurable response changes. These findings provide preliminary evidence that prompt-based personalization can support adaptive behavior in LLM-powered educational agents.

0.16

From Dialogue to Task: How Cognitive Engagement With Generative Artificial Intelligence Relates to Post-Dialogue Activity and Task Outcomes

cs.CY cs.HC ·2026-11-08 ·Xiaoyu Zhuang, Aneesha Bakharia, Dragan Gašević, Hassan Khosravi
similarity 0.58
Abstract · 31 pages, 12 figures. Submitted to the Journal of Computer Assisted Learning

Generative AI is increasingly embedded within learning tasks, yet interaction frequency reveals little about cognitive engagement in student-AI dialogue. Little is known about how such engagement is situated within task workflows or how dialogue characteristics and subsequent task activity relate to students' behaviour and task outcomes. This study examined cognitive engagement in student-AI dialogue, its position within task workflows, and associations of dialogue and task-process characteristics with post-dialogue activity and task outcomes. Data were collected from 1,681 students across 40 university courses using three AI companions embedded in resource creation, peer moderation, and practice. Cognitive engagement in 829 conversation threads was coded with a dialogue-oriented Interactive-Constructive-Active-Passive (ICAP) framework, and 2,969 task sequences were modelled with first-order Markov models. Regression models linked dialogue and task-process characteristics to behavioural re-engagement, peer-rated artefact quality, and practice performance. Passive and Active engagement predominated, although resource creation involved more Constructive and Interactive dialogue, peer moderation more Passive, and practice more Active, while higher-engagement dialogue was less often followed by session termination. In resource creation, student-composed messages were associated with more dialogue-to-task returns than template-only requests at the same Passive level, whereas deeper cognitive engagement was associated with higher final artefact quality independent of return frequency; in practice, AI dialogue after an incorrect attempt was associated with fewer further attempts but higher next-attempt correctness among continuers. Post-dialogue activity and task outcomes are distinct aspects of AI use that interaction frequency cannot capture and evaluations should assess separately.

0.12

"I Wrote, I Paused, I Rewrote": A Mixed-Methods Study of Learner Perceptions of Process-Aware LLM Feedback

cs.HC cs.CL ·2026-09-04 ·Samra Zafar, Shifa Yousaf
similarity 0.57
Abstract · 8 pages, 1 figure, 6 tables

Large language models increasingly provide formative writing feedback, but most systems evaluate only the submitted text. Process-aware systems can additionally incorporate intermediate drafts and interaction records, yet access to these traces does not establish whether learners perceive the resulting feedback as useful, fair, or aligned with their own revision experience. We conducted a convergent mixed-method post-task evaluation with 43 undergraduate English-language learners. Participants completed a timed essay, received LLM-generated feedback based on the final essay, intermediate snapshots, and keystroke-derived records, and then answered six five-point Likert items and four open-ended questions. Ratings were concentrated above the neutral midpoint across all six items. Between 74.4% and 95.3% of participants selected ratings of 4 or 5, and all six one-sided exact sign tests remained significant after Holm correction. Perceived correspondence between the revision analysis and the participant's reported thought process was positively associated with overall perceived usefulness, Spearman's $ρ_s=.327$, 95% bootstrap CI $[.009,.609]$, $p=.032$, $n=43$. Qualitative responses were analysed through an LLM-assisted codebook development procedure followed by independent human coding of 14 binary codes. Participants frequently described the product feedback and revision rationale as accurate, while a smaller set reported misinterpretation of surface corrections, time constraints, or rhetorical intention. The findings distinguish perceived detection of revision activity from perceived interpretation of why revision occurred.

0.26

Critical Thinking with Generative AI: A Constraint-First Design Pilot of a Thinking-Partner Intervention

cs.HC ·2026-09-29 ·Fatima Tuz Zahra, Jiangen He, David M. Bowers, Wei Wang
similarity 0.57
Abstract · 53 pages, 5 figures

Generative AI (GenAI) tools entered higher education classrooms faster than the field was able to study their effects on learning. One concern is that GenAI may displace the critical thinking and AI literacy that students will need after graduation. This paper reports a Design-Based Research pilot of a GenAI-assisted critical thinking framework, in which ChatGPT was used as a thinking partner in an undergraduate research methods and statistics course during Spring 2025 (N = 14). The mixed-methods design combined pre- and post-intervention measures of statistical learning (AASCDM), AI literacy (MAILS), and critical thinking (WGCTA) with instructor field notes, student artifacts, and student-AI interaction logs. Pre-post tests showed gains on every AASCDM dimension and on eight of nine MAILS dimensions, while WGCTA percentiles did not change. Qualitative analysis identified four themes: the ways students positioned the LLM (as answer generator, validator, or co-thinker); the depth of student engagement (procedural vs. conceptual); occasional humanizing of the tool; and the role of curriculum design in shaping each of the prior three. Read together, the findings indicate that one semester of GenAI-assisted instruction can move domain learning and self-reported AI literacy but does not move standardized critical thinking, and that the modal student-LLM relationship is one of validation instead of dialogue. We end with design principles for the next iteration of the framework and implications for research on adaptive and personalized learning.

0.20

Three Pathways of Student-AI Interaction: Constraint-First Design for Higher-Order Thinking

cs.HC ·2026-09-29 ·Fatima T. Zahra
similarity 0.57
Abstract · 26 Pages, 2 figures

How students interact with artificial intelligence (AI) systems in educational settings may determine whether that interaction supports or displaces critical thinking. This paper introduces two contributions. The first is the Three Paths of Student-AI Interaction, a typological framework identifying three qualitatively distinct modes of student-AI engagement: Passive Review, Direct Question, and Strategic Dialogue. The second is the Next Level Teaching Blueprint (NLTB), a three-stage instructional design system intended to make Strategic Dialogue more likely. Qualitative content analysis of 50 randomly sampled student-AI interaction messages from an undergraduate research methods course was used to examine the typology. Two human coders achieved 68% path-level agreement ($κ$ = .48), with 80% agreement on Strategic Dialogue identification specifically. GPT-5, used as a third coder, produced a similar overall distribution and introduced a coding category absent from the human scheme. Path 1 (Passive Review) accounted for 46% of exchanges in the primary researcher's classifications, Path 2 (Direct Question) for 18%, and Path 3 (Strategic Dialogue) for 36%. A second, descriptively examined dataset contained predominantly Strategic Dialogue content, offering a preliminary indication that instructional framing may influence which path students take. Together, the Three Paths framework and the NLTB contribute a language for describing student-AI interaction and a design approach for supporting higher-order engagement.

0.28

LLM Pedagogical Behavior in AI Tutoring Interactions

cs.CL cs.HC ·2026-08-24 ·Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin
top venue similarity 0.56
Abstract

Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.

0.46