CS Paper Radar

Bias and Reliability in Learning-Style Inference from Student--AI Dialogues: An Evaluation of GPT-4o and Gemini 2.0 Flash

Samaneh Zamanifard, Sajad Goudarzi, Andrew Robb

cs.HC ·submitted 2026-10-20 00:50 UTC · v1

Abstract

Large Language Models (LLMs) are increasingly integrated into educational technologies for personalized support, creating opportunities for real-time learner modeling from dialogue. This study evaluates whether GPT-4o and Gemini 2.0 Flash can infer three Felder-Silverman Learning Style Model dimensions (Active-Reflective, Sensing-Intuitive, and Sequential-Global) from short, task-focused student-AI problem-solving dialogues. Using questionnaire-derived Index of Learning Styles labels from 153 undergraduate participants, we compared deterministic zero-shot and few-shot prompting under five-fold cross-validation. Although accuracy and support-weighted F1 were moderate for Sensing-Intuitive and Sequential-Global, majority-class baselines, macro-F1, balanced accuracy, and per-class recall showed that performance was often driven by dominant-label prediction rather than reliable recognition of both classes. The models substantially under-identified Intuitive and Global learners; few-shot prompting produced mixed effects and sometimes amplified majority-class bias. Active-Reflective classification remained unstable across models. These findings indicate that LLM-generated learner-style labels should be treated as uncertain, bias-prone signals rather than stable learner profiles, and that dialogue-based personalization systems require bias-aware evaluation and uncertainty-sensitive design.

Closest papers

Nearest neighbours by meaning across the whole index.

A Prompt-Engineering Approach to Develop Scalable, Flexible, and Real-Time Hybrid Micro-Level Personalization in a General Purpose AI Teaching Assistant

cs.AI ·2026-09-03 ·Saptarshi Basu, Sandeep Kakar, Ashok Goel
similarity 0.65
Abstract · 7 pages, 9 figures, IAAI27 conference

Artificial intelligence (AI) teaching assistants powered by large language models (LLMs) offer scalable educational support but often provide limited personalization. This study presents a prompt-engineering-based framework for personalizing general-purpose LLM/RAG-based AI teaching assistants such as Jill Watson across academic disciplines and courses. The framework adapts responses using six learner-specific dimensions: self-assessment, abstraction preference, verbosity preference, perceptual orientation, information processing style, and level of understanding, yielding 96 distinct learner profiles. Student queries are additionally analyzed using Bloom's Taxonomy to estimate cognitive complexity at the interaction level. Learner attributes and cognitive assessments are encoded in structured prompts that condition the LLM without requiring model retraining. The framework is evaluated through experiments using NLP metrics and a human study with five participants. Results show perceived differences in response style and structure across personalization conditions, with statistical analyses identifying learner attributes associated with measurable response changes. These findings provide preliminary evidence that prompt-based personalization can support adaptive behavior in LLM-powered educational agents.

0.16

How Many Examples, and Which Ones? A Study of In-Context Learning for Natural Language Inference

cs.CL cs.AI ·2026-10-13 ·Georgos Dikaios, Isidoros Perikos, Ioannis Hatzilygeroudis
benchmark similarity 0.64
Abstract

Natural language inference (NLI) is a fundamental task of natural language processing and crucial for evaluating the strengths and weaknesses of AI methods. Research has mostly focused on encoder or encoder-decoder architectures. However, the field of decoder-only Large Language Models (LLMs) remains underexplored. While prior work has mainly investigated complex sampling and prompting techniques, the effectiveness of a minimalistic prompting approach is yet to be examined in decoder-only LLMs. In this work, we examine and provide a comprehensive benchmark of LLMs (Llama, Mistral, Gemma, and Phi 4) in this domain, across the MNLI, QNLI and SciTail datasets. The performance of these models was evaluated using zero-shot and few-shot prompting strategies with simple and direct prompts. The results of the article indicate that LLMs generally perform exceptionally well when they are provided with a simple, straightforward prompt, challenging the notion that complex prompting techniques are always required for NLI tasks. Additionally, we find that the choice of demonstration label in one-shot prompting significantly affects performance for most models, with entailment-primed examples performing best. Our findings also show that the use of a few representative examples, in the case of few-shot, produced performance comparable or superior to prompts with higher example counts, although the efficacy of the prompting techniques is model and dataset-dependent. Moreover, our simple prompting approach consistently matched or outperformed more elaborate methods such as ICS and MCQ-based prompting.

0.23

Interaction Effects Between Learner Characteristics and Dialogue Format in LLM- and TTS-Generated Dialogue-Based Lessons

cs.CY ·2026-08-21 ·Fumie Watanabe, Gendo Kumoi, Tota Suko, Takashi Ishida, Yuko Kuma +2 more
similarity 0.64
Abstract · Revised version of a manuscript submitted to Education Sciences (MDPI); not yet peer-reviewed. The title has been changed from v1. 23 pages,…

Large language models and text-to-speech technology enable low-cost production of dialogue-based lessons: videos in which two speakers converse. The speakers' roles can vary, but little is known about whether all learners respond alike to such differences. In a repeated-measures design, 222 first-year high school students viewed lessons in three formats: a teacher and a student (TS), two students (SS), and two teachers (TT). Within the aptitude-treatment interaction framework, linear mixed-effects models tested whether associations between learner characteristics and outcomes differed by format. The learner characteristics were experiential learning style - seeking challenging, new experiences (CE factor) and learning by reflecting on experience (RCE factor) - and critical thinking disposition; the outcomes were motivation based on the ARCS model, self-reported learning outcomes, and overall evaluation. For motivation, both factors interacted with format: the CE factor's association differed between TS and TT, and the RCE factor's slope was significantly positive in TS and SS and significantly larger in TS than TT. For learning outcomes, no interaction was detected, but critical thinking disposition was positively associated with them. For overall evaluation, again no interaction was detected, but TT was rated lowest. Given small-to-medium interaction effects, these findings are preliminary evidence to inform personalized learning design.

0.28

EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues

cs.CL cs.CYcs.LG ·2026-09-22 ·Julian Bernado, Ana Trindade Ribeiro, Xander Beberman, Susanna Loeb
code/data similarity 0.64
Abstract

Large language models have enabled rapid deployment of language classification and analysis, allowing users to describe complex constructs in natural language, such as pedagogical moves or expressions of learning and engagement, and label them in conversational datasets such as classroom or tutoring transcripts. However, because LLM reasoning is opaque, we have no verifiable, mechanistic insight into why a model chose a label for an utterance, and output differences over time or across models challenge replication. We introduce the EduBehaviors framework to generate interpretable, reliable, and scalable construct annotations in educational dialogue data that retain human agency while leveraging the strengths of generative AI, encoders, and machine learning models. We evaluate the framework on the TalkMoves dataset, predicting the Teacher TalkMoves labels. Our best configuration results in a macro-F1 of 0.764 and 0.756 Cohen's kappa, demonstrating improvements over direct prompting approaches and showing competitiveness with fine-tuned classifiers while gaining the benefit of greater transparency and comparability. To facilitate adoption and operationalization of the EduBehaviors framework in educational dialogue data by other researchers, we also release the EduBehaviors Toolkit with two complementary open-source tools based on the TalkMoves application.

0.31

When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness

cs.CL cs.AI ·2026-08-17 ·Mahdi Dhaini, Adam Dejl, Juraj Vladika, Volkan Özer, Barbara Plank +1 more
similarity 0.64
Abstract

Natural language explanations (NLEs) are increasingly used as inputs, for example, as few-shot rationales that influence model behavior in in-context learning (ICL). However, it remains unclear how different types of NLEs compare in their effects on downstream model performance in explanation-augmented prompting. Therefore, we provide a comparative evaluation across six benchmarks and four instruction-tuned models, studying how NLE source (human-written when available, self-generated explanations, generated by an external LLM) and NLE selection (random vs faithfulness-based filtering) affect downstream utility of NLEs when used in ICL settings. Our extensive evaluation shows that, on classification-style benchmarks, adding NLEs to few-shot prompts often improves accuracy over few-shot prompting without explanations; among NLE sources, externally generated LLM-NLEs often provide strong downstream utility and remain competitive with human rationales where both are available, whereas self-NLEs are more sensitive to the selection strategy. On math reasoning, the effects are more model- and source-dependent. We further show that faithfulness-based selection of self-NLEs yields small average gains overall, but can improve or reduce performance depending on the metric, task, and model. Different faithfulness metrics can disagree substantially, affecting which self-NLE examples are selected and their downstream predictive utility. Robustness tests with randomly swapped and out-of-distribution rationales indicate partial robustness, suggesting that semantic alignment contributes to performance gains. Overall, our results provide insights for selecting and reporting explanations that influence model behavior in practical prompting pipelines.

0.19

Making AI-Generated Feedback Matter: A Large-Scale Study of Feedback Workflows and Student Enactment

cs.AI ·2026-08-12 ·Omar Alsaiari, Nilufar Baghaei, Jason M. Lodge, Dragan Gaševi'c, Naomi Winstone +1 more
similarity 0.63
Abstract

Feedback processes strongly influence student learning, yet their educational value depends on addressing two distinct challenges: providing high-quality, timely, and individualised feedback at scale, and supporting students to interpret, evaluate, and act on that feedback productively. Generative AI offers a credible means of addressing the provision challenge, but students' uptake of AI-generated feedback remains limited. We conducted a large-scale quasi-experimental sequential cohort study comparing three AI-mediated feedback workflows across 13,037 students and 51,296 student-authored resources. In Directed Feedback (n = 3,723), students received AI-generated feedback comments without structured support. In Self-Directed Feedback (n = 3,951), students could initiate optional AI-supported dialogue. In Enacted Feedback (n = 5,363), students were prompted to select feedback suggestions, evaluate their relevance, and engage in targeted AI-supported dialogue anchored to those selections. Enacted Feedback was associated with significantly higher uptake of AI-generated feedback, with an estimated probability of 26.2%, compared with 14.1% for Directed Feedback and 0.1% for Self-Directed Feedback. It was also associated with significantly higher self-assessment confidence and submitted-work quality than both comparison conditions. These findings suggest that the educational value of AI-generated feedback depends not only on the quality of feedback comments, but also on workflows that actively structure students' enactment of feedback literacy processes. The results have implications for the design of AI feedback systems that position learners as active participants in judgement, dialogue, and improvement rather than passive recipients of comments. Overall findings show that AI access alone is insufficient; purposeful workflow design is central to productive feedback use.

0.04

LLM Pedagogical Behavior in AI Tutoring Interactions

cs.CL cs.HC ·2026-08-24 ·Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin
top venue similarity 0.63
Abstract

Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.

0.46

CLUE: Uncovering Social Context for Pragmatic Understanding

cs.CL ·2026-10-15 ·Chengfei Wu, Zhaoqing Wu, Dan Goldwasser
similarity 0.63
Abstract

Understanding a conversation requires connecting what is said to the speakers' relationships, expectations, and earlier exchanges. Prior work shows that supplying relevant context can improve models' social judgments, but much of this information may already be present in the conversation. This motivates training models to discover the cues that matter and use them to support an interpretation. We propose CLUE, a two-stage training pipeline that combines supervised evidence reasoning with GRPO guided by Grounded Information Utilization ($\mathrm{IU}_{\mathrm{G}}$). This reward credits new, evidence-supported social inferences while accounting for reasoning effort. We train on a small set of conversational examples to assess effectiveness and generalization across tasks, languages, and genres. At inference time, the trained model reasons over the supplied interaction in a self-contained manner. Compared with base Qwen, CLUE achieves relative gains of 17.3% in-domain and 9.3% out-of-domain in average weighted F1, with higher F1 on four of five out-of-domain evaluations. These gains are accompanied by over 80% fewer reasoning tokens on average in both domains. Human evaluation further finds that CLUE more often grounds its explanations in specific conversational content, with closer alignment between its quoted evidence and human-selected context.

0.28