CS Paper Radar

What Do Digital Traces Predict? Task Effects, Student Differences, and Critical Online Reasoning

Maxim Konca, Alexander Mehler

cs.CL cs.CYcs.LG ·submitted 2026-11-16 09:49 UTC · v1

Abstract

Digital assessment traces are increasingly used to predict student performance, often evaluated by correlating predicted and observed scores pooled across tasks. However, such correlations may reflect between-task differences rather than performance differences among students within tasks. To investigate this, we analyzed up to 527 browsing sessions from 424 university students completing open-Internet Critical Online Reasoning tasks and assessed the cross-task stability of their ratings. We evaluated the same out-of-fold predictions within and across tasks, compared three validation tiers up to leave-one-task-out, and tested incremental predictive contributions of consultation order and relational structure against matched permutation and randomization controls. None of the twelve prespecified primary tests was significant: on held-out tasks, consulted-page features showed no significant association with relative student performance, nor did observed order or relational structure add predictive value. Pooling observations across tasks substantially increased correlations, particularly for factual correctness, whose scores showed limited evidence of stable between-student differences across tasks. However, sentence embeddings yielded modest positive correlations with generic response quality on held-out tasks, without establishing added predictive value from observed order or relational structure. Our findings show that pooled predictive correlations may not reflect within-task student differences, highlighting the need for within-task evaluation and criterion generalizability audits.

Closest papers

Nearest neighbours by meaning across the whole index.

Rubric-Based AES with LLM Ensembles: Scoring of Student Responses in Critical Online Reasoning Using Ensembles of LLMs

cs.CL ·2026-11-16 ·Daniel Baumartz, Alexander Mehler, Patrick Schrottenbacher, Leon Hammerla, Sebastian Gombert
similarity 0.62
Abstract

Grading student responses is costly and time-consuming. Moreover, human annotators are prone to bias, fatigue, and rater drift, which further limits grading reliability. Various methods have been employed to develop automated essay scoring (AES) systems to reduce these costs and address bias-related issues. These methods range from early text-feature-based systems to current machine-learning and transformer-based models. Specifically, large language models (LLMs) offer the potential to streamline the development of AES systems. However, they are subject to well-known limitations, such as hallucinations and prompt sensitivity. This creates a need to explore the potential of a mixed-methods approach that leverages the synergies among individual LLMs through ensemble formation. We evaluate the potential of ensembles of LLMs to match human-rater agreement when grading student responses in economics and medical education across 103 criteria. We use 631 German-language responses and corresponding human ratings by 2 to 5 annotators from a previously conducted assessment. We experiment with 523 rubric-based prompt configurations across 15 open-weight LLMs and assess their consistency with human raters using Krippendorff's alpha for inter-annotator agreement (IAA) and the Alternative Annotator Test (alt-test). Our results show that replacing one human rater with the best-performing LLM--prompt configuration maintains or improves IAA for 91 of the 103 criteria and that LLMs pass the alt-test for 90 criteria. Our findings suggest that current state-of-the-art LLMs can support AES of German responses in economics and medical education, reducing human workload and enabling faster processing, while validation against human ratings remains necessary.

0.13

ClickSight: An LLM Framework for Interpreting Student Clickstreams Through Learning Strategies

cs.AI cs.CLcs.CY ·2026-11-01 ·Bahar Radmehr, Lan Tran, Ekaterina Shved, Fatma Bet{ü}l G{ü}reş, Adish Singla +1 more
code/data similarity 0.61
Abstract · 29 pages, 12 figures, 19 tables. Under review. Code: https://github.com/epfl-ml4ed/ClickSight-CEAI

Digital learning environments produce rich clickstream data that can reveal how students approach learning tasks, yet interpreting these fine-grained traces often requires substantial expert effort. We introduce ClickSight, a framework for interpreting student clickstreams through learning strategies defined in the literature. Given a student clickstream and a predefined set of learning strategies, ClickSight uses a staged LLM-based pipeline to identify strategy-relevant evidence, assess adherence to each strategy, and generate an evidence-grounded interpretation of the student's behavior. We evaluate ClickSight across two learning environments, 365 students, and 12 model backbones using an expert-grounded quality rubric and an LLM judge calibrated with human-expert ratings. Our results show that ClickSight can generate high-quality strategy-based interpretations across different models and environments, with reasoning models performing more consistently than instruction-tuned counterparts. We further find that providing external clickstream-extracted guidance can improve interpretation quality for strategies that require precise quantitative information from the trace. By connecting fine-grained clickstreams with theoretically defined learning strategies and preserving the evidence underlying each judgment, ClickSight provides a flexible approach for studying student behavior in digital learning environments.

0.40

VIRA-Bench: Diagnosing Pedagogical Contingency Understanding in Online Tutoring Videos

cs.HC ·2026-10-21 ·Weihang Wang, Hansu Gu, Guanming Liu, Peng Zhang, Ning Gu +1 more
top venuebenchmark similarity 0.61
Abstract · Accepted to EMNLP 2026 Main Conference (Oral). 21 pages

Understanding a one-to-one tutoring video requires more than recognizing its events. A model must connect a visible teacher intervention to the learner response it elicits and to the follow-up adjusted to that response. Existing video and educational-dialogue benchmarks typically emphasize event recognition, temporal reasoning, social-state inference, or transcript-based interaction, but rarely diagnose this action-response-adjustment relation. We introduce VIRA-Bench, a focused diagnostic benchmark for short-horizon pedagogical contingency in one-to-one online tutoring. VIRA-Bench contains 1,005 tutoring clips and 1,322 multiple-choice questions with researcher evidence audit and expert verification, organized into Observable Evidence Grounding (OEG), Interaction Contingency Reasoning (ICR), and Support-Response Modeling (SRM). Across 11 models, the best reaches 61.3%, well below the 91.0% human reference on a stratified subset, and ICR generally has the lowest layer accuracy among leading models. Under controlled input manipulations, removing video most strongly degrades OEG, whereas disrupting temporal order most strongly degrades ICR. Free-response, Stem+Options, and synchronized-shuffle controls show that response format, static question cues, and video-ASR mismatch do not explain these layer-specific patterns. The results identify distinct behavioral evidence dependencies without claiming an internal architectural mechanism. VIRA-Bench provides a focused evaluation resource for multimodal dialogue grounding, pedagogical discourse, and temporal reasoning in online tutoring.

0.35

Beyond the Answer: Student-Conditioned Trace Construction for Reasoning Distillation

cs.CL cs.LG ·2026-10-16 ·Shaoang Li, Yanhang Shi, Yufei Li, Mingfu Liang, Frank Shyu +4 more
benchmark similarity 0.60
Abstract

A reasoning trace can accompany a correct answer without providing effective supervision for a particular student. We study how to construct traces that improve learning beyond answer-only supervision by considering both their intermediate content and their accessibility to the target learner. We introduce ReDiST, a student-conditioned trace-construction and distillation framework for evidence-rich reasoning. Starting from the student's own attempts, ReDiST revises answer-correct drafts against the supplied evidence and repairs unsuccessful drafts with additional guidance from the verified answer. A frozen student-matched reader screens and ranks original and edited candidates using answer recovery and verified-answer probability, given the question, evidence, and candidate trace. We distill the selected targets while balancing trace and final-answer supervision with an auxiliary answer loss. Construction and selection occur offline, and inference uses only the distilled student. Experiments on question-answering benchmarks show that ReDiST can improve both source-task and cross-benchmark performance over answer-only, original-trace, and the evaluated trace-selection and rewriting baselines. Across supervision recipes, performance rankings can differ between source and transfer benchmarks, highlighting the importance of evaluating trace teaching utility across distributions.

0.21

Edhivo: An AI-Native Platform for Scalable Assessment and Instant Feedback in Higher Education

cs.CY ·2026-10-27 ·Moraish Kapoor, Jiawei Zhou
similarity 0.59
Abstract

Scaling assessment in higher education faces two intertwined bottlenecks: the cognitive burden of authoring assessments and the inability to semantically evaluate open-ended responses, which together create a persistent feedback gap that hinders student learning outcomes. To address these challenges, we present Edhivo, an AI-native pedagogical platform that addresses both bottlenecks through a vertically integrated system for content delivery, assessment, and reporting. Edhivo employs a Dual-LLM pipeline: one model generates context-aware assessments for both multiple-choice and open-ended questions, from instructor-provided materials, while a second model performs multi-criteria, rubric-based evaluation of student responses, enabling near-instant qualitative feedback at multiple hierarchical levels. All AI interactions are source-grounded to reduce hallucination risk, and a Human-in-the-Loop review process preserves instructor authority. We demonstrate Edhivo's efficacy through a pilot deployment at Stony Brook University in a real classroom environment in 2026 spring, where it supports high-frequency assessment creation and evaluation for a cohort of 36 students throughout a full semester, significantly reducing authoring effort and feedback latency while enabling novel pedagogical designs centered on active learning.

0.17

Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis

cs.HC cs.AIcs.CL ·2026-08-16 ·Alona Strugatski, Licol Zeinfeld, Giora Alexandron
similarity 0.58
Abstract

The rapid development and growing deployment of large language models (LLMs) have made it increasingly important to understand their capabilities. A common approach is to evaluate LLMs using assessment instruments originally designed to measure skills and competencies in humans, such as standardized exams, and to use performance on these instruments as evidence for generalizable claims about LLMs' underlying abilities on the same skills the assessments are intended to measure in humans. However, from a validity perspective, such inferences require that the relationship between observed performance and underlying constructs established for humans also holds for LLMs. In particular, a necessary condition for transferring score interpretations is similarity in the latent structure of responses to the assessment. In this study, we examine whether this condition holds in two educational contexts: high-school chemistry and a quantitative reasoning section of a university entrance exam. Using a case study design, we compare human response data with responses generated by six multimodal LLMs. Our analytical approach combines exploratory factor analysis, factor congruence, and resampling to assess latent structure similarity across human learners and LLMs. Across both instruments, we find systematic differences between human and LLM factor structures, showing evidence that the analyzed assessments may not measure the same constructs for humans and LLMs. These findings call into question the validity of evaluation practices that use educational assessments to make claims about AI capabilities.

0.38

Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structured Assessment in Higher Education

cs.CY ·2026-10-07 ·Xi Zhao, Xinyue Jiao, Zhen Xu
similarity 0.58
Abstract

As Large Language Models (LLMs) are increasingly adopted for automated grading and feedback in higher education, their structured outputs, including multi-dimensional rubric scores, detailed feedback comments, and improvement suggestions, create an appearance of thorough analytic evaluation. This study examines whether these outputs deliver what they appear to offer. Using the JorGPT dataset of 3,041 student responses to 50 open-ended computer science questions, scored by both human instructors and three commercial LLMs, we identify three systematic discrepancies between the apparent and actual quality of LLM-generated grading and feedback. The sub-dimension scores are highly correlated (r = 0.82-0.99, VIF up to 45), providing redundant rather than independent diagnostic information. The textual feedback rarely detects student misconceptions (5-7% vs. 15-31% for teachers), functioning as a coverage checklist rather than a diagnostic instrument. The feedback tone remains uniformly positive regardless of response quality, lacking the severity modulation observed in human feedback. Additionally, grading accuracy varies significantly by knowledge domain, with procedural topics most reliable. These findings provide empirically grounded guidance on which aspects of LLM-generated grading and feedback can be relied upon and which require continued human oversight.

0.08

From Source-Based Quizzes to Reteaching Decisions: A Practitioner Guide to ClassQuest

cs.CY ·2026-11-02 ·Zihao Sun
similarity 0.58
Abstract · 8 pages. Author preprint; submitted to ACE 2027, decision pending

A source-based quiz can help a computing teacher identify material to explain again, but its report must preserve the limits of the evidence behind that decision. We present ClassQuest, a prototype connecting teacher-reviewed question drafts, private student responses, topic reports and targeted retries. A Year 10 classroom account of an earlier prototype supplies 20 usefulness ratings from 24 invitees, with a mean of 8.0 and median of 9 on a ten-point scale. A Boolean-logic episode documents how immediate feedback, a teacher explanation and a fresh question were combined. We separate this account from a completed technical audit and reusable teaching guide. An exhaustive single-item audit found no count mismatches across 5,456 constructed response configurations, while diagnostic cases exposed a retry increase caused entirely by changing participation and a progression suggestion based on one learner's three answers. A further case shows that an unoffered retry can be labelled missing. A separate reporting companion distinguishes opportunity to respond, pending review and matched retry cohorts. Its completion-sensitivity check matched an independent oracle in 46,375 constructed states across roster sizes 1-30, identifying when unknown responses can change a reporting band. Our contributions are a source-linked formative workflow, executable examples of interpretation failures, and reporting and teaching materials for examining those failures. Neither the questionnaire nor the audit establishes improved learning; the findings identify reporting requirements and practical directions for formative teaching.

0.13