CS Paper Radar

Measuring Student Self-Assessment against Viva-Demonstrated Mastery in a Large First-Year Programming Course

Sakshi Sharma, Pavani Ayinampudi, Aditya B. M. V., Jinal Gupta, Prakash Hegade, Rohit Sharma, Meenakshi V, S. R. S. Iyengar

cs.HC ·submitted 2026-09-30 04:19 UTC · v1

Abstract

Mastery-based education increasingly places the reporting of learning progress in students' hands, who record task completion on learning dashboards. The usefulness of such self-reports depends on how closely reported mastery corresponds to demonstrated competence. Most evidence on student self-assessment compares an overall self-rating with an overall examination score and therefore provides limited evidence about which tasks or which students account for the mismatch. This study examines first-year students' self-assessment against viva-demonstrated mastery at the level of individual tasks across a ladder of sixty programming tasks. The study draws on a large first-year programming course taught in 2023, involving 203 students and 12 examiners, in which every reported task was verified through an oral viva. Because a task entered the Viva only after it was reported, the design is one-sided and captures over-estimation but not under-estimation. Of 11,093 reported tasks, 10,885 (98.1%) were demonstrated, indicating a high degree of correspondence between self-report and demonstrated mastery. The remaining 208 overestimations were not evenly distributed. A small number of students accounted for most of the errors, and they occurred mainly on difficult tasks near the end of the task ladder rather than on higher-point tasks. This task-level analysis shows that high overall self-assessment accuracy can coexist with specific areas where reported and demonstrated mastery diverge. It also provides a practical basis for directing additional verification and formative feedback toward students and tasks where such divergence is more likely.

Comment
14 pages, 3 figures, 2 tables, Submitted in ICTIEE and under Review

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structured Assessment in Higher Education

cs.CY ·2026-10-07 ·Xi Zhao, Xinyue Jiao, Zhen Xu
similarity 0.60
Abstract

As Large Language Models (LLMs) are increasingly adopted for automated grading and feedback in higher education, their structured outputs, including multi-dimensional rubric scores, detailed feedback comments, and improvement suggestions, create an appearance of thorough analytic evaluation. This study examines whether these outputs deliver what they appear to offer. Using the JorGPT dataset of 3,041 student responses to 50 open-ended computer science questions, scored by both human instructors and three commercial LLMs, we identify three systematic discrepancies between the apparent and actual quality of LLM-generated grading and feedback. The sub-dimension scores are highly correlated (r = 0.82-0.99, VIF up to 45), providing redundant rather than independent diagnostic information. The textual feedback rarely detects student misconceptions (5-7% vs. 15-31% for teachers), functioning as a coverage checklist rather than a diagnostic instrument. The feedback tone remains uniformly positive regardless of response quality, lacking the severity modulation observed in human feedback. Additionally, grading accuracy varies significantly by knowledge domain, with procedural topics most reliable. These findings provide empirically grounded guidance on which aspects of LLM-generated grading and feedback can be relied upon and which require continued human oversight.

0.08

It's Complicated: Factors Influencing the Effectiveness of LLM-Generated Feedback in a Live Programming Course

cs.HC ·2026-10-18 ·Tsvetomila Mihaylova, Arto Hellas, Bita Akram, Narges Norouzi, Peter Brusilovsky +1 more
top venue similarity 0.58
Abstract

We report on three iterations of an LLM-generated feedback system deployed in a live web development course, spanning three underlying models and three feedback formats. We ask how feedback presence and its characteristics (length, complexity, format) relate to student performance, whether prior feedback exposure is associated with final-project performance, and how performance changes after first exposure. Feedback was associated with more favorable immediate outcomes in several comparisons, but the pattern varied by outcome and period. In pooled models, longer feedback was associated with lower next-submission correctness, more remaining attempts, and longer resubmission time. Prior same-part feedback was associated with lower final-project solve probability. Student-level comparisons found differences on final-project steps begun before exposure, while within-student changes across steps were uncertain. These comparisons do not identify the causal effect of feedback. Changes on standard exercises were more favorable among exposed students, but the groups' different course-stage splits limit causal interpretation. Bounded message length and student choice of format are directions for experimental evaluation. Our final-project analysis captures follow-up performance rather than directly measuring learning or understanding.

0.36

Outcome Over Learning? Evidence from Student Use of AI Code Assistants for Unit Test Writing

cs.SE cs.AI ·2026-11-10 ·Daniel Hinbest, Jeremy S. Bradbury, Michael A. Miljanovic
similarity 0.57
Abstract · 8 pages

Artificial intelligence (AI) code assistants are rapidly being adopted in programming education, yet little is known about how students naturally use these tools when given unrestricted access. In a user study investigating AI-assisted unit test writing, we observed that a majority of students adopted a one-shot interaction strategy, asking the AI to generate a complete unit test directly from the initial requirements. While this approach is capable of efficiently producing a solution, it shifts students' focus from learning how to write effective unit tests to simply completing the task. Moreover, because students rarely engaged in critically evaluating, refining, or iterating on AI-generated output, they may be missing opportunities to develop the critical thinking skills that are essential for effective and responsible use of AI. In an effort to shift student focus from outcome to learning we also discuss how training, accountability and improved learning task design can better support a more learning-centric responsible use of AI.

0.37

Your Programming Students' Cognition with ChatGPT: Higher Performance, Lower Retention, and Reduced Ownership

cs.CY ·2026-09-18 ·Christian Bergh, Benjamin Tag, Alexandra Vassar, Jake Renzella
hot direction similarity 0.56
Abstract · 17 pages, 9 figures, 7 tables

Generative AI can improve students' programming performance, but successful task completion may not reflect what they retain. We examined performance, retention, cognitive load, and ownership in a controlled between-subjects experiment with 59 undergraduate computer science students, 55 were retained for analysis. Participants completed three introductory C programming tasks with access to ChatGPT-4.5 or conventional web search without generative AI. We measured task performance, self-reported mental effort and difficulty, pupillary responses, heart rate variability, and ownership, and assessed cued recall immediately and 48 hours later. ChatGPT-assisted students achieved higher coding scores (89% vs. 69%) but lower recall scores immediately (41% vs. 53%) and after 48 hours (39% vs. 52%). There was no significant difference in the loss of recall information over 48 hours between the groups. Self-reported mental effort increased less across tasks in the ChatGPT condition (Holm-adjusted p = .047), and students attributed less of the submitted code to themselves (45% vs. 81%). Confirmatory physiological tests did not detect significant differences in trajectories between conditions; substantial data loss limits their interpretation. These findings reveal a gap between assisted task performance and subsequent recall and sense of ownership in this setting. They motivate the need for assessment practices and AI learning tools that require students to explain, retrieve, and contribute to the work they submit as active participants in their education.

0.56

Argus: Academic Integrity in the Era of Generative AI

cs.CY cs.AI ·2026-09-28 ·David Racovan, Ajay Rawat, Christopher K. May, Jeffrey A. Turkstra
positionhot direction similarity 0.56
Abstract · 7 pages, 7 figures

The rapid proliferation of large language models (LLMs) in the context of education has introduced significant challenges in enforcement of academic integrity, especially in programming courses. We present Argus, an automated detection system for LLM-assisted student work in undergraduate C programming assignments. Argus integrates behavioral and stylistic indicators to create a holistic picture of the student's progress through an assignment and surfaces anomalies that point to potential misuse of LLM assistance. We quantify and analyze data over six years of Spring semester offerings in a large-enrollment CS2 course at Purdue University using Argus, finding that 45% of enrolled students exhibited patterns consistent with LLM-assisted code development in Spring 2026. To contextualize these results, we analyze the relationship between flagged LLM use and student performance on written, in-person proctored examinations, and find a significant negative correlation. We also explore the problem of mitigating false positives, recognizing that erroneous accusations of academic integrity carry significant consequences for students and instructors alike, particularly in the context of large enrollment courses. We argue that any automated detection system must be accompanied by a structured process for human review. We discuss the consequences for future course design, changing academic policy as these tools become more ubiquitous, and the pedagogical implications of LLM-based tools in computer science education.

0.33

MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring

cs.CL ·2026-10-01 ·Woojin Lee, Yoonkyu Woo, Jinah Kim, Hyeok-Min Gwon, Jin-Xia Huang
publishedbenchmark similarity 0.55
Abstract · Accepted to the AACL-IJCNLP 2026 Main Conference. 20 pages, 1 figure, 16 tables

Large language models are increasingly used as math tutors, raising two questions: can a single LLM detect a student's error, diagnose the underlying misconception, and remediate it through multi-turn dialogue - and can we trust the numbers that report how well it does? We address both with MISCONCEPTDIALOGUE, a fully automated framework that runs the full pipeline on five open-weight LLM tutors (7B-14B) interacting with LLM-simulated students over 5,000+ dialogues - a deliberately simulated protocol (paired conditions, a wrong-diagnosis control unsafe with real learners, scale) whose results do not establish real-student learning. On measurement, our headline finding is uncomfortable for the field: benchmark-style exact-match scoring common in standard math NLP - a lower-bound stress test - reports 42.1% measured Remediation Success Rate on identical dialogues where a human-calibrated LLM-as-judge reports 81.3%, a 39.2 percentage-point spread larger than the combined effects of tutor model, teaching strategy, and supplied diagnostic information. This is not a single-judge artifact: four independent LLM judges and 200 human annotations converge to within a few points (kappa up to 0.91). On capability, the verdict is equally striking: the score barely changes whether we tell the tutor the student's actual misconception or a deliberately swapped one - under the tested diagnosis-presentation format, these tutors show little evidence of reliably conditioning on the diagnostic information they are handed. For LLM tutoring evaluation to mature, multi-criterion scoring with multi-judge convergence and human anchoring should become standard practice - and the measurement question deserves as much scrutiny as the capability question it claims to answer.

0.32

Scaling Peer Assessments: An Integrity Report from a Large Engineering Internship

cs.HC cs.CY ·2026-10-01 ·Jinal Gupta, Pavani Ayinampudi, Aditya B. M. V., Prakash Hegade, Rohit Sharma +3 more
survey similarity 0.55
Abstract · 12 pages, 3 figures, 5 tables; submitted to ICTIEE and under review

Assessing learning in large classrooms presents a significant challenge for individual instructors, who may have limited capacity to evaluate the understanding, participation, and assessment behaviour of every student. Peer assessments have been a way of distributing this responsibility among learners, allowing them to evaluate and provide feedback to one another while reducing dependence on instructor-led assessments. Building on this approach, we implemented a peer validation model within a large, multi-institutional internship programme in which students who demonstrated sufficient understanding were authorised to assess and validate their peers through short oral discussions. The assessment process began with the instructor validating a small group of students, who were then authorised to validate their peers, allowing the process to gradually expand across the cohort and operate at scale. This study examines how participants experienced the model and the extent to which assessment integrity was maintained, using an end-of-programme survey of 238 consenting respondents. Most participants regarded the activity as worthwhile, with 79.8% reporting that they solved problems they could not previously solve. However, 29.0% acknowledged at least one instance of reduced effort, a lowered validation standard, or reciprocal validation, while 88.7% believed that at least a little validation had occurred without proper examination. When asked how the process could be strengthened, participants selected post-validation discussion of solutions approximately twice as often as closer auditing or mentor-led validation. These findings provide descriptive evidence of both the potential and the integrity challenges of using peer validation as a scalable assessment approach in large learning environments.

0.29

ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry

cs.AI ·2026-10-08 ·Yafeng Tang, Hao Li, Hongsheng Yu, Qiang Fu
similarity 0.55
Abstract

Self-distillation can improve reasoning without a separately trained, more capable teacher, but its effectiveness depends on how the self-teacher gains an advantage over the student. Conditioning the teacher on reference answers or solutions can provide such an advantage, but this information may be unavailable. Reflection offers a way to derive explicit error diagnoses and revision guidance from self-generated attempts, yet existing reflection-based methods often combine it with reference information, rich task feedback, or persistent memory. We introduce ReTeach, a Reflective self-distillation framework that constructs its self-Teacher through multi-round reflection and retry using only self-generated attempts and outcome-level verification. Starting from an unsuccessful student rollout, the teacher alternates explicit reflection with renewed attempts until success or the retry budget is exhausted, without reference answers or solutions, external diagnostic feedback, or cross-example memory. Each failed retry informs subsequent reflection, while successful correction provides outcome-level evidence for the potential utility of the resulting teacher context. An outcome-aware selection and weighting strategy distinguishes initially correct, reflection-corrected, and unresolved examples, assigning separate weights to their category-normalized distillation losses. Through on-policy distillation, the student matches the teacher's context-conditioned token-level predictive distributions at prefixes of its own rollouts, transferring the benefits of iterative correction while retaining single-pass inference. Across six benchmarks spanning mathematical reasoning, science question answering, and tool use, ReTeach improves average accuracy over GRPO by 1.39 percentage points.

0.22