CS Paper Radar

ExplainRoute: A Pre-Deployment Audit Framework for Non-Answer-Giving Programming Tutors

Yiming Gai, Yingying Zhang, Xuefei Huang

cs.IR ·submitted 2026-09-03 07:28 UTC · v1

Abstract

Programming tutors should support learners' own explanations rather than immediately providing model answers. We present ExplainRoute, a pre-deployment audit framework for non-answer-giving programming tutors. Given a code line and a learner explanation, it estimates the explanation state and selects one of two bounded responses: a Feynman-style self-explanation prompt or a Socratic scaffold. The framework exposes its state, strategy, cited code fragment, and leakage risk through a machine-checkable contract. Unlike benchmarks that rank tutors by fluency alone, ExplainRoute audits information boundaries, response polarity, failure closure, and the value of learner-explanation visibility before classroom deployment. We evaluate it offline on the 1,770-pair SelfCode corpus using a code-group split, with 443 pairs reserved in 11 untouched holdout groups. The evaluation compares direct answers, fixed open self-explanation, fixed Socratic scaffolding, adaptive routing, and an adaptive no-state ablation. Contract validity reaches 100% for all pedagogical conditions. Adaptive routing matches the frozen reference rule on 60.5% of records, with state macro-F1 of 0.238 (Open: 0.229; Socratic: 0.246), showing no reliable adaptive advantage. An independent language-model judge scores adaptive responses 4.516/5, outperforming the no-state ablation (2.819/5) but slightly below fixed open self-explanation (4.598/5) and Socratic scaffolding (4.658/5). A blinded rubric evaluation on a stratified 40-row subset confirms that visible learner explanations improve information value while adaptive routing does not outperform fixed strategies. The contribution is a validated audit protocol and a boundary finding, rather than evidence of improved learning, retention, or causal instructional effectiveness.

Closest papers

Nearest neighbours by meaning across the whole index.

Self-Explanation Tutor for Active Study of CS1 Worked Examples

cs.CY cs.AIcs.HC ·2026-08-25 ·Arun-Balajiee Lekshmi-Narayanan, Mohammad Hassany, Kamil Akhuseyinoglu, Rully Hendrawan, Peter Brusilovsky
similarity 0.71
Abstract

Worked examples are an important part of introductory programming, but reading their expert explanations is passive. Self explanation, students explaining the problem and its solution to themselves with subgoal level analysis, converts passive reading into an active study of worked example, yet it is hard to scale because assessing free-text explanations and returning timely feedback has had no easy automated solution. We investigate whether a large language model (LLM) can fill that gap. We build a self-explanation tutor for introductory programming, ESSE, in which students explain lines of worked examples and receive immediate LLM feedback on the correctness and completeness of each explanation, and we pursue two goals. First, we ask whether the LLM judges student explanations well enough to serve as the engine of the tutor; we assess its judgments against two independent human reference standards of different kinds, a single domain expert and a crowd of non-expert raters, each with its own strengths and weaknesses, characterizing both where the LLM is reliable and the systematic tendencies in how it diverges. Second, we ask whether the LLM-based tutoring benefits students; deploying it in an introductory Java course, we find that its feedback leads students to persist and revise rather than abandon a line, that their explanations grow more complete and conceptually richer across attempts, and that students show evidence of learning. These indicate that LLM-based assessment is good enough to power a self-explanation tutor, and that the tutor positively shapes how students study worked examples.

0.10

Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior

cs.CY ·2026-08-12 ·Yusuf Pisan
similarity 0.69
Abstract

An effective large language model (LLM) tutor must often decline to give an answer it could easily produce. In a randomized study, students who used an unguarded chatbot scored higher while practicing but lower on a later test taken without it, whereas a Socratically guarded version of the same model kept the practice gain and removed the later loss [4]. Reliable answer-withholding is therefore central to a tutor's value, yet a capable model pressed by a frustrated student does not withhold reliably on a prompt alone. We report a deployed tutoring system that enforces answer-withholding as a per-turn, machine-checkable contract, and a method for tuning that withholding against evidence. A non-LLM policy core, reading only trusted learner state, sets a per-turn ceiling on an eight-rung help ladder; a deterministic detector strips solution code; and a separate LLM judge checks each risky reply against the contract. We tune the behavior with an automated evaluation that uses no human subjects: scripted student personas are driven through the live pipeline and re-scored by a stronger model, and we record each rejection's stated reason so failures are fixed by cause. Doing so revealed an interpretable "over-help ladder," from blatant solution leaks, to naming the exact bug, to over-citing general facts, with each fix exposing the next. The tutor reached full compliance on all four acceptance criteria. We offer the measure, diagnose, and fix loop as a reusable recipe for any LLM agent that must refuse a capability it has.

0.42

Turning Errors into Successes: Designing LLM Tutoring Around Scaffolded Error Correction

cs.HC ·2026-11-15 ·Michael W. Asher, Meng Cao, Paulo F. Carvalho
code/data similarity 0.65
Abstract · Includes main text (40 pages, 8 figures, 2 tables) and supplementary material (12 pages, 5 tables)

Large language models (LLMs) could make personalized tutoring available at scale, but it remains unclear how they can best support the instructional processes that make one-on-one instruction effective. We compared two ways of using LLMs after students make errors: providing personalized explanations of those errors, or engaging students in additional cycles of scaffolded practice and feedback, with a Socratic tutor guiding students toward the correct answer. In a preregistered online experiment, 400 adults practiced setting up constraint equations and, after each attempt, received the correct answer, a generic explanation, a personalized LLM explanation of their error, or Socratic tutoring. Socratic tutoring helped: participants resolved 64% of their errors through the dialogue and scored 0.24 SD higher on practice-aligned posttest questions than those who received generic or personalized explanations. Socratic participants also found the session more interesting, less frustrating, and less costly, even though it took nearly twice as long. In contrast, we found no evidence of a benefit of personalized explanations over generic ones, and neither type of explanation reliably outperformed simply seeing the correct answer. These findings suggest that LLM tutors may add the most value not by explaining errors more precisely but by helping students correct those errors themselves through scaffolded practice. This pattern fits a mastery-experience account of tutoring, in which an effective tutor notices when a student is struggling, steps in with guidance, and works with the student until the struggle resolves into success. AI tutors may be most effective when designed around this principle.

0.24

Correctness, Convergence, and AI-Generated Code Detection: A Longitudinal Study of Student and Large Language Model Code in Introductory Programming

cs.CY cs.SE ·2026-10-01 ·Runlong Ye, Jing Fan, Angela Zavaleta Bernuy, Oscar Karnalim, Paul Denny +2 more
top venue similarity 0.65
Abstract · accepted at Koli Calling '26

Large language models can generate plausible solutions to programming assignments, making it tempting to detect their use by matching student code against a reference bank of generated solutions. Yet similar code can also arise when an assignment admits only a few natural implementations, which leaves open what a match actually shows. We investigate generated-reference matching using 29,970 student submissions from ten Python labs offered in 2021, 2023, and 2025, together with 90,000 solution attempts generated retrospectively by three frontier LLMs. We validate the generated solutions using hidden instructor tests, compare code with MOSS after excluding the starter code, and examine the exact abstract syntax tree (AST) forms of selected functions. The models usually produced correct solutions and, across most assignments, converged on similar implementations. Student submissions matched the generated references more often in later cohorts, including among submissions that passed every hidden test. On tightly specified functions, the models converged on a few exact abstract-syntax-tree forms, and the number of distinct student forms also declined across cohorts, whereas open-ended functions remained diverse in both sources. Most reported overlaps were short, making the minimum match length an important choice when reviewing students' code. Finally, we discuss how instructors can build a reference bank of generated solutions before releasing an assignment to identify tasks on which generated solutions converge, decide how much review a match warrants, and redesign tasks to elicit tests, reasoning, and intermediate work. These findings support tracking population-level changes in submitted code, while attributing AI use to an individual submission would require additional evidence about how it was produced, such as prompts, revisions, intermediate code, and student disclosures.

0.38

SocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception Repair

cs.AI cs.LG ·2026-10-10 ·Mykhailo Kopytskyi, Jianhua Yang, Mir Seyedebrahimi
similarity 0.63
Abstract

Large language models are increasingly used in computer science education. However, the teaching abilities of even the most powerful LLMs are limited: they tend to reveal the solution prematurely and may lose focus on the learner's current debugging objective. To address this, we introduce SocraticRepair, a planning-guided Socratic tutoring framework for program debugging and miscon- ception repair that adapts prior planning-guided tutoring work (Li et al., 2026b). The framework is instantiated in the Python programming domain by fine-tuning Qwen2.5-Coder-7B-Instruct on synthetic multi-turn tutoring dialogues constructed from four preprocessed programming datasets. The system is evaluated using a predefined set of independent metrics and benchmarked against a general-purpose LLM. The experiments show that SocraticRepair achieves higher scores on stepwise scaffolding and question quality while also substantially reducing premature solution disclosure. The results also identify reliable pedagogical plan generation as an important remaining challenge.

0.15

Making Small Code LLMs More Efficient for Secure Code Review with Neuro-Symbolic Fine-Tuning

cs.SE ·2026-10-26 ·Mostafijur Rahman Akhond, Mahmuda Khatun, Gias Uddin
similarity 0.62
Abstract

As AI coding assistants generate more code, the need to review it for security flaws grows. Existing approaches leave a gap: static analyzers identify program sites but offer little explanation, while language-model explanations can be difficult to verify. Small open-weight code models could make security review available without sending proprietary code to a hosted service. We study whether fine-tuning helps these models turn security evidence into explanations that developers and tools can check. We present secSemScan, which pairs a static analyzer with a locally deployable model that writes a schema-constrained security monologue naming program sites, assigning a CWE, and giving a machine-parseable verdict. We also introduce OGSM (Oracle-Grounded Security Monologue), a dataset of 1,993 C/C\texttt{++} functions covering 22 CWEs in the taint and lifetime families. Each record pairs a function with analyzer-derived or fix-diff-derived evidence and a monologue. We fine-tune three open-weight code models in two stages, first on the output format and then on the monologue content. Because evaluation prompts supply the verdict and evidence, our results measure how well models use that information, not whether they detect vulnerabilities independently. On 355 test records, fine-tuning raised Qwen2.5-Coder-7B's verdict accuracy from 4.8\% to 96.6\%; DeepSeek-Coder-6.7B reached 78.9\%, and StarCoder2-7B reached 43.7\%. Qwen2.5-Coder-7B cited 87.5\% of Tier~1 oracle sites, and every line it cited belonged to the oracle evidence. The results suggest that fine-tuning helps small models produce more structured, evidence-linked security reasoning.

0.14

Code as Explanations: Executable and Verifiable LLM Explanations

cs.LG cs.AIcs.CLstat.ML ·2026-11-10 ·Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath
top venuecode/data similarity 0.62
Abstract · 30 pages, 11 figures, 3 tables, The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS 2026)

As large language models (LLMs) increasingly make consequential decisions and solve problems alongside humans, we must understand how models arrive at their conclusions. Understanding how an LLM behaves requires observing it across multiple inputs, yet free-text explanations such as chain-of-thought (CoT) are generated per instance and cannot be easily verified on new inputs. We propose {code explanations to address the mismatch in how LLM explanations are generated versus how they are evaluated: a program explains LLM behavior across a dataset of inputs, and its execution enables automatic verification. We define Consistency metric that measures the fraction of inputs on which the program reproduces the LLM's outputs, and use it to evaluate and select candidate programs. Across three open-source code-capable LLMs and eight tasks, we find that LLMs can produce highly consistent code explanations in several settings. Code explanations are also 6$\times$ to 100$\times$ shorter than CoT, and better align with model behavior in 9 out of 24 settings. However, consistency is significantly lower on tasks with more complex linguistic components, where generated programs reproduce model outputs no more frequently than an uninformative explanation baseline, and these programs often rely on longer hard-coded patterns. In experiments with Claude Opus 5.5, providing the model with consistency feedback and code execution improves explanation consistency across tasks, but consistency remains significantly lower on more complex tasks. Our findings establish executable code as a novel, verifiable, and compact approach to explaining LLM behavior and highlight verification feedback as a promising direction for improving explanation consistency. The code for this paper is at https://github.com/rajesh-lab/code-as-explanations.

0.55

LLM Pedagogical Behavior in AI Tutoring Interactions

cs.CL cs.HC ·2026-08-24 ·Suhyeon Lee, Juneha Baek, Jaehyeong Park, Donghyuk Shin
top venue similarity 0.62
Abstract

Students increasingly use LLMs as tutors for coursework and problem solving. Little is known about the level of assistance LLMs provide when students use them as tutors in authentic learning interactions. This matters because tutoring responses can differ substantially in how directly they help students complete a task. We operationalize this dimension as scaffolding level and develop a five-level scale, validated against human annotations, that characterizes responses according to the degree of direct assistance they provide. We apply the scale to 14,637 LLM responses from 203 students in a university AI course. Responses are overwhelmingly concentrated at high levels of assistance, with more than 95% classified as either Explaining or Solving. Scaffolding level is systematically associated with students' subsequent conversational behavior, but provides little additional predictive information about performance on three subsequent exams beyond prior achievement and dialogue behavior. These findings provide an empirical baseline for LLM assistance in tutoring interactions and a measurement framework for evaluating how alternative tutoring designs change that assistance.

0.46