CS Paper Radar

Understanding Student Use of Large Language Models Across Computer Science Subfields

Sehrish Basir Nizamani, Yoonje Lee, Nikitha Donekal Chandrashekar, Margaret Ellis, Naren Ramakrishnan

cs.CY cs.HC ·submitted 2026-10-01 06:38 UTC · v1
published

Abstract

This research full paper examines how undergraduate students use large language models (LLMs) across computer science subfields. As LLMs become increasingly integrated into computing education, understanding how their use varies across technical and pedagogical contexts is essential for designing effective, subfield-aware instruction. This paper presents a cross-subfield analysis of LLM usage among 211 undergraduate students in a problem-solving course intentionally designed to support responsible and effective LLM use through structured instruction and reflection. Using post-assignment reflection data collected across seven instructional modules spanning multiple computer science subfields, we examine prompt counts, LLM role conceptualization, and verification behavior. Results show that LLM adoption varies substantially by assignment, with higher usage in algorithms and web development and lower usage in software engineering. Students predominantly treat LLMs as assistive tools rather than authoritative sources, and verification is common across all subfields, with most students using multiple strategies. Verification behavior also varies by assignment context, with testing more common in structured tasks and web search more common in open-ended tasks. These findings suggest that assignment characteristics play a central role in shaping how students interact with and evaluate LLM outputs, even under a single, consistently applied instructional design. This work contributes empirical evidence on how LLM adoption, role conceptualization, and verification behavior vary across computer science subfields, extending our prior work on structured, reflective LLM instruction to show how its effects differ by task rather than only in aggregate.

Comment
Accepted to the 2026 IEEE Frontiers in Education Conference (FIE 2026). 3 figures, 4 tables. (c) 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses

Closest papers

Nearest neighbours by meaning across the whole index.

Argus: Academic Integrity in the Era of Generative AI

cs.CY cs.AI ·2026-09-28 ·David Racovan, Ajay Rawat, Christopher K. May, Jeffrey A. Turkstra
positionhot direction similarity 0.70
Abstract · 7 pages, 7 figures

The rapid proliferation of large language models (LLMs) in the context of education has introduced significant challenges in enforcement of academic integrity, especially in programming courses. We present Argus, an automated detection system for LLM-assisted student work in undergraduate C programming assignments. Argus integrates behavioral and stylistic indicators to create a holistic picture of the student's progress through an assignment and surfaces anomalies that point to potential misuse of LLM assistance. We quantify and analyze data over six years of Spring semester offerings in a large-enrollment CS2 course at Purdue University using Argus, finding that 45% of enrolled students exhibited patterns consistent with LLM-assisted code development in Spring 2026. To contextualize these results, we analyze the relationship between flagged LLM use and student performance on written, in-person proctored examinations, and find a significant negative correlation. We also explore the problem of mitigating false positives, recognizing that erroneous accusations of academic integrity carry significant consequences for students and instructors alike, particularly in the context of large enrollment courses. We argue that any automated detection system must be accompanied by a structured process for human review. We discuss the consequences for future course design, changing academic policy as these tools become more ubiquitous, and the pedagogical implications of LLM-based tools in computer science education.

0.33

Self-Explanation Tutor for Active Study of CS1 Worked Examples

cs.CY cs.AIcs.HC ·2026-08-25 ·Arun-Balajiee Lekshmi-Narayanan, Mohammad Hassany, Kamil Akhuseyinoglu, Rully Hendrawan, Peter Brusilovsky
similarity 0.65
Abstract

Worked examples are an important part of introductory programming, but reading their expert explanations is passive. Self explanation, students explaining the problem and its solution to themselves with subgoal level analysis, converts passive reading into an active study of worked example, yet it is hard to scale because assessing free-text explanations and returning timely feedback has had no easy automated solution. We investigate whether a large language model (LLM) can fill that gap. We build a self-explanation tutor for introductory programming, ESSE, in which students explain lines of worked examples and receive immediate LLM feedback on the correctness and completeness of each explanation, and we pursue two goals. First, we ask whether the LLM judges student explanations well enough to serve as the engine of the tutor; we assess its judgments against two independent human reference standards of different kinds, a single domain expert and a crowd of non-expert raters, each with its own strengths and weaknesses, characterizing both where the LLM is reliable and the systematic tendencies in how it diverges. Second, we ask whether the LLM-based tutoring benefits students; deploying it in an introductory Java course, we find that its feedback leads students to persist and revise rather than abandon a line, that their explanations grow more complete and conceptually richer across attempts, and that students show evidence of learning. These indicate that LLM-based assessment is good enough to power a self-explanation tutor, and that the tutor positively shapes how students study worked examples.

0.10

Who (or what) are they prompting? How CS students conceptualize and use LLMs

cs.CY ·2026-11-10 ·Nikitha Donekal Chandrashekar, Margaret Ellis, Sehrish Basir Nizamani, Naren Ramakrishnan
published similarity 0.65
Abstract · Accepted to SIGCSE 2027

Large language models (LLMs) are increasingly used by students in computing courses. Yet computing education research has not fully examined how teaching and inclusion of LLMs shape students' conceptualization and understanding of LLMs, and how these factors relate to their subsequent use. Although prior work has examined prompting strategies and AI literacy, the relationships among students' conceptualizations of LLMs, their understanding of LLM mechanisms, and their prompting behavior remain underexplored. We studied these relationships in a post CS1 course that introduces students to the various CS subfields. We adapted the course to teach and allow the usage of LLMs. We administered an end-of-course survey, using open-ended items to elicit students' representations of LLMs and their explanations of how LLMs work, alongside closed-response measures of prompting style and prompt count (N=283 students consented and provided codable responses). Using qualitative coding, we categorized students' understanding of how LLMs work into four ordinal knowledge levels based on the Structure of Observed Learning Outcomes (SOLO) taxonomy and coded their metaphoric conceptualizations along two dimensions: persona type and relational role. Our results show that students most often conceptualized LLMs as human assistants, and context-rich prompting was the preferred style. The study contributes a two-dimensional characterization of students' LLM metaphoric representation and exploratory analyses also identified associations among these constructs showing how understanding these systems relates to their usage.

0.43

An Empirical Study on how Computing Students Interact with Large Language Models for Learning Purposes

cs.CY ·2026-09-27 ·Opetunde Ibitoye, Saheed Popoola
survey similarity 0.64
Abstract · 16 pages, single column, 3 figures, 2 tables

Large Language Models (LLM) have become popular tools in education. Therefore, understanding how students interact with LLMs for learning purposes is an important aspect of the responsible design and development of LLM based learning tools. However, existing research has focused on evaluating model performance, specific use cases, and students' attitudes toward LLM in education. There has been limited empirical work exploring students' interaction strategies with LLMs in computing education. This study draws on Self Regulated Learning (SRL) theory and conducts a survey of 225 undergraduate and graduate computing students to investigate how students use LLM tools during learning, the perceived learning impact, and the strategies they rely upon when using these tools. Students used LLMs across nine functional categories, most often to explain concepts, learn new material, find information, debug, and generate ideas. They also adopted some strategies such as supplying detailed context and iterative prompting. The most common difficulties encountered with LLM tools were incorrect information, generic responses, and difficulty trusting the model's reasoning. We synthesized these patterns through an SRL lens and offer a Probe Monitor Shape heuristic to map students' LLM interaction strategies onto the phases of self-regulated learning.

0.22

What the submission does not show: Complexifying cognitive offloading in students' mathematical work with LLMs

cs.CY math.HO ·2026-10-22 ·Igor' Kontorovich, Miriam N. Wallach
similarity 0.64
Abstract · 44 pages. Preprint under consideration at Digital Experiences in Mathematics Education; not peer reviewed

A common concern is that using large language models (LLMs) lets students offload mathematical work that tasks are designed to elicit. Yet such claims often rest on outcomes (e.g., submitted solutions), rather than on evidence of students' work with LLMs. We followed 23 first-year linear algebra students as they solved three differently designed homework tasks with LLMs. Drawing on self-regulated learning and choice-affluent environments, we interpreted each of 429 prompts as a trace of a regulatory event and compared 59 chatlogs with the problem solutions the students submitted. Prompts formed five structures, casting the LLM mainly as a solver, explainer, or validator. The mix of structures shifted across the tasks, and ten of the 15 students who completed all three tasks changed their dominant structure. Most final submissions stayed close to LLM output, including those that followed exchanges in which students had explained, conjectured, and checked their own ideas. We conclude that an LLM-aided submission is not a trustworthy marker of students' offloading of the mathematical work, and that offloading needs to be examined in each regulatory space and part by part rather than as a whole.

0.28

Generative AI and the Crisis of Higher Education Assessments: Examining 'AI-Proofing' Strategies for College-Level Assessments

cs.AI ·2026-11-16 ·Bonam Mingole, Davis Yadav, Hui Min Lee, S. Shyam Sundar, Amulya Yadav
similarity 0.63
Abstract

Higher education is confronting an academic-integrity challenge unlike any in its history. Students can now outsource homework, take-home assignments, and online examinations to large language models (LLMs) that complete them in seconds. Existing research on evaluating LLM performance on academic assessments has focused largely on standardized examinations, or on questions from college-level courses drawn from a single discipline. Far less is known about how LLMs perform on college-level assessment questions spanning multiple disciplines, that instructors have purposely designed to be AI-resistant, and about which features of such questions, if any, meaningfully constrain LLM performance. We address this gap by collecting 75 questions from authentic college coursework assessments, spanning 17 disciplines, which were contributed by 37 faculty members from ten U.S. universities and one U.K. university. Of these, 61 questions were intentionally designed by faculty members to be 'LLM-proof', while the remaining 14 questions were sourced from online courses. We then (i) evaluate three state-of-the-art LLMs' ability to correctly answer these questions with and without supplementary course materials; (ii) identify five strategies, through thematic analysis, that faculty used to make their questions harder for LLMs to solve; and (iii) test whether these strategies impact LLMs' ability to answer the questions accurately. Our experimental results show that the best-performing model scored approximately 83% without supplementary course materials and 85% with them, and we observed noteworthy differences across models, answer formats, and instructor tweaking strategies. We discuss implications for assessment design, academic-integrity policy, and the broader ethics of deploying LLMs in educational settings.

0.25

On-Premises Multi-Course RAG Tutoring for Business Education: Hardware-Software Trade-offs in a Campus AI Tutor

cs.AI cs.IR ·2026-10-01 ·Sidney Shapiro, Joshua Lindemann
similarity 0.63
Abstract · 23 pages, 3 figures, 4 tables

Campus AI tutors based on retrieval-augmented generation (RAG) must ground answers in assigned course materials while keeping textbooks and student dialogue on institutional infrastructure. We present CourseChat, an on-premises, multi-course RAG tutor for undergraduate business education, deployed behind a campus web gateway and intended for use embedded in Moodle. Six isolated course offerings, each keyed by its own course reference number (CRN), share twin-edge AI hosts running a FastAPI service, a local vector database, and a local large language model (LLM) served by Ollama. We report two generation-model bake-off rounds, a separate fixed-evidence source-fidelity comparison, and conversation and quiz audits. Several larger models failed the classroom speed gate, but a 12B model and a 7B alternative passed. A separate mixture-of-experts candidate improved some corrections while introducing new factual and continuity errors. We therefore retain the 8B production model pending a demonstrated overall improvement, rather than claiming that 8B is universally optimal. Software changes improved follow-up topic resolution while preserving course scope; 435 prebuilt questions across 65 modules decouple practice from live generation. The results support treating model choice, evidence selection, serving compatibility, and product design as a joint engineering decision. They do not establish learning gains: faculty ratings, peak-load capacity, and complete public-gateway acceptance remain separate evaluation needs.

0.16

Analyzing Code Provenance in LLMs for Programming Education

cs.AI cs.SE ·2026-11-06 ·Subhankar Maity, Faiza Belbachir, Omar Boucelma, Guilherme Medeiros Machado
similarity 0.62
Abstract

Large language models (LLMs) play an increasing role in programming education, challenging traditional assessment methods as they can generate fully functional code instantly. This disrupts the reliability of student submissions as indicators of learning and effort. To address this issue, code provenance is essential for tracing the origins of code and distinguishing student-authored, AI-generated, and externally sourced components. We extend the Min-K% probability approach by incorporating synthetic non-member code generation and apply it to Python code for membership inference and provenance tracking. Evaluations cover free LLMs such as GPT-2, DistilGPT-2, and DialoGPT, as well as state-of-the-art commercial models including GPT-5, Gemini 2.5, Mistral, DeepSeek, and LLaMA 3.3. Before evaluating our approach on higher-education student assignments, we validate its feasibility using Stack Overflow data. The results show that free models enable more sensitive code memorization detection than commercial models. By systematically comparing these LLMs, we assess trade-offs between detection accuracy and privacy protection, taking an initial step toward effective assessment of AI-assisted programming education through combined provenance detection and process visibility.

0.15