CS Paper Radar

Exploring the Role of Security Experience and ChatGPT Usage Strategies on Secure Software Engineering Education

Alessio Ferrari, Minh An Nguyen, Kushal Ramkumar, Liliana Pasquale

cs.SE ·submitted 2026-09-10 09:31 UTC · v1

Abstract

The rapid adoption of Large Language Models (LLMs) is reshaping software engineering education, but their role in secure software engineering education remains underexplored. We report an exploratory empirical study of how 26 graduate students in a part-time MSc Cybersecurity programme used ChatGPT during a vulnerability-fixing assignment. To characterise ChatGPT use, we analysed students' ChatGPT interaction logs using a structured double-coding procedure and examined whether usage patterns and prior cybersecurity expertise were associated with assignment performance. The results show that students with varying levels of cybersecurity expertise used broadly similar ChatGPT strategies. Individual usage patterns showed descriptive differences by grade, but none remained statistically significant after correcting for multiple comparisons. In contrast, diversity of ChatGPT usage, i.e., the number of distinct usage patterns adopted, was positively associated with performance, even after controlling for cybersecurity expertise. These exploratory findings suggest that the way students engage with ChatGPT may be more informative than whether they use it, and motivate future controlled studies to guide students toward effective LLM use in secure software engineering education.

Closest papers

Nearest neighbours by meaning across the whole index.

An Experience Report on Teaching Students to Evaluate AI Agent Skills as Authorization Decisions

cs.HC ·2026-11-11 ·Zikai Alex Wen, Donald Chinn, Yan Bai
similarity 0.64
Abstract · Preprint version. To appear in SIGCSE TS 2027

Cultivating a security mindset has long been a goal of computing education, and students now need it when deciding whether to trust an AI agent skill. Skills are add-ons that can shape an agent's behavior through natural language instructions and configuration settings, not just executable code. To prepare students to evaluate skills as authorization decisions, we designed a classroom activity and taught it in a senior undergraduate and a graduate cybersecurity-focused course. Students inspected three example skills, each pairing a real workflow task with one agent-specific authority, such as acting without asking for user consent before each action. The activity guided them to make initial authorization recommendations and then use a static security scanner to test and revise their recommendations. We observed in class that students needed guidance to extend their security review from a skill's code to what they would be authorizing the agent to do through the skill. Pre/post surveys from 11 volunteer respondents in this preliminary pilot showed greater self-reported confidence in evaluating skills, and post-activity responses placed more emphasis on the authority that skills carry. We release all activity materials for reuse.

0.33

On the Prospects of Dynamic LLM Conversations in Software Development

cs.SE cs.AI ·2026-08-31 ·Annemarie Wittig, Alina Mailach, Janet Siegmund, Norbert Siegmund
similarity 0.62
Abstract

Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a \emph{context}-aware group received intent-based conversation augmentation, (2) a \emph{proactive} group received follow-up suggestions and tailored advice, and (3) a \emph{control} group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the \emph{proactive} treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.

0.22

Investigating Developer-Reported Software Security Testing Challenges

cs.SE cs.CR ·2026-09-10 ·Md Erfan, Ahmed Ryan, Md Rayhanur Rahman
similarity 0.61
Abstract · 34 pages

Software security testing (SST) is essential for identifying vulnerabilities and improving software security, but developers often face practical challenges when selecting tools, configuring test environments, interpreting scanner outputs, testing authentication workflows, and acting on reported vulnerabilities. This study empirically characterizes developer-reported SST challenges in Stack Overflow (SO) discussions and examines their prevalence, difficulty, temporal evolution, co-occurrence, and taxonomy stability. We analyze 17,743 SO questions collected using SST-related keywords. Through manual labeling, we identify 582 SST-related questions and construct a taxonomy of 8 categories and 31 subcategories. We then analyze these challenges using prevalence, difficulty, correlation, temporal trend, co-occurrence, and held-out stability analyses. The results show that developers frequently discuss security finding interpretation and reliability, security testing guidance and tool selection, authentication and authorization testing, and security tool integration and automation. Validation-related challenges often require more technical context and longer resolution time. Security finding interpretation and remediation-related actionability show increasing trends over time. Co-occurrence analysis shows that false positives frequently appear with explainability, while integration challenges often appear with tool suggestions and documentation/resources. Overall, developer-reported SST challenges extend beyond vulnerability detection and include workflow, configuration, interpretation, and remediation concerns. These findings can help researchers, practitioners, educators, and tool providers improve SST usability, documentation, result interpretation, and remediation support.

0.25

Beyond the Hype: Evaluating LLM Integration and Practical Limitations in Security Operation Centers

cs.CR ·2026-08-17 ·Elnaz Rabieinejad, Ali Dehghantanha, Fattane Zarrinkalam, Sarina Dastgerdy
similarity 0.61
Abstract

Large Language Models (LLMs) are increasingly being explored within Security Operation Centers (SOCs) to support text-heavy analytical work such as alert contextualization, incident summarization, and drafting investigative artifacts. Despite this interest, practitioners describe critical operational concerns, most notably hallucinations (plausible but incorrect outputs), opaque reasoning, and the verification effort required to safely use model-generated content in security workflows. In this paper, we present findings from semi-structured interviews with 20 SOC practitioners spanning frontline analysts, SOC managers, and tool developers. Participants report perceived time savings for low-stakes tasks that are quickly verifiable (e.g., summarizing logs or drafting initial investigative leads), but they consistently frame LLM outputs as preliminary drafts and suggestions rather than decision-grade conclusions. Participants also describe limited trust in LLMs for high-stakes security decisions due to unreliable outputs and unclear model reasoning, and they report relying primarily on ad-hoc verification norms and continuous human oversight rather than standardized mitigation procedures. Based on these interview-grounded accounts, we introduce a maturity rubric to characterize readiness for LLM integration and outline a research agenda emphasizing auditability and transparent explanation mechanisms to support safer adoption in SOC workflows.

0.50

Advancing Inclusivity in Cybersecurity Education: Integrating Intersectionality to Enhance Student Engagement in Australian Higher Education Curriculums Strategies, Barriers, and Future Directions

cs.CY cs.CR ·2026-08-18 ·Nalin A. G. Arachchilage, Asangi Jayatilaka, Senuri Wijenayake, David Herbert, Kaie Maennel +4 more
unusual similarity 0.61
Abstract

Australian women, gender-diverse individuals, and culturally and linguistically diverse (CALD) communities are often more susceptible to phishing and other forms of cybercrimes due to factors such as language barriers, limited access to cybersecurity education, and social isolation. These communities encounter substantial obstacles both entering and progressing in the cybersecurity field. In Australia, the Higher Education sector still leans heavily on a largely uniform cybersecurity curriculum, focusing heavily on technical proficiency, overlooking the vital impact of intersectionality and user-centered thinking for boosting student engagement and learning. Without gender inclusivity and proper consideration of intersectionality forms such as CALD, the workforce is deprived of the varied perspectives necessary to tackle today's intricate cybersecurity issues. In this study, we conducted semi-structured interviews with 15 experienced academics teaching and coordinating cyber security programs from a diverse range of Australian universities, covering all states, to explore their perspectives on: i) current strategies for addressing the women, gender-diverse and CALD perspective in cyber security education in the Australian HE sector; ii) barriers to incorporate women, gender-diverse and CALD perspective in cybersecurity curriculums in higher education; iii) future work and support that is needed. Our research highlights a lack of systematic methods for integrating intersectional perspectives into cybersecurity curriculums. In particular, we identified four key barriers and four areas where support and future efforts are needed to address this issue. Our findings offer vital insights that can substantially guide curriculum development in cybersecurity education.

0.28

Understanding Student Use of Large Language Models Across Computer Science Subfields

cs.CY cs.HC ·2026-10-01 ·Sehrish Basir Nizamani, Yoonje Lee, Nikitha Donekal Chandrashekar, Margaret Ellis, Naren Ramakrishnan
published similarity 0.59
Abstract · Accepted to the 2026 IEEE Frontiers in Education Conference (FIE 2026). 3 figures, 4 tables. (c) 2026 IEEE. Personal use of this material is…

This research full paper examines how undergraduate students use large language models (LLMs) across computer science subfields. As LLMs become increasingly integrated into computing education, understanding how their use varies across technical and pedagogical contexts is essential for designing effective, subfield-aware instruction. This paper presents a cross-subfield analysis of LLM usage among 211 undergraduate students in a problem-solving course intentionally designed to support responsible and effective LLM use through structured instruction and reflection. Using post-assignment reflection data collected across seven instructional modules spanning multiple computer science subfields, we examine prompt counts, LLM role conceptualization, and verification behavior. Results show that LLM adoption varies substantially by assignment, with higher usage in algorithms and web development and lower usage in software engineering. Students predominantly treat LLMs as assistive tools rather than authoritative sources, and verification is common across all subfields, with most students using multiple strategies. Verification behavior also varies by assignment context, with testing more common in structured tasks and web search more common in open-ended tasks. These findings suggest that assignment characteristics play a central role in shaping how students interact with and evaluate LLM outputs, even under a single, consistently applied instructional design. This work contributes empirical evidence on how LLM adoption, role conceptualization, and verification behavior vary across computer science subfields, extending our prior work on structured, reflective LLM instruction to show how its effects differ by task rather than only in aggregate.

0.30

Cibersapiens: Gamifying Cybersecurity Education to Engage Secondary Students

cs.CY ·2026-11-03 ·Soraya Frías-Guadalupe, Pedro Paredes, Carlos Castaneda-Marroquín
published similarity 0.59
Abstract · 9 pages, 2 tables. Presented at JNIC 2025, Zaragoza, Spain, June 4-6, 2025

Cybersecurity education and awareness are often tailored for businesses, leaving general citizens, especially young people, unprepared. Despite the availability of educational resources, cybersecurity training is not provided equally to all individuals, particularly children and teenagers. As digital risks such as sexting, grooming, and cyberbullying, among others, increase, young users remain vulnerable. However, existing awareness efforts fail to engage them effectively and current content lacks the motivation needed to drive meaningful behavioral change. This paper introduces "Cibersapiens: Digital Well-being for a Connected Generation," an interactive, gamified environment designed to teach cybersecurity to secondary school students. The project focuses on developing and implementing gamified educational resources that actively engage students, improve their understanding of cyber risks, and promote best practices in digital security. By leveraging active learning methodologies, this approach transforms cybersecurity education into an engaging and meaningful experience.

0.25

Evaluating the Impact of Personalization in Conversational Cybersecurity Assistants

cs.CR ·2026-09-15 ·Lea Duesterwald, Anika Jain, Shreya Kochar, Norman Sadeh
similarity 0.58
Abstract · Accepted as a poster at the HAIPS Workshop at COLM 2026

Users increasingly turn to Large Language Models to answer a variety of questions, including cybersecurity questions. We study how personalization strategies can help improve the effectiveness of answers to questions asked to an LLM-based cybersecurity assistant. Beyond accuracy, we focus on the understandability, actionability and, most importantly, motivating power of answers, given how often users fail to follow cybersecurity recommendations. Specifically, we investigate four personalization strategies, ranging from static user profiles to interaction-history-based personalization, using a corpus of 1,045 real-world cybersecurity questions and a 7-day deployment involving 57 participants and 1,066 user questions. Across both a large-scale automated LLM-based evaluation and human evaluation, conversation-based personalization is consistently favored in comparative ratings of perceived helpfulness and likelihood of following security advice. Importantly, the relative trends observed in the LLM-based evaluation align with those obtained from human evaluation, suggesting that LLM-based evaluation can provide a scalable mechanism for comparing personalization strategies before costly user studies. These results indicate that behavior-driven personalization is a promising direction for LLM-powered cybersecurity assistants and highlight the value of combining LLM-based and human evaluation when studying personalized language-model systems.

0.21