CS Paper Radar

Vibe Coding and Web Application Security: A Twin-Prompt Study

Darko Andročec

cs.CR cs.AI ·submitted 2026-08-21 10:34 UTC · v1
published

Abstract

Large language models increasingly generate complete web applications from natural-language prompts, raising the question of whether explicitly requesting security best practice improves the result. We study six functionally distinct web applications, each generated in two prompt variants that are identical except for an appended security-requirements section: a baseline (A) and a security-aware (B) variant. All twelve programs were produced by the same agentic coding assistant and the same model version in a single, non-iterative generation round, and were then analyzed with static, dependency, dynamic and manual techniques, yielding 75 confirmed findings out of 85 candidates. The security-aware variant produced fewer confirmed findings in every application (24 versus 51) and contained no Critical or High issues; the most severe finding was detected only by manual testing. Because the corpus is small and each variant was generated once, we report descriptive observations rather than statistically established effects, and position the work as a preliminary study whose pipeline is being scaled to multiple models and repeated runs.

Comment
Accepted for presentation at the 37th Central European Conference on Information and Intelligent Systems (CECIIS 2026), September 16-18, 2026, Varazdin, Croatia. Author's accepted manuscript

Closest papers

Nearest neighbours by meaning across the whole index.

SecureVibe: Making Vibe Coding More Secure

cs.CR cs.CL ·2026-09-29 ·Danqing Wang, Baolin Peng, Zhepei Wei, Isadora White, Wenlin Yao +6 more
similarity 0.72
Abstract

As vibe coding becomes increasingly capable and widespread, security vulnerabilities in even functionally correct solutions are a growing concern. When investigating functionally correct but insecure solutions, we find that the insecure agent is less than half as likely to conduct effective planning and testing for the hidden security risks behind the functional requirements. Motivated by this, we develop SECUREVIBE, a training recipe that explicitly targets planning and testing for code security. SECUREVIBE constructs training signals around these security behaviors. It includes supervised fine-tuning on the security suite with 4 security tasks, and post-training methods, SECUREVIBE_rl and SECUREVIBE_hg, to enhance security capabilities from verifiable execution feedback and hint-based self-supervision. Our SECUREVIBE outperforms the baseline on two types of security coding tasks across 4 benchmarks. Specifically, SECUREVIBE improves the security pass@1 by 6.9 points on BaxBench. The gains extend to unseen CWE categories, with improvements of 11.5 points on SusVibes. Meanwhile, it also improves functionality pass@1 by 13.6 points on the security coding task SusVibes and 4.1 points on the generic coding task SWE-bench Verified. Further analysis offers two practical insights: (i) diversifying supervision across security planning, coding, and testing strengthens security behaviors more effectively than adding coding trajectories alone, and (ii) hint-guided supervision is particularly valuable when the agent's existing security capabilities are insufficient to learn effectively from outcome feedback.

0.20

Threat Landscape of Vibe Coding and the Road to Secure Autonomy

cs.CR cs.ET ·2026-11-16 ·Devki Nandan Jha, Shubh Anand, Karthick Keeranagere Krishnaiah, Roman Ahearn, Suhas Nale
similarity 0.70
Abstract · 35 pages, 6 figures, 5 tables, Submitted to ACM Computing Surveys

Vibe coding is reshaping software development, enabling faster implementation and allowing individuals with limited programming experience to describe requirements in natural language while AI agents write, test, and deploy the code. As these agents evolve into autonomous systems that execute commands, manage dependencies, access cloud resources, and interact with other agents, their security risks extend beyond incorrect code to actions with direct consequences for software systems. Although research on these risks is expanding, it remains fragmented across areas including prompt injection, insecure code generation, supply-chain attacks, and human factors, with limited consensus on terminology, threat assumptions, and evaluation. This paper consolidates emerging research through the lens of the vibe coding workflow. We establish a reference architecture, characterise six levels of agent autonomy, and develop a threat model centred on five trust boundaries spanning developers, AI models, tools, cloud infrastructure, and the wider software ecosystem. Building on this foundation, we propose a taxonomy of twenty-one threats across nine classes. For each threat, we review the available evidence, assess proposed defences, and identify limitations. Our analysis highlights that risks increase with agents' operational authority, while most defences rely on model judgement and address isolated threats. Finally, we identify eleven research challenges, including enforceable limits on agent authority and shared benchmarks and methodologies for assessing vibe coding security.

0.20

Prompt Structure Redistributes, Not Reduces: An Empirical Analysis of Security-Weaknesses in LLM-Generated Python Code

cs.CR cs.SE ·2026-08-25 ·Maitreyee Das Urmi, Jessica Pourleyli, Fabio Santos, Glaucia Melo
published similarity 0.69
Abstract · Accepted at CASCON 2026

Large Language Models (LLMs) increasingly generate code from natural-language prompts, making prompt engineering a key mechanism for shaping the security of generated software. Structured and security-oriented prompts are widely used to encourage safer code, yet their effects extend beyond whether detected weaknesses are simply present or absent. Using 424 security-sensitive Python tasks, we generate solutions with GPT-4o and LLaMA 3.1-8B under five prompt variants that progressively add structural and security guidance, and evaluate them with Bandit and CodeQL along two axes: generation compliance and security weakness prevalence, severity, and CWE distributions. Structured prompting substantially reduces refusals (e.g., GPT-4o invalid outputs drop from 338 of 424 to 37-52), enabling large-scale analysis, but security-oriented refinements do not consistently reduce overall weakness prevalence. For GPT-4o, stronger prompts primarily redistribute risk: high-severity findings fall (20.8% to 13.6%) while low-severity findings rise (32% to 43.5%); LLaMA shows weaker, less consistent shifts. We also observe security-driven semantic drift, where stricter prompts silently remove or rewrite explicitly requested unsafe constructs. Overall, prompt structure improves compliance but is an unreliable substitute for robust security controls in LLM-assisted development.

0.27

Last Secure Code Benchmark: Is the Code You Vibe Secure?

cs.CR cs.AI ·2026-10-10 ·Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu +10 more
benchmark similarity 0.68
Abstract · 10 pages

AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.

0.25

CAT-RAG: Improving Secure Code Generation with Category-Aware Agentic Retrieval

cs.CR cs.SE ·2026-10-17 ·Mahdieh Abazar, Lorenzo De Carli, Benjamin Tan
similarity 0.67
Abstract

Despite rapid progress in LLM-based code generation, security vulnerabilities remain widespread, and retrieval-augmented code generation (RACG) can further amplify risk when retrieved knowledge is verbose or vulnerable to poisoning. Existing secure RACG approaches inject security knowledge at a fine-grained vulnerability level, causing prompt overhead and diluted model attention when a query implicates multiple related weaknesses. We propose CAT-RAG, an agentic security category-aware retrieval-augmented code generation framework. CAT-RAG performs adaptive category selection via an LLM as a categorizer agent and operates either as a standalone security layer or as a drop-in add-on to RACG pipelines, requiring no modification to the underlying model. We replace CWE-level retrieval with compact, language-specific security categories. These categories provide a higher-level abstraction over individual CWE weaknesses, organizing them by the security concern they share, such as Input Validation and Injection Attacks, rather than merely grouping related entries. Each category consolidates best practices and secure code examples into a single, prompt-efficient representation. We perform a comprehensive evaluation across CyberSecEval, CodeGuard+, and HumanEval. On CyberSecEval, CAT-RAG nearly doubles the secure code generation improvement over the state of the art (23.57% vs. 12.11%) while reducing token overhead by 41.71%.

0.12

Decalage: Uncovering the Gap Between Model-Generated Reasoning and Actual Code Implementation in Small Language Models

cs.SE ·2026-11-09 ·Saif Uddin Mahmud, Afsana Tasnim, Ahmed Ryan, Md Rayhanur Rahman
similarity 0.66
Abstract

Agentic code generation often asks a model to explain its reasoning before it writes code, and a common way to make such a workflow safer is to add a security instruction to the prompt. We test what that instruction does in small language models, which are attractive because they run locally and cheaply. Four open code models of 1.5B to 3B parameters generate C++ from 166 sections of eleven public requirements documents, once with a general prompt and once with a secure prompt. The secure prompt more than doubles the share of reasoning about validation, resource handling and threats, and we find no evidence that the rise is larger for requirements with security content. The code changes only in part. Among functions with a body and reasoning, the share with a check that our lexical test recognises rises from 23% to 39%, but more than a quarter of the functions remain stubs, compile rates stay low, and half or more of the functions whose reasoning mentions a check contain no such check. Two rounds of compiler-feedback repair leave more than 80% of sections unusable, and neither cppcheck nor an exploratory ESBMC run separates the prompts on the usable code. A hand check by two annotators supports the stub measure and the direction of the reasoning result. For small models, a security instruction changes the reasoning broadly but the code only in part, so the reasoning should be checked against the code rather than trusted.

0.29

From Instructions to Exploits: Applying Program Analysis to Malicious Agent Skills Across Natural Language and Code

cs.CR ·2026-10-24 ·Shiwu Zhao, Baizhu Wang, Guoqiang Zheng, Tingyu Gao, Yuheng Xie +4 more
benchmark similarity 0.66
Abstract

LLM-powered agents increasingly rely on third-party skills that combine natural-language instructions with executable code, creating supply-chain risks that may span both artifacts. Existing skill scanners often rely on rule matching or direct LLM judgments and provide limited support for tracing such cross-domain behavior. We present the Supply-Chain Scanner for Agent Skills (SCS), a static analysis framework that unifies instructions and code for security analysis. SCS compiles natural-language instructions into a constrained Python-style intermediate representation using a limited set of security-oriented APIs. It then constructs program dependence graphs for instructions and bundled code and connects them into a cross-domain graph (xPDG). A Trust-Sensitivity-Capability (TSC) model propagates security properties over the graph to support structured, largely deterministic risk detection. On a manually verified benchmark of 117 malicious and 117 benign skills, SCS achieves the highest average accuracy (85.6%), recall (95.2%), and F1-score (86.9%) among evaluated methods across five LLMs, with 80.2% precision and average token consumption of 13.7 million. Its NL-to-IR compiler achieves 92.5%-94.5% step-level correctness on 400 human-annotated instruction steps. Finally, analyzing 66,157 real-world ClawHub skills yields 2,473 candidates, of which 627 are confirmed malicious through auxiliary dynamic validation and manual review.

0.18

Security in LLM-Assisted Coding: A Systematic Literature Review

cs.CR cs.AIcs.SE ·2026-10-11 ·Liang Yuan, Maryam Khanian Najafabadi, Mashud Rana, Varvara Vetrova
publishedsurveybenchmark similarity 0.66
Abstract · 35 pages Accepted for publication in ACM Computing Surveys (CSUR). A copy of the formal acceptance letter can be verified via the publisher'…

Large Language Models accelerate software development but introduce severe security vulnerabilities. This systematic review investigates security risks in automated coding, benchmark limitations, and the role of human developers. Findings indicate that models frequently generate insecure code and faulty patches due to training data biases. Additionally, the models themselves are susceptible to direct attacks. Current evaluation methods suffer from data contamination and lack robust security metrics. Consequently, human oversight remains essential. Safer integration of automation will require more security-aware model architectures, stronger security-specific benchmarks, and more effective human-AI collaboration frameworks to mitigate these systemic risks.

0.45