CS Paper Radar

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset

Norbert Sándor Szolnoki, Gábor Antal

cs.SE ·submitted 2026-11-04 11:34 UTC · v1
publishedbenchmark

Abstract

Recent studies report promising results from supervised fine-tuning of pre-trained code language models for automated vulnerability detection on benchmarks such as PrimeVul. In this paper, we present a large-scale empirical evaluation of both model performance and dataset quality in this setting. We systematically fine-tune three widely used encoder models (CodeBERT, GraphCodeBERT, and UniXCoder) across five imbalance mitigation strategies, extensive hyperparameter sweeps, and multiple random seeds, resulting in 770 independent training runs. Our results reveal a consistent pattern. While standard metrics such as F1 and ROC-AUC occasionally suggest improvements, security-critical and calibration-aware measures tell a different story. Matthews Correlation Coefficient remains near zero, predicted probabilities are poorly calibrated, and evaluation under strict false-positive constraints reveals high false-negative rates. Neither imbalance handling, threshold optimization, nor post-hoc calibration recovers a meaningful discriminative signal. We also conduct a data quality assessment of the PrimeVul benchmark and identify structural issues such as minimal semantic differences between vulnerable and fixed functions, and annotation noise introduced by commit-level labeling. These findings suggests that the PrimeVul dataset, in its current form, presents significant challenges for reliable AI-based vulnerability detection in software engineering with the methodologies we used. Our study highlights the need for more robust dataset design, improved evaluation protocols, and rethinking of fine-tuning practices for security-critical tasks.

Comment
Presented at EASE 2026 AI Models/Data track: https://conf.researchr.org/track/ease-2026/ease-2026-ai-models---data?

Closest papers

Nearest neighbours by meaning across the whole index.

PatchCon: Patch-aware Weighted Supervised Contrastive Learning for Vulnerability Detection

cs.SE ·2026-11-04 ·Chaomeng Lu, Pooya Rostami Mazrae, Bert Lagaisse
similarity 0.76
Abstract

Automated vulnerability detection is critical for securing modern software systems, yet existing learning-based approaches often treat functions as independent samples and underuse the supervision provided by vulnerability-fixing patches. In this paper, we propose PatchCon, a CodeBERT-based patch-aware contrastive learning model for function-level vulnerability detection. PatchCon jointly optimizes vulnerability classification and supervised contrastive learning, and treats vulnerable-patched function pairs as hard negatives with larger contrastive weights to capture subtle security-relevant differences introduced by fixes. To evaluate patch-level discrimination, we use two pairwise metrics that measure whether a model can correctly distinguish vulnerable functions from their patched counterparts. We evaluate PatchCon on PairVul-Linux through extensive experiments, including state-of-the-art comparisons, ablation studies, hyperparameter sensitivity analysis, and statistical significance testing. We also conduct fine-grained analyses by Common Weakness Enumeration (CWE) type, function length, and patch size, and further evaluate PatchCon on Linux kernel vulnerabilities disclosed after the training data cutoff. On PairVul-Linux, PatchCon outperforms representative pretrained code models and contrastive learning baselines on both classification and pairwise metrics, improves pairwise accuracy over the strongest reproduced baseline from 0.2060 to 0.3264, and achieves pairwise accuracy comparable to that of a GPT-4o-based large language model (LLM) method at a substantially lower cost. Among the reproduced models, PatchCon also achieves the highest pairwise accuracy on test pairs from Common Vulnerabilities and Exposures (CVE) entries unseen during training and on newly disclosed vulnerabilities.

0.06

Structured Security Explanations for Learning-Based Vulnerability Detection

cs.CR cs.SE ·2026-10-27 ·Fares Trad, Yizhou Chen, Hung Viet Pham, Zeyu Sun, Gias Uddin
similarity 0.76
Abstract · 21 pages, 3 figures, 9 tables

Software vulnerability detection remains difficult because existing approaches rely on predefined analysis rules, learn vulnerability patterns implicitly from code and labels, or use large language models (LLMs) to make the final security judgment. It remains unclear whether structured descriptions of security-relevant program behavior can improve supervised detection. We present VulnLens, an explanation-augmented framework for binary C/C++ vulnerability detection that separates program-behavior analysis from the final decision. A label-blind LLM-based Explainer examines each function from seven fixed perspectives: data flow, memory access, object lifetime, arithmetic and type behavior, control flow, shared state, and external-resource access. Without receiving vulnerability labels or CWE information, it generates structured explanations. A supervised Detector combines source code and explanations through sequential cross-attention. We evaluate VulnLens on a PrimeVul-derived dataset covering ten CWE categories and test transfer to SecVulEval without retraining. On the 526-function PrimeVul test set, VulnLens achieves 75.5% accuracy and 77.2% F1, exceeding DeepSeek V4 Pro by 4.2 percentage points in accuracy and Gemini 2.5 Pro by 2.2 points in F1. Matched explanations improve F1 by 7.3 points over a separately trained code-only detector. On SecVulEval, VulnLens achieves 66.4% accuracy and 65.8% F1, outperforming DeepSeek V4 Pro by 7.6 points in accuracy and 20.0 points in F1, and Gemini 2.5 Pro by 16.6 points in accuracy and 7.8 points in F1.

0.16

Enhancing LLM-Based Vulnerability Detection via Representation-Aware Logical Ensemble

cs.CR ·2026-10-24 ·Francesco Crescenzo Grasso, Pietro Liguori, Roberto Natella, Domenico Cotroneo
published similarity 0.73
Abstract · Accepted for publication at QRS 2026

Automated vulnerability detection in C source code is a long-standing challenge, as traditional static analysis tools often trade precision for scalability and struggle with complex vulnerability patterns. Recently, large language models (LLMs) have shown promising results; yet, most approaches rely on a single code representation, implicitly assuming uniform suitability across vulnerability types. In this work, we systematically study LLM-based vulnerability detection under multiple code representations, including raw source code, Abstract Syntax Trees (AST), and Control Flow Graphs (CFG). Through extensive empirical analysis, we show that different representations exhibit complementary strengths, with specific views being more effective for particular classes of vulnerabilities. Building on these findings, we propose a Representation-Aware Logical Ensemble (RALE) that integrates LLMs trained on different code representations through explicit logical rules and majority voting. While single-representation models achieve up to 73% F1-score, our integrated approach reaches 78% F1-score, with commensurate gains in accuracy, precision, and recall, consistently outperforming individual models. These results highlight the importance of explicitly leveraging complementary code representations when combining LLMs for effective vulnerability detection.

0.29

Mitigating fix reversal: Counterfactual constraints for function-level vulnerability detection

cs.CR cs.AIcs.SE ·2026-11-18 ·Yun Bian, Shihao Li, Haiquan Wang, Yi Chen, Zhao Yang +1 more
similarity 0.72
Abstract · 21 pages, 9 figures

Function-level vulnerability detectors often exhibit fix reversal, assigning a higher vulnerability probability to a fixed version than to its vulnerable counterpart. Fixes usually lengthen functions, and larger length increases are associated with more severe reversal. Code-transformation experiments indicate that fix reversal mainly stems from a length shortcut: vulnerability probability increases with function length when content is fixed. We propose CFVul to address this problem. Sliding-window encoding covers complete functions, retaining fix information beyond the truncation boundary. During training, a content constraint encourages the vulnerable version to receive a higher vulnerability probability under a common length input. During inference, a length constraint averages vulnerability probabilities over fixed reference lengths, reducing the influence of actual length changes. On PrimeVul, CFVul achieves a fix-pair ranking metric, PairRank, of 59.4\%, exceeding the existing methods compared. For fixes that increase function length by more than 15\%, PairRank rises from CodeBERT's 25.8\% to 48.4\%. PairRank also improves in experiments with four additional models and three additional datasets. By weakening the length shortcut, CFVul reduces cases in which fixed versions receive higher vulnerability probabilities than vulnerable versions. It requires only existing fix pairs and applies to multiple pretrained code models, facilitating adoption in existing detectors.

0.17

Detecting Contaminated Code-Generation Prompt Batches via Influence Functions

cs.LG ·2026-08-14 ·Francesco Quinzan, Noor Munir, Yishun Lu, Stephen Roberts
benchmark similarity 0.71
Abstract

Large language models (LLMs) are increasingly used for code generation, yet they remain vulnerable to prompts that elicit insecure implementations. Existing defenses typically rely on predefined threat models or known vulnerability patterns, limiting their effectiveness against novel attacks. We propose CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of prompts that induce anomalous model behavior. Rather than detecting specific vulnerabilities, CodeSIFT measures the parameter-space influence of generated code and uses a statistical test to determine whether a candidate prompt set deviates from a benign reference distribution. To evaluate our approach, we introduce two benchmark datasets covering a variety of vulnerabilities. We evaluate CodeSIFT on three open-weight code LLMs ranging from 3B to 7B parameters, achieving AUROC scores of up to 0.98 at moderate-to-high injection rates, while maintaining well-calibrated false positive rates and substantially outperforming static analysis baselines. These results suggest that influence-function-based detection is a promising direction for identifying malicious code-generation prompts without requiring prior knowledge of the underlying attack class.

0.34

Diagnosing Generalization Boundaries in AI-Based Vulnerability Detection via Calibrated Embedding Distances

cs.SE cs.LG ·2026-10-09 ·Konstantin Ievtushenko, Ralf Lämmel, Jan Jürjens
benchmark similarity 0.71
Abstract · Submitted for review

High benchmark scores in learning-based vulnerability detection mask severe degradation under distribution shifts. This paper demonstrates that calibrated embedding geometry exposes these hidden generalization boundaries on syntactically similar vulnerable and patched code pairs. An evaluation of CodeBERT and Qwen2.5-Coder across pattern, project, and temporal shifts on a strictly paired dataset reveals three main findings. First, whitened intermediate-layer embeddings with local nearest-neighbour scoring separate unfamiliar samples with a mean out-of-distribution AUROC of 0.739 (RQ1). Second, distance scores reveal a shift-specific reliability gradient where accuracy drops by over 35 percentage points and cross-project shifts cause the sharpest degradation, even though temporal shifts drift geometrically farthest (RQ2). Third, false positives on patched code systematically correlate with geometric proximity to vulnerable training neighbourhoods, increasing from 9.4% in the lowest-risk decile to 58.8% in the highest (RQ3). These calibrated distances establish a diagnostic signal that maps representation geometry directly to model degradation without requiring auxiliary models.

0.15

When Models Know but Can't Linearly Say: Relational Encoding Limits Linear Vulnerability Classification in Code LLMs

cs.CR cs.AI ·2026-10-19 ·Rui Melo, Andre Catarino, Claudia Mamede, Corina Pasareanu, Rui Abreu
top venuehot direction similarity 0.71
Abstract · Accepted at EMNLP Findings

Large language models can detect vulnerable code, yet the form of vulnerability encoding in their representations remains unclear. We analyze 8 code models spanning 6.7B-15B parameters on C programs using three datasets (DELTASECOMMITS, SVEN, and PRECISE-BUGS), comparing two diagnostic tasks on the same representations: identifying what patterns the code instantiates, and determining whether it is vulnerable. Models encode the first with high fidelity. Under the studied paired C-code datasets and linear-probing setup, vulnerability information is more accessible as a paired relational signal-whether code A is riskier than code B-than as a globally separable categorical label. This structural difference has implications for classification: the model reliably ranks which code is riskier but standard classifiers on isolated samples achieve limited accuracy because vulnerable and secure samples form overlapping distributions in representation space.

0.67

How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?

cs.CR cs.SE ·2026-10-19 ·Haoran Xue, Nikta Akbarpour, Gias Uddin, Song Wang
benchmark similarity 0.69
Abstract

Large language model (LLM) coding agents, such as Claude Code, are increasingly used for security auditing, with task-specific skills providing reusable vulnerability-analysis procedures. However, their effectiveness in detecting real vulnerabilities and the reasons for their failures remain unclear. We conduct the first large-scale empirical study of 37 security skills using Claude Code with four underlying LLMs on 100 real Linux kernel vulnerabilities across ten classes, totaling 14800 runs. We evaluate vulnerability detection at the function, statement, and vulnerability-class levels. Results show that 24.4 percent of runs locate the correct function, 18.3 percent identify both the function and responsible statement, and 15.7 percent correctly classify the vulnerability. Our failure analysis reveals limitations in evidence acquisition, state-relation reasoning, code interpretation, candidate verification, investigation coverage, and reporting. Based on these findings, we propose SKILL-SIF, a self-improving framework that iteratively evolves security skills by learning from missed vulnerabilities and failure patterns. We evolve five top-performing skills using the original benchmark and evaluate them on 100 previously unseen vulnerabilities. SKILL-SIF improves fully correct vulnerability detection by up to eight times (from 2 percent to 16 percent) on unseen vulnerabilities, while aggregate accuracy across four LLMs increases by 1.85 times (from 6.05 percent to 11.20 percent). These findings highlight the limitations of existing security skills and demonstrate the potential of failure-driven skill evolution to improve vulnerability detection in coding agents.

0.26