CS Paper Radar

Vulnerable Code Search: Transferable Attack for Code Language Models

Kaicheng Wang, Liyan Huang, Jesse Thomason, Weihang Wang

cs.SE cs.CR ·submitted 2026-08-26 17:11 UTC · v1
top venue

Abstract

Reliable code retrieval is crucial for developer productivity and effective code reuse. However, current neural code language models (CLMs) powering search tools are susceptible to adversarial attacks targeting non-functional textual elements. In this paper, we introduce a programming language-agnostic, transferable, adversarial attack that exploits this CLM vulnerability. Our approach perturbs identifiers within a code snippet without altering the snippet's functionality to artificially align the code with a target query. We demonstrate that our attack, even when computed using smaller code embedding models, such as CodeT5+, is highly effective and transferable to larger, closed-source embedding models, like Voyage-code-3, or LLMs like Gemini-3.1-Pro. Our attack can increase the similarity between the query and arbitrary, irrelevant code snippets, consequently degrading key retrieval metrics such as the Mean Reciprocal Rank (MRR) of state-of-the-art models by up to 77%. The experimental results highlight the fragility of current code search methods and underscore the need for more robust, semantic-aware approaches.

Comment
EMNLP Findings 26

Closest papers

Nearest neighbours by meaning across the whole index.

Towards Effective Black-Box Adversarial Attacks on Deep Code Models via Structural and Identifier Perturbations

cs.CR cs.SE ·2026-08-12 ·Bin Duan, Jintao Lin, Dan Dongseong Kim, Guowei Yang
similarity 0.75
Abstract

Deep code models (DCMs) are increasingly embedded in code intelligence tasks. However, their robustness under adversarial attacks remains insufficiently understood. Prior black-box attacks mainly rely on identifier- only substitutions or structural edits transferred from reference samples, yielding perturbation spaces defined largely independently of the attacked input. We introduce Strike, an input-conditioned black-box adversarial robustness-testing framework that constructs and searches a hierarchical perturbation space for each input. Strike first partitions code into blocks and uses an LLM to generate context-specific structural candidates, filtered for syntactic validity, ranked by similarity, and adaptively combined. It then refines the best structural variant through similarity-guided identifier substitutions drawn from dynamically constructed candidate pools. Evaluations across representative code intelligence tasks, including clone detection, vulnerability detection, and code summarization, demonstrate that Strike achieves higher attack success rates with competitive target- model query overhead. Strategy-specific static checks across all three tasks and execution-based validation on the executable Juliet vulnerability-detection subset provide complementary, task-bounded evidence regarding the validity of the generated variants. Representation similarity and human evaluation further indicate that the variants remain highly similar to the original code and contextually natural. Fine-tuning with Strike- generated samples improves cross-attack performance on fixed perturbed evaluation sets while preserving clean performance.

0.04

Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks

cs.SE ·2026-09-09 ·Dongdong Zhao, Jian Chen, Guancheng Lin, Jianwen Xiang, Jacky Wai Keung +1 more
benchmark similarity 0.71
Abstract

Code generation benchmarks are widely used to evaluate Large Language Models (LLMs), but benchmark data leakage into training sets can inflate performance and undermine evaluation validity. DetectLeak, a method specifically designed for code generation benchmark leakage detection, relies on perplexity scores to identify likely leaked samples. However, perplexity mainly reflects general familiarity with code patterns and may perform poorly on complex or rare samples. It also overlooks other useful signals, such as code similarity, functional correctness, and semantic representations. To address these limitations, we propose CGMIA (Code-Generation-specific Membership Inference Attack), a method for detecting leakage in code generation benchmarks. CGMIA fine-tunes a shadow model on a subset of benchmark samples to construct labeled member and non-member data. For each sample, it collects the input prompt, generated code, and reference solution, and extracts expert features, including CodeBLEU, edit distance, test pass rate, and perplexity, together with semantic features from CodeBERT embeddings. An integrated learning module combines these features to capture both surface-level memorization signals and deeper behavioral patterns, enabling a classifier to predict whether a sample was included in the target model's training set. Experiments on eight code generation benchmarks show that CGMIA outperforms eight existing membership inference methods in most cases. It also effectively detects known leaked APPS samples in StarCoder-7B's training data.

0.19

CodePoisonRAG: Knowledge Poisoning Attacks on Retrieval-Augmented Code Generation

cs.CR cs.LG ·2026-09-02 ·Varun Gadey, Ziad Marey, Alexandra Dmitrienko
similarity 0.70
Abstract · 16 pages, 1 figure. Under review

Retrieval-Augmented Code Generation (RACG) improves LLM-based software development by retrieving external code artifacts, documentation, and patches, and incorporating them into the generation context. This reliance on external knowledge introduces a critical trust boundary: poisoned artifacts can influence generated code without modifying the underlying LLM. Prior work shows that selecting existing vulnerable examples can increase the general vulnerability rate of RACG outputs, but leaves open whether a black-box attacker can construct a single task-matched artifact that propagates an attacker-selected weakness. We introduce CodePoisonRAG, a targeted upstream knowledge-poisoning framework that transforms benign fixed-code entries into poisoned artifacts. Its attack chain combines CWE-specific Vulnerability Injection, which embeds a selected source-to-sink flow while retaining task alignment, with Semantic Mislabeling, which adds false safety claims without repairing the vulnerable behavior. The attacker has no access to the victim's deployed knowledge base, retriever, re-ranker, generator, prompt, or defense mechanism and injects at most one artifact per anticipated programming task. We construct 85 poisoned artifacts covering ten CWE classes across Java and C, yielding an aggregate corpus-poisoning ratio of 0.7%. Across three generators, all 85 artifacts appear among the Top-3 results for their corresponding queries, and CodePoisonRAG achieves attack success rates between 0.80 and 0.93. Against CodeGuarder, which injects vulnerability-specific security knowledge into the generation context, the attack retains success rates between 0.40 and 0.71. These results show that RACG poisoning extends beyond the incidental propagation of existing vulnerabilities to the targeted construction and propagation of attacker-selected weaknesses.

0.41

Embedding Drift in Code Vulnerability Models Under Intended Behaviour-Preserving Transformations

cs.SE ·2026-09-18 ·Hasti Ghaneshirazi, Tahsin Reza, Ladan Tahvildari
similarity 0.69
Abstract

Code edits that preserve program behaviour can shift frozen code embeddings across classifier decision boundaries, causing correctly detected vulnerabilities to be predicted as benign. Such instability is problematic because semantically neutral changes, including comment removal, insertion of unreachable code, variable renaming, and loop rewriting, should not alter a model's security judgment. We investigate this issue using 15,000 C/C++ functions organized as 7,500 vulnerable-patched pairs from the Big-Vul dataset. Four mutation tracks are applied to both vulnerable and benign samples: comment removal, unreachable code insertion, variable renaming, loop rewriting, and a combined setting that applies all transformations. Using frozen microsoft/codebert-base embeddings with six downstream classifiers, we evaluate the robustness of vulnerability predictions and introduce a train-only defence that projects clean and mutated representations closer together while preserving vulnerability-class information. Under combined mutations, the baseline Vulnerable Flip Rate (VFR) ranges from 35.55% to 42.15%. The proposed defence reduces the mean common-set VFR from 22.60% to 11.32% and improves mean mutated accuracy from 55.33% to 57.10%. For Logistic Regression, VFR decreases by 17.29 percentage points, although benign-to-vulnerable flips increase by 11.95 percentage points. These results indicate that the defence improves robustness to semantics-preserving code transformations but introduces a false-positive trade-off and does not fully eliminate prediction instability.

0.15

ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval

cs.SE cs.AIcs.CLcs.IR ·2026-09-01 ·Aaryan Kapoor, Md Abdullah Al Hafiz Khan
top venuebenchmark similarity 0.69
Abstract · Accepted to EMNLP 2026 (Main Conference). Camera-ready version. 17 pages, 6 figures, 8 tables

Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query's canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals -- execution-verified buggy variants near-identical to each canonical -- so that a retriever's rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNemar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D.

0.39

SemVul: Semantic-Enhanced Graph Neural Networks for Code Property Graph-based Vulnerability Detection

cs.SE cs.CR ·2026-09-05 ·Younas Affan, Leonardo Regano, Giorgio Giacinto
publishedbenchmark similarity 0.68
Abstract

Vulnerabilities in source code are often the root cause of cyberattacks worldwide, as attackers exploit weaknesses in software to gain unauthorized access, steal data, or disrupt services. In this study, we evaluated existing research approaches and propose SemVul, a vulnerability detection pipeline that demonstrates better generalization and higher accuracy in learning vulnerable code patterns. We propose a Code Property Graph-based vulnerability-detection approach combined with semantic-level enhancement, enabling the model to capture both the program's structural flow and the semantic meaning of the code. Our approach integrates both node-level and edge-level semantic embeddings using pre-trained code embedding techniques. We systematically evaluate multiple GNN architectures on publicly available benchmark datasets. SemVul is generic with respect to the programming language and supports multiple architectures. By integrating structural and semantic information, the proposed approach improves vulnerability detection performance. Our results show that SemVul outperforms existing approaches and provides better generalization.

0.33

Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models

cs.SE ·2026-09-03 ·Vishnu Asutosh Dasu, Ashish Kundu, Gang Tan
benchmark similarity 0.68
Abstract

This paper investigates whether large language models (LLMs) for code generation can recognize a programming request that cannot be satisfied. Coding assistants are widely deployed, yet they are evaluated almost exclusively on solvable tasks where a test suite can score the output. A request may instead name a package that does not exist or ask for an algorithm that is mathematically impossible, and the appropriate response is to refuse. Models nevertheless respond with plausible code, and such fabrications are also a security concern because attackers can register the package names that models invent. We define code hallucination as ungrounded code generation, distinguish it from ordinary programming errors, and introduce a taxonomy of these failures together with a benchmark of 270 adversarial requests and 91 matched solvable controls across six languages and 24 subcategories. We evaluate 17 open-weight models with 6.7B to 671B parameters, collecting 6,137 responses. Models hallucinate on 54% of the adversarial requests and correctly refuse 31%. Larger models do better on algorithmic impossibilities and conflicting requirements, where hallucination falls from 84% for models at or below 7B parameters to 32% for those at or above 80B. For plausible but nonexistent dependencies and features, the rates are 88% and 61%, and even the largest model hallucinates on all 22 requests for nonexistent JavaScript and Rust packages. Two-thirds of hallucinations are confident fabrications without caveats, and some reasoning models fail to return an answer on solvable tasks. Scale teaches models to recognize what is impossible but not what does not exist, so grounding code generation in the real software ecosystem, for example by checking package names against registries, remains necessary regardless of model size.

0.16

CAT-RAG: Improving Secure Code Generation with Category-Aware Agentic Retrieval

cs.CR cs.SE ·2026-10-17 ·Mahdieh Abazar, Lorenzo De Carli, Benjamin Tan
similarity 0.68
Abstract

Despite rapid progress in LLM-based code generation, security vulnerabilities remain widespread, and retrieval-augmented code generation (RACG) can further amplify risk when retrieved knowledge is verbose or vulnerable to poisoning. Existing secure RACG approaches inject security knowledge at a fine-grained vulnerability level, causing prompt overhead and diluted model attention when a query implicates multiple related weaknesses. We propose CAT-RAG, an agentic security category-aware retrieval-augmented code generation framework. CAT-RAG performs adaptive category selection via an LLM as a categorizer agent and operates either as a standalone security layer or as a drop-in add-on to RACG pipelines, requiring no modification to the underlying model. We replace CWE-level retrieval with compact, language-specific security categories. These categories provide a higher-level abstraction over individual CWE weaknesses, organizing them by the security concern they share, such as Input Validation and Injection Attacks, rather than merely grouping related entries. Each category consolidates best practices and secure code examples into a single, prompt-efficient representation. We perform a comprehensive evaluation across CyberSecEval, CodeGuard+, and HumanEval. On CyberSecEval, CAT-RAG nearly doubles the secure code generation improvement over the state of the art (23.57% vs. 12.11%) while reducing token overhead by 41.71%.

0.12