CS Paper Radar

Enhancing User Resilience Against AI-Augmented Phishing: A Two-Stage Framework for Detection and Personalized Training

Weihao Qu, Gurmeet Singh, Daniel Crawford, Bingjun Li, Jalen Smith

cs.CR ·submitted 2026-08-21 18:36 UTC · v1
published

Abstract

The rapid development of artificial intelligence, including agents and deepfake techniques, has accelerated phishing attacks and lowered the threshold for attackers. Modern phishing attacks now blend multiple tactics, including social engineering, URL spoofing, and AI deepfakes enabling adversaries to craft highly convincing messages that exploit human vulnerabilities and bypass traditional detection systems. At the same time, current security awareness education struggles to keep up with the speed, sophistication, and complexity of these evolving threats. To address this challenge, we propose a two-stage anti-phishing framework, CyberGLA, that combines technical defense and user-centered security education. In the Detection stage, we introduce EmailKnight, a spoof detection tool that performs multi-level email analysis. To enhance user awareness, the Training stage incorporates a large language model (LLM)-based security coach that dynamically selects personalized training modules based on the outcomes of the Detection stage. This dual purpose design philosophy enables effective protection against the evolving threats of modern email phishing attacks.

Comment
7 pages, 2 figures, 2 tables. Published in the Journal of The Colloquium for Information Systems Security Education
Journal ref
Journal of The Colloquium for Information Systems Security Education, 13(1), Article 7 (2026)
DOI
10.53735/cisse.v13i1.230

Closest papers

Nearest neighbours by meaning across the whole index.

AgentPhishing: A Benchmark of LLM Email Agents Under Adaptive Attack

cs.CR cs.AI ·2026-10-22 ·Justin Tan, Miao Li, Ding Zhao
benchmark similarity 0.76
Abstract · 28 pages, 6 figures

Computer-use agents increasingly handle private information and act on users' behalf across desktop and web applications, including email. Phishing therefore poses privacy and security risks that depend not only on whether agents recognize malicious messages, but also on how they respond. Existing phishing datasets were developed primarily for conventional detection, leaving agent susceptibility to persuasive malicious requests insufficiently characterized. To expose vulnerabilities that pre-existing datasets may miss, we introduce AgentPhishing, a benchmark for evaluating whether persuasive phishing attacks can evade detection and induce LLM email agents to comply with malicious requests. We develop PhishOpt, which optimizes reusable email-generation skills against individual agents under fixed benchmark constraints and a realism constraint. We analyze the generated emails using Cialdini's persuasion principles to identify strategies associated with agent failures and use these findings to construct a universal generation skill. Our evaluation shows that agents can handle pre-existing phishing emails safely yet remain susceptible to attacks adapted to agent behavior, while in an AgentPhishing evaluation of nine agents, four exhibit phishing-detection rates below $10\%$ and malicious-request compliance rates above $45\%$. These results highlight the limits of inferring agent safety from conventional phishing evaluations and the need to assess how agents respond to persuasive malicious requests.

0.26

Breaking and Defending LLM-Powered Social Media Bot Detection Systems

cs.AI ·2026-08-16 ·Nof Orenstein, Yoni Birman
published similarity 0.71
Abstract · Accepted at ACISP 2026 (Australasian Conference on Information Security and Privacy). Also accepted as a poster at IEEE Symposium on Securit…

The rise of social media bots poses a persistent threat, enabling misinformation, opinion manipulation, and the erosion of trust in online platforms. To combat this, machine learning systems have been developed to detect and limit bot activity, but attackers continuously adapt through techniques such as adversarial learning and behavior imitation, fueling an ongoing arms race between bots and detection tools. Recent advances in large language models (LLMs) have significantly improved bot detection by enabling deeper semantic and contextual analysis of accounts and their content. However, this shift also introduces new attack surfaces, allowing adversaries to craft exploits that directly target the reasoning and generation mechanisms of LLM-based classifiers. Industry tools such as Anthropic's Claude Code Security similarly leverage LLMs for security-critical decisions, further motivating a careful study of their attack surfaces. In this work, we investigate both the offensive and defensive aspects of LLM-powered, threat-specific cybersecurity applications. While centered on the challenge of social media bot detection, our methodology and insights generalize to a broad class of LLM-powered cybersecurity systems, including phishing detection, email classification, and fraud analysis. We introduce two novel adversarial attack strategies that systematically exploit the semantic and contextual weaknesses of LLM-based classifiers, degrading their detection accuracy by up to 48%. To counter these threats, we propose a robust multi-LLM defense architecture designed to preserve detection reliability under adaptive adversarial conditions. Our solution, LSABRE (LLM-powered Social Adversarial Bot Recognition Ensemble), is a multi-LLM framework that substantially improves robustness across a range of attacks, maintaining 86% detection accuracy even under strong, adaptive adversarial pressure.

0.40

Nebulon Enterprise Simulated Threats for Phishing Research (NEST-Phish): A Synthetic Enterprise Phishing Email Dataset for Behavioral and Machine-Learning Research

cs.CR ·2026-09-03 ·Emily J. Winokur, Lauren S. Treiman, Allen G. Moore, Paul Schutte, Joe Ingram +1 more
benchmarkdatasetcode/data similarity 0.71
Abstract · Dataset paper; associated public dataset available on Kaggle: https://www.kaggle.com/datasets/emilywinokur/synthetic-email-corpus

Phishing remains one of the most persistent cyber threats, yet publicly shareable datasets for studying phishing in realistic enterprise email settings remain limited. To address this gap, we introduce a synthetic enterprise phishing email dataset built around a fictitious organization, Nebulon. The dataset spans a broad set of workplace communication themes and includes matched synthetic legitimate and phishing emails with interpretable phishing-cue annotations. Here, ``legitimate'' denotes the non-phishing class, not legitimately occurring organizational emails. Human-subject categorizations and classifier evaluations show that the dataset supports meaningful variation in phishing judgments while also providing learnable signal for supervised detection. This publicly released resource is intended to support future work on phishing detection, human susceptibility, explainability, and benchmark development in enterprise-like contexts.

0.21

AURA: Adaptive Uncertainty-Routed Analysis for Email Threat Detection

cs.LG ·2026-09-17 ·Omran Berjawi, Walid fahs, Rida Khatoun
similarity 0.67
Abstract · Under review

Email spam and phishing attacks remain a critical security threat. Adversaries increasingly exploit large language models to craft contextually convincing malicious messages, and existing spam detection systems often struggle to keep pace. Generalization across diverse and evolving attack scenarios is limited, which reduces effectiveness once these systems are deployed in practice. This paper introduces Adaptive Uncertainty-Routed Analysis (AURA), a multimodal email threat detection system that analyzes both the content of an email and its embedded URLs. AURA is built around two layers: the first quantifies prediction uncertainty from a URL classifier, and only ambiguous messages are escalated to a fine-tuned transformer encoder for semantic analysis. The system is evaluated on eight heterogeneous training corpora together with two held-out real-world corpora spanning a decade of adversarial campaigns. AURA reaches a macro F1-score of 0.9858 in-distribution, and on NazPhish-Eval and GuenterTrap-Eval it maintains 0.9502 and 0.9436, respectively, which is evidence of robust generalization under genuine distribution shift.

0.20

PHISHREV+: An ASP-Based Post-Hoc Defeasible Machine Learning Framework for Phishing Detection

cs.AI cs.LG ·2026-11-06 ·Mainak Sen, Kumar Sanka Ray, Amlan Chakrabarti
benchmark similarity 0.66
Abstract

Phishing website detection remains a critical cybersecurity challenge. Although machine learning classifiers achieve high accuracy on benchmark datasets, they suffer from a fundamental limitation: once trained, their decision boundaries are fixed and cannot incorporate new contextual evidence without retraining the model. This work presents \textbf{PHISHREV+}, a hybrid framework that instantiates the notion of \textit{post-hoc defeasible machine learning}. This framework does not replace the classifier. It treats the initial classifier predictions as defeasible beliefs and revises them through an Answer Set Programming based non-monotonic reasoning(NMR) layer. The framework operates in two phases. In the machine learning phase, four classifiers (SVM, KNN, DT and RF) are trained on the 87 feature benchmark dataset. In the post-hoc NMR phase, four defeasible rules derived from exploratory data analysis of the training set, and validated through Chi-square and Mann-Whitney U tests, formally revise initial predictions. The proposed framework achieves consistent false positive reduction across all our classifiers SVM (41$\rightarrow$39), KNN (47$\rightarrow$46), DT (62$\rightarrow$61), and RF (52$\rightarrow$51) while maintaining complete false negative parity with zero increase overhead. A comprehensive ablation studies across seven rule configurations confirms that the hierarchical combination of all four defeasible rules is necessary to achieve this optimal balance. Instance level explainability is provided through eXplainable ASP (XASP), which generates formal derivation graphs showing which defeasible rules fired and why the initial belief was retracted, a capability absent from all existing phishing detection approaches.

0.13

CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents

cs.CR cs.AI ·2026-10-06 ·Rafid Ahmed, Joseph Fioresi, Mubarak Shah, Yuzhang Shang
benchmark similarity 0.66
Abstract

Language model agents are increasingly deployed to automate everyday digital chores from managing emails and social media to handling banking and bills allowing users to step away from supervision. However, this capability also exposes sensitive information to phishing. Safe execution requires distinguishing malicious requests from genuine ones without simply refusing to act. Despite its practical importance, this problem remains underexplored and it is unclear whether current agents or existing defenses can achieve it. To study this problem, we first propose CredLeak-Bench, a comprehensive benchmark designed to evaluate how effectively and securely agents automate human workflows when confronted with phishing and identity verification. The benchmark covers both user-directed authentication and autonomous inbox monitoring, where agents are not explicitly instructed to log in. It systematically varies deceptive cues and pairs phishing scenarios with legitimate counterparts, enabling joint evaluation of information leakage and utility on genuine tasks. Within a sandboxed environment, leakage is measured through actual submissions of information rather than agents' self-reported behavior. Our evaluation reveals that all tested models are vulnerable to leakage. Agents also disclose sensitive information during autonomous inbox monitoring, demonstrating that phishing can induce disclosure without a user request to authenticate. Furthermore, most evaluated mitigations that reduce leakage also impair performance on genuine tasks, exposing a security utility trade off in existing defenses. These findings show why reducing leakage alone is insufficient: effective defenses must prevent unauthorized disclosure while preserving legitimate task completion. CredLeak-Bench provides a controlled framework for measuring both objectives and evaluating progress toward secure, useful agents.

0.18

PhishGen: Understanding LLM Capabilities in Automated Phishing Website Generation

cs.CR ·2026-10-21 ·Fujiao Ji, Yerim Kim, Yena Cho, Seonhye Park, Hyoungshick Kim +1 more
code/data similarity 0.66
Abstract

Phishing relies on visual deception, traditionally requiring web-development skills or pre-built kits. LLMs erode this barrier by generating functional websites from natural-language prompts, yet prior work does not answer how far LLM-driven generation actually reaches: functionality is assessed by inspection rather than execution; visual fidelity is scored by automated metrics or a few internal coders rather than the humans such pages aim to deceive; evaluation is confined to commercial models; and no study benchmarks generated pages against phishing kits deployed in the wild. We present PhishGen, a framework that evaluates the appearance fidelity and end-to-end functionality of LLM-generated phishing pages. PhishGen constructs prompts across three input modalities in seven combinations with twelve safeguard-bypassing rewriting strategies, generates pages using 14 LLMs spanning commercial, open-source, and uncensored categories, and assesses each via automated similarity metrics, reference-based detectors, a crowdsourced user study, and execution-based functionality testing, benchmarking all results against 64 real-world phishing kits. Although 96.2% of safeguard-bypassing outputs render, their perceived fidelity is modest (2.02/5), and only 38.7% complete the full credential-harvesting pipeline, with the Unsaved error as the largest failure mode. Contrary to the belief that uncensored models pose the greatest threat, GPT 4.1 reaches 86.3 to 94.1% functional success while uncensored models rarely exceed 45.4%. Phishing kits remain more complete per sample, yet three strong LLMs surpass the kit-level rate (70.9 to 89.7%), showing that LLMs trade average completeness for scalable, low-barrier generation of working phishing pages. We further identify generator-level fingerprints that current similarity-based defenses miss, and distill recommendations for hybrid detectors.

0.40

A Large-Scale Empirical Study of Modern Phishing Email Content

cs.CR ·2026-09-25 ·Jaehwan Park, Woonghee Lee, Fujiao Ji, Doowon Kim
similarity 0.66
Abstract · 22 pages

Phishing remains one of the most pervasive threats to Internet users, and email remains its predominant delivery channel. Email content is the attack surface of phishing: it is what the victim reads and what automated defenses inspect. Yet the composition of modern phishing content is poorly measured. Prior work has characterized dimensions such as theme, call-to-action (CTA), and impersonation, but not at scale, and their associations and temporal changes remain unclear, owing to small or source-specific corpora, bag-of-words topic models, and a focus on text alone. We present a content-focused measurement study of 2.9M distinct real-world phishing emails collected over 13 months (June 2025 - June 2026) in collaboration with the Anti-Phishing Working Group (APWG). We treat each email as a composite artifact comprising message text and its attachments: 272K images, 143K PDFs, and 57K calendar invitations. Using an LLM pipeline validated against human-annotated samples, we analyze these components along three dimensions (theme, CTA, and impersonation), examine the associations among them, and measure longer-term change against a historical dataset. We find that attackers diversify what they use to deceive but converge on how victims should respond: no theme exceeds 21.3% of emails, while a single CTA, URL navigation, accounts for 73.0%. CTA and impersonation choices are conditioned on theme. Attachments play three roles: images supplement the message text, PDFs substitute for it by carrying the pretext, and calendar invitations reinforce it by replicating interaction endpoints into a persistent medium. Over the longer term, the dominant CTA for invoice-themed phishing shifted from URL navigation to offline communication, rising from 6.7% in 2015 to 46.9% in 2025.

0.26