CS Paper Radar

Fine-Grained RLAIF for Visual Safety: Learning to Zoom into Hazards for Multimodal Safety Alignment

Chunxue Xu, Yiwei Wang, Yujun Cai, Songze Li

cs.CV ·submitted 2026-11-15 04:23 UTC · v1
benchmark

Abstract

Recent multimodal safety alignment heavily relies on a text-centric paradigm, assuming risks are explicit in language or globally prominent in images. However, real-world hazards are often visually subtle or peripheral, causing existing models to fail due to their reliance on coarse, global visual representations. In this work, we study fine-grained visual safety alignment and introduce a challenging 1,200-sample benchmark across four domains, where hazards necessitate localized inspection. To address this, we propose Fine-Grained RLAIF (FGR), a reinforcement learning framework that casts image zoom-in as a learnable perception policy. FGR employs a decomposed reward design to jointly align three core safety behaviors: (i) where to look via a visual evidence reward, (ii) when to act via a tool-formatting constraint, and (iii) how to reason via visually grounded safety explanations. FGR achieves accuracy ranging from 86.71% to 93.00%, consistently outperforming strong multimodal and agent-based baselines.

Closest papers

Nearest neighbours by meaning across the whole index.

SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning

cs.CV cs.AI ·2026-08-11 ·Caoyuan Ma, Wenpu Liu, Weichu Xie, Tian Gu, Shilei Zhao +10 more
code/data similarity 0.75
Abstract · 16pages, 4 figures. Preprint. Project page: https://safe-vlm.github.io/SafeCap/ ; code: https://github.com/Safe-VLM/SafeCap

Large vision-language models (LVLMs) remain vulnerable to jailbreak attacks that exploit visual inputs to bypass safety alignment inherited from their language backbones. We propose SafeCap, a reinforcement-learning framework that aligns LVLMs through learned self-captioning. SafeCap trains a policy model to first generate a safety-relevant image caption and then produce a final answer; the caption is further optimized by whether it enables a frozen LLM to reach a safety-aligned decision. This caption-mediated objective encourages the policy to expose visual cues relevant to safe response generation rather than relying solely on direct refusal supervision. Across five multimodal safety benchmarks and six vision-utility benchmarks, SafeCap substantially improves aggregate safety performance under its intended DirectCap protocol, with gains of 3.7-19.0 points in safety average across four model settings while maintaining comparable or improved vision utility. Under controlled comparisons on matched backbones and data, SafeCap outperforms safety SFT, DPO, and SafeGRPO, demonstrating the effectiveness of caption-mediated reinforcement learning for multimodal safety alignment.

0.62

It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption Bank

cs.CV cs.LG ·2026-10-07 ·Samuel Tetteh, Cody Fleming
similarity 0.73
Abstract

Frozen vision-language models increasingly provide safety signals for reinforcement learning. Their use assumes that similarity to language describing danger indicates the hazard itself. Yet policy return and collision rate cannot reveal whether a score detects hazards or responds to correlated features of the scene. VLM-based methods have reported gains in driving and safe-RL benchmarks by converting image-text similarity into rewards, costs, or confidence weights. Such signals promise to reduce reliance on manually designed feedback. They may also reflect prompt structure, embedding geometry, or camera viewpoint, leaving their safety meaning unverified. To address this gap, we present a controlled evaluation of a frozen CLIP prompt-margin safety score. We apply the score to trajectories generated by policies that never receive it, match pre-contact observations to contact-free observations with comparable hazard geometry, and vary the captions, encoder, and camera view. Across three policies, 180 episodes, and 130 isolated contact onsets, the score decreases for about twenty steps before contact. Mechanism controls indicate that the score mainly tracks resemblance to the scene shared by its captions and changes with caption separation and camera view. A constant-confidence control retains the lower catastrophe-rate point estimate, so policy gains do not establish hazard perception.

0.15

Compositional Unalignment: Auditing Safety in Large Vision-Language Models

cs.LG cs.AIcs.CL ·2026-11-04 ·Hong kyu Lee, Kai Shu, Li Xiong
benchmarkposition similarity 0.73
Abstract

Large vision-language models (LVLMs) are vulnerable to compositional harm: an image and a prompt that are benign in isolation become harmful through the model's cross-modal reasoning. Recent alignment methods report strong scores on compositional safety benchmarks, but an LLM judge only measures observed behavior and cannot reveal how robustly safety is integrated into the model. We argue that un-alignment provides a complementary audit of alignment robustness. We first show that minimal supervised fine-tuning reaches an LLM-judge score comparable to a multi-stage reinforcement-learning pipeline, so similar benchmark scores cannot establish robustness by themselves. We then show that recognition and refusal of compositional harm are decoupled: nine standard direction-finding methods detect compositional harm with up to AUC=0.99, yet ablating the recovered directions either leaves refusal intact or destroys model coherence. We therefore propose CMaG, a training-free un-alignment method for compositional safety. CMaG measures refusal intention with a refusal-versus-compliance log-probability margin, takes its gradient with respect to mid-layer activations, and contrasts it against partially benign pairs to isolate the cross-modal interaction term. A single backward pass over 80 examples yields a direction along which steering monotonically converts refusal into compliance. On Qwen2.5-VL-7B and LLaVA-1.5-7B, CMaG reduces compositional safety scores by up to 65 points while over 98% of outputs remain coherent, outperforming gradient-based baselines; the direction also transfers to text-explicit harm and un-aligns models hardened by defensive training. Compositional safety can thus appear behaviorally strong while remaining fragile to targeted internal intervention. Warning: this paper contains unsafe model outputs, including references to self-harm.

0.09

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

cs.CL cs.CVcs.LG ·2026-09-03 ·Nicholas Sansoterra, Zishuo Zheng, Sachin Kumar
top venue similarity 0.73
Abstract

Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings, vision-language models (VLMs) introduce new challenges for IH: instructions may be embedded in images, split across modalities, visually transformed, or encountered during agentic tasks. Positing multimodal IH alignment as a reasoning problem, we train VLMs using reinforcement learning with rule-based rewards, comparing text-only, image-only, and mixed-modality supervision. We find that text-only IH training partially transfers to multimodal attacks, failing when models must decode, reconstruct, or reason over instructions across modalities. Image-based training improves robustness beyond text-only supervision, while mixed-modality training performs best overall. Importantly, the benefits generalize beyond the synthetic typographic training setting to real-image and web-agent safety tasks, while largely preserving general multimodal capability, showing that lightweight, verifiable supervision can meaningfully improve VLM robustness under adversarial, cross-modal, and interactive instruction conflicts.

0.36

SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Models

cs.AI cs.CV ·2026-08-29 ·Zongrui Wang, Xiangyang Zhu, Sicheng Wang, Han Wang, Dingyi Rong +11 more
datasetcode/data similarity 0.73
Abstract

Multimodal safety moderation requires distinguishing risks arising from visual content, user intent, and assistant behavior. Existing safeguards, however, are typically trained for a single judgment target and reduce safety assessment to a binary decision. Consequently, risk becomes difficult to compare across a multimodal interaction, and ambiguous cases are obscured. We introduce SafeAtlas-VL, a dataset of 1.5M training instances that places image-, request-, and response-level judgments on a five-level ordered scale. We curate a broad collection of safety-relevant data from both real-world and synthetic sources and apply a disagreement-aware annotation procedure. The resulting dataset spans 15 harm categories and 55 fine-grained subcategories, covering a broad range of multimodal safety scenarios. We also construct SafeAtlas-Bench, a held-out set of 5,000 instances for evaluating five-level predictions and continuous risk scores. Upon this dataset, we train the SafeAtlas Guard series of models via target-conditioned tuning for multimodal safety detection. Our models not only perform five-way classification of safety levels but also map safety to continuous scores through a soft cumulative ordinal head. Experimental results demonstrate that guard models trained on our dataset exhibit strong generalization: even without using the training sets of other benchmarks, they achieve competitive performance on the corresponding test sets. Notably, our 8B model attains the overall best performance, outperforming the previous SOTA by approximately 4% in F1 score. Code, data, and models are released to support further research. Warning: this paper contains example data that may be offensive, harmful, graphic, or disturbing.

0.25

Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding

cs.CV ·2026-10-21 ·Xin Zhang, Haochen Wang, Beibei Lin, Zhaoxiang Zhang, Robby T. Tan +2 more
benchmarkcode/data similarity 0.72
Abstract · 31 pages, 14 figures, 16 tables

Fine-grained multimodal intelligence requires models to both understand a designated region and localize it from language. Existing methods learn these directions with separate objectives and task-specific annotations. We present Generalized CycleGRPO, the first unified reinforcement learning framework coupling region understanding and localization across spatial, temporal, and spatio-temporal domains. Given a region, a multimodal large language model generates candidate descriptions and then localizes each description to reconstruct the input region. Reconstruction overlap provides a verifiable reward for captioning, while individual localization rewards improve the inverse mapping. A common region-space formulation extends this cycle from image masks and boxes to video intervals and timestamped spatial evidence, without reference captions or an external language-model judge during reinforcement learning. In videos, the cycle encourages descriptions that distinguish a target event from similar recurrences. We introduce DOVE (Distinctive Occurrence-level Video-caption Evaluation), a human-verified contrastive benchmark for this temporal distinctiveness. CycleGRPO improves SAMTok's overall gIoU from 57.5% to 68.2% on GroundingSuite and average gIoU from 71.2% to 78.7% on GRES. For temporal tasks, it achieves 51.9% tIoU on Charades and 33.3% Temporal Evidence Coverage on DOVE, compared with 45.6% and 14.8% for the Qwen3-VL initialization. These results establish region-text cycle consistency as a general self-evaluation principle for multimodal understanding and localization. Models, code, and DOVE are available at https://devinxzhang.github.io/Generalized-CycleGRPO-Page/.

0.37

ContraSafe-VL: A Contrastive Safety Preference Dataset for Vision-Language Models

cs.CV ·2026-09-30 ·Di Wu, Yanyan Zhao, Xin Lu, Hao Yang, Taotao Zhang +1 more
code/data similarity 0.71
Abstract · 5 figures, 5 tables

Vision-Language Models (VLMs) are increasingly deployed as general-purpose multimodal assistants, making multimodal safety alignment an important challenge. Safety depends on the interplay between textual and visual content, with visual context often playing a critical role. However, VLMs may fail to use visual information appropriately, either overlooking safety-relevant visual content or over-interpreting sensitive visual content, leading to unsafe responses or over-refusal. Existing multimodal safety alignment datasets largely follow the response-preference paradigm, which teaches models how to respond but provides limited supervision on how visual context should inform the safety judgment, making it difficult to balance safety and over-refusal. To address this limitation, we introduce \textbf{ContraSafe-VL}, a \textbf{Contra}stive \textbf{Safe}ty preference dataset for \textbf{V}ision-\textbf{L}anguage Models. For each fixed textual query, ContraSafe-VL constructs paired visual contexts that result in multimodal queries with different safety labels, providing direct supervision for how visual context should inform multimodal safety judgments. It further provides response preference pairs based on harmlessness and helpfulness to encourage models to generate responses that are both safe and helpful. Experiments across three VLM architectures and multiple benchmarks demonstrate that training with ContraSafe-VL improves cross-modal safety recognition and achieves a better safety--over-refusal trade-off while preserving general multimodal capabilities. The dataset is available at https://github.com/pikepokenew/ContraSafe-VL

0.25

ShieldVLA: Feasibility-Aware Safety Alignment for Vision-Language-Action Models

cs.RO cs.AI ·2026-09-02 ·Manan Tayal, Akshay Nambi
similarity 0.70
Abstract · 24 pages, 3 figures, 14 tables

Vision-Language-Action (VLA) models demonstrate strong generalization in robotic manipulation and navigation, but existing fine-tuning methods provide limited safety guarantees. Current approaches primarily rely on Lagrangian optimization that enforces safety through soft penalties on expected cumulative cost, often resulting in residual constraint violations or overly conservative behavior. Moreover, learning safety in visual domains is challenging due to the absence of dense per-step safety annotations. We propose ShieldVLA, a safety-aligned fine-tuning framework for VLA models based on Hamilton-Jacobi (HJ) reachability. ShieldVLA learns a model-free approximation of the HJ reachability value function directly from visual observations to estimate the safe operating region. The learned safety critic gates policy optimization by separating reward maximization within feasible regions from recovery near unsafe states, avoiding persistent reward-cost trade-offs. To enable scalable supervision in visual environments, we introduce rubric-based VLM safety scores that convert semantic safety feedback into structured critic targets without requiring manual cost labels. Across five navigation and manipulation benchmarks spanning multiple VLA backbones, ShieldVLA reduces cumulative safety cost by 57% on average and improves task success rate by +0.13 over SafeVLA.

0.26