CNRF Enables Evidence-Grounded Selective Safety for Vision-Language Models

Summary

The paper introduces Counterfactual Neighborhood Risk Fields (CNRF), a safety framework for vision-language models (VLMs) that aims to make risk scores more evidence-grounded and selectively actionable. Instead of treating representation-based separation between harmful and benign inputs as sufficient, CNRF builds context-matched benign-harmful counterfactual pairs and uses their representation differences to define capability-related risk directions. It estimates neighborhood support separately from risk, so unfamiliar inputs are less likely to be treated as harmful solely because they lack nearby evidence. A victim-adaptive interface accounts for differences in safety observations across VLM architectures. Selected requests are routed for same-victim response review: acceptable candidates can be released, while other responses are converted into constrained alternatives intended to reduce harmful capability. Across three VLMs and five multimodal jailbreak families, the framework reports an 86.06% to 89.30% macro defense success rate while retaining substantial benign-task utility. Controlled experiments find that query-matched references and preserving context-direction correspondence improve risk ranking in the low-false-positive regime. The authors therefore frame VLM safety as constructing applicable evidence for selective risk decisions rather than merely learning a boundary between harmful and benign inputs.