Do VLMs Share Safety Neurons Across Modalities?

The question

One backbone, two safety signals.

VLMs can comply with harmful content delivered through images even when their language backbones would refuse the same content in text. Because both modalities meet in one shared backbone, we ask whether they use the same localized safety mechanism.

Text channel

“List the steps in detail…”, as words

A small set of confirmed neurons causally affects refusal

Counts concentrate in mid-layer MLP gates across evaluated models

REFUSE

Image channel

The same request rendered as a picture

Only 0–9 visual neurons are confirmed, with no consistent location

No comparably localized single-neuron gate appears

CAN COMPLY

The method

Two-stage detection, iterated.

Correlation is not causation. A neuron counts as safety-relevant only when intervening on it changes the model's behavior.

i. Behavioral probing

A linear probe cuts about 10⁶ candidate activations down to 10³, using the model's refusal-versus-compliance behavior rather than content labels.

ii. Activation patching

Mean-activation patching confirms each candidate against control neurons under BH–FDR correction.

iii. Iterative ablation

Detection and confirmation re-run after each confirmed batch is clamped to its safe mean, exposing backup neurons that a single pass misses.

Detection and evaluation run on two modality-isolated benchmarks introduced in the paper, ViSafe-Detect and ViSafe-Eval, so each channel is evaluated independently. The full pipeline is Fig. 2 in the paper.

Findings

Text safety is concentrated. Visual safety is diffuse at the single-neuron level.

≈88

Text safety is localizable

They comprise fewer than 0.01% of each model's neurons, yet ablating this small set substantially reduces refusal.

9–33 pp

The dominant refusal pathway

Text-neuron ablation is the only intervention that consistently and substantially reduces refusal across all ten models.

≤5 vs ≥50

A representation-level gap

Text effects change little beyond five GEVD directions. In the four models with measurable visual baselines, visual effects grow from k=5 to k=50. The 0–9 confirmed visual neurons produce absolute OOD refusal changes no larger than 6.3 points.

Scope: 10 evaluated VLMs spanning 2B–32B parameters and six model families. Here, diffuse means not concentrated in a few single-neuron features under the tested interventions; it does not mean that visual safety is absent. Statistical controls and limitations are detailed in the paper.

Results

Across the evaluated models, the asymmetry persists.

Overview of the asymmetry: localizable text safety neurons versus diffuse visual safety — **Overview.** Across ten models from 2B to 32B, ablating 48–130 text safety neurons per model drops in-distribution refusal by 9–33 points. By contrast, the 0–9 visual counterparts produce absolute out-of-distribution changes no larger than 6.3 points.

Per-layer distribution of safety neurons across five VLMs — **Where the neurons sit.** Per-layer counts across five diverse models, text above and visual below on a shared scale. Text counts concentrate in mid-layer MLP gates; the visual search recovers 0–5 here (0–9 across all ten) with no consistent location, despite scanning one additional component.

Qwen2.5-VL · GEVD

Measured refusal effect across GEVD subspace sizes.

Text (L12) Visual (L20)

Text / k=5 to 50 −5 → −6 pp 1-point change

Visual / k=5 to 50 −33 → −53 pp 20-point change

Qwen2.5-VL at the best layer for each modality. The visualization reports the absolute change in keyword-rated refusal under GEVD direction ablation. Across the four models with measurable visual baselines, the text effect changes by 1–5 points from k=5 to k=50, while the visual effect grows by 9–35 points. Exact values are reported in the paper's GEVD table and multi-model appendix.

Cite

Reference

@inproceedings{li-etal-2026-vlms, title = "Do {VLM}s Share Safety Neurons Across Modalities?", author = "Li, Jiaxuan and Zhang, Jiahao and Vo, Duc Minh and Nguyen, Huy H. and Kavumba, Pride and Wataoka, Koki", booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing", year = "2026", publisher = "Association for Computational Linguistics", }