This study puts adversarial text in a robot’s camera view to hijack a VLM planner doing a sorting task. It defines four attack types (indirect signage, task redefinition, authority impersonation, conflict injection) and runs 5,670 trials on GPT-4o, Gemini 2.5 Flash and Qwen3-VL-32B, with success rates of 27.0%, 29.4% and 5.0%. Prompt-based defences stop 75 to 100% of attacks, two-stage verification 85 to 100%, and masking text in the image 100%, though masking would break tasks that need to read labels.
Relevance
A printed sign is about the cheapest attack on a robot there is, and this gives real numbers for it. Reading for robot security under robust robotics, and for any team whose robots read signs, markers or labels in the field.
Links
- arXiv
- Subfield: Security, Robot learning, Perception