vLLM Vision Placeholder Token DoS Payload

How the attack works

vLLM's OpenAI-compatible endpoint accepts requests using Qwen2.5-VL style placeholder tokens like <|vision_start|>, <|image_pad|>, <|video_pad|>, and <|vision_end|> to mark where media content belongs in a prompt. A legitimate multimodal request always pairs these tokens with actual image or video data. An attacker sends a request with the placeholder tokens but omits the media, causing vLLM's internal position-computation code to look up an empty array and throw an unhandled error, which kills the worker process. The rule watches for this exact mismatch — placeholder tokens present, media absent — whether sent as a direct HTTP request or built via a command-line tool like curl, wget, or a Python script.

Netzilo detection

Why it matters

A single crafted request can crash a model-serving worker, causing denial of service for everyone relying on that vLLM instance until it is restarted or patched.

What you can do

• →Reject or sanitize inbound requests to your vLLM endpoint that contain vision/video placeholder tokens without an accompanying media payload.
• →Upgrade vLLM to a version that returns a clean error instead of crashing on this malformed input, and confirm the fix with a regression test rather than relying on this rule alone.
• →Run model workers with automatic restart/supervision so a crash from this input does not cause extended downtime.
• →Review any application code that renders chat templates client-side, since it may unintentionally produce the same malformed pattern and should attach media in a field your monitoring actually inspects.

Known benign look-alikes

• Client-side chat-template rendering where the application expands the Qwen2.5-VL template into a flat prompt string and attaches the image through a field this rule does not enumerate; the request is malformed in the same way but the intent is benign (and the same request will still crash an unpatched worker).
• A user or developer asking a model to explain vLLM tokenizer internals and quoting the literal placeholder tokens in the question. Advisory and tokenizer-config wording is filtered, but a bare "what does <|image_pad|> do" prompt will still match.
• CI and regression suites that deliberately send placeholder-without-media prompts to confirm a patched build now raises a clean 400 instead of crashing the worker.
• Prompt-debugging tooling that echoes a fully rendered multimodal template to a completions endpoint with the media stripped for readability.

References

• https://github.com/vllm-project/vllm
• https://atlas.mitre.org/techniques/AML.T0029

Related threats