A Vision Language Model (VLM) is a multimodal neural architecture that jointly processes visual inputs (images or video) and natural language text, learning shared representations that enable cross-modal tasks such as image captioning, visual question answering, visual grounding, and instruction-following based on image context. VLMs typically pair a visual encoder with a large language model backbone connected by a learned projection mechanism.
Content
- Vision-language alignment research began with CLIP (OpenAI, 2021), which trained dual image and text encoders via contrastive learning on 400 million image-text pairs scraped from the internet, producing a shared embedding space where semantically related images and texts are proximate. ALIGN (Google, 2021) scaled this approach to 1.8 billion pairs. The crucial architectural innovation came with Flamingo (DeepMind, 2022), which inserted cross-attention layers into a frozen language model to condition text generation on visual tokens — demonstrating few-shot visual question answering. LLaVA (Liu et al., 2023) democratised VLM construction by showing that a simple linear projection connecting CLIP image embeddings to LLaMA could be trained effectively on GPT-4-generated visual instruction data.
- A VLM’s architecture divides into three stages: visual encoding, vision-language alignment, and language generation. The visual encoder (typically ViT-L/14 or a variant) processes input images as patch embeddings capturing spatial and semantic features. A projection module — ranging from a simple linear layer to a more complex Q-Former (BLIP-2) or MLP adapter — maps visual tokens into the language model’s embedding space. The language model decoder then generates text conditioned on both visual tokens and the text prompt. During training, the visual encoder and language model may be frozen (efficient) or jointly fine-tuned (higher performance). Instruction tuning datasets containing diverse visual question-answer pairs teach the model to follow varied task specifications.
- VLMs have practical applications across numerous domains. In medical imaging, VLMs such as Med-PaLM M and LLaVA-Med generate radiology report drafts from X-rays and CT scans, assisting radiologists. In document understanding, models like Donut and GOT-OCR parse complex layouts including tables, charts, and multi-column text. Robotics systems use VLMs as perception modules to convert raw images into task-relevant natural language descriptions that guide manipulation policies. Content moderation platforms use VLMs to identify policy-violating imagery across multiple languages simultaneously. Autonomous driving systems use VLMs to generate natural language scene descriptions that improve scene understanding under distribution shift.
- In 2024–2025, VLMs have achieved GPT-4V-level performance in open-source releases (LLaVA-1.6, Qwen-VL-Max, InternVL2), democratising multimodal capability. Architectural innovation has focused on native high-resolution image processing (dynamic tiling, pixel shuffle) to preserve fine-grained detail for document and chart understanding. Video understanding capabilities have extended VLMs to temporal reasoning across frame sequences (Video-LLaVA, Qwen2-VL). Models such as GPT-4o and Gemini 1.5 Pro handle interleaved text-image-audio inputs in a single context window. The Cambrian benchmark suite has systematically characterised capability gaps in spatial reasoning and counting, driving targeted improvement in next-generation VLM architectures.