A multimodal model handles more than one kind of input — text and images, audio, or video — in a single system. It's how an assistant can "look at" a photo and discuss it.
How modalities meet
The key idea: project everything into a shared embedding space where a picture of a dog and the word "dog" land near each other.
image → vision encoder ─┐
├─→ shared space → transformer → text out
text → token embeds ───┘A vision encoder (often a CNN or ViT) turns an image into tokens the transformer processes alongside text.
What it unlocks
| Capability | Example |
|---|---|
| Visual Q&A | "What's wrong with this diagram?" |
| OCR + reasoning | Read a receipt, total it |
| Grounded assistants | "What's in this photo?" |
| Image/video generation | Text → image (Diffusion Models Explained) |
Multimodality is also a partial answer to the symbol grounding problem: when "apple" connects to actual pixels of apples, the symbol is grounded in something beyond more words.
Related: How Image Recognition Works (CNNs) · What Are Embeddings · Symbol Grounding and Embodiment