Retrieval-Augmented Generation (RAG) lets a local LLM answer from your documents without fine-tuning. You embed your text into vectors, store them in a vector database, and at query time retrieve the most relevant chunks and stuff them into the prompt as context.
The pipeline
docs → chunk → embed → store (vector DB)
query → embed → similarity search → top-k chunks → prompt + chunks → LLM → answerLocal embeddings with Ollama
ollama pull nomic-embed-text
curl http://localhost:11434/api/embeddings -d '{"model":"nomic-embed-text","prompt":"your text"}'Why RAG over fine-tuning
| RAG | Fine-tuning |
|---|---|
| Updates instantly (add a doc) | Re-train to update |
| Cites sources | Bakes knowledge in opaquely |
| Cheap, no GPU training | Expensive |
Gotchas
- Chunking matters most — too big and retrieval is noisy, too small and you lose context. Start ~500 tokens with overlap.
- Watch the context budget — retrieved chunks compete with the answer for
num_ctx(Ollama Parameters Guide). - Use a reranker to reorder top-k for precision.
Related: Vector Databases for Local AI · Agentic Tool-Use Loops · Ollama Parameters Guide