Picking a local model comes down to two questions: what fits your hardware, and what's it for. Here's how to decide without endless trial and error.
Step 1 — what fits (VRAM)
A 4-bit quantized model needs roughly (billions of params × ~0.6 GB) of VRAM, plus context overhead.
| VRAM | Comfortable model size (Q4) |
|---|---|
| 8 GB | up to ~8B |
| 12–16 GB | ~13B (or a small MoE) |
| 24 GB | ~32B, or 70B split/offloaded |
| CPU only | small models, slowly |
Drop a tier in quantization to fit a bigger model — usually worth it.
Step 2 — match the task
| Task | Look for |
|---|---|
| Coding | A coder-tuned model |
| General chat / reasoning | A strong instruct model |
| Tool use / agents | One trained for function calling |
| Embeddings / RAG | A dedicated embedding model |
| Speed over smarts | A small model or an MoE |
Step 3 — verify, don't vibe
Don't trust leaderboards alone — they get gamed. Run the model against your own prompts (Evaluating LLMs with Local Benchmarks) before committing.
Then pull it and tune the knobs (Ollama Parameters Guide, Temperature and Sampling Explained).
Related: GGUF Quantization Tiers Compared · Evaluating LLMs with Local Benchmarks · vLLM vs Ollama vs llama.cpp