Three popular ways to run LLMs locally, each tuned for a different priority.
| Engine | Strength | Use when |
|---|---|---|
| Ollama | Simplest UX, model registry, great DX | Personal use, dev, the fleet default |
| llama.cpp | Runs anywhere (CPU, Metal, modest GPUs), GGUF | Edge devices, Macs, Jetson, quantized models |
| vLLM | Highest throughput, continuous batching, OpenAI API | Serving many concurrent users on a real GPU |
The throughput difference
vLLM's PagedAttention manages the KV cache like virtual memory, enabling continuous batching — new requests slot into the batch as others finish. For concurrent load it can be many times faster than one-request-at-a-time engines.
# vLLM — OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-8B-InstructRule of thumb
- One user, convenience → Ollama (which wraps llama.cpp).
- CPU / tiny GPU / Mac → llama.cpp directly.
- Many users, one big GPU, max tokens/sec → vLLM.
For multi-model GPU layout see Serving Multiple Models on a Dual-GPU Box.
Related: Serving Multiple Models on a Dual-GPU Box · GGUF Quantization Tiers Compared · Ollama Parameters Guide