Running several models on a two-GPU server is mostly a question of where each model's weights land and how many you keep resident at once. This extends the VRAM-pooling notes in Pooling VRAM Across Two GPUs for a 32B Model.
Two strategies
1. Pool both GPUs for one big model
For a single model too large for one card, let Ollama split layers across both GPUs. This is the pooling case — one 32B model spanning 2×24 GB. Throughput is slightly lower than single-card (PCIe hop between layers) but it fits models that otherwise wouldn't.
2. Pin one model per GPU
For serving several models concurrently, pin each to its own card so they don't contend:
# Instance A — GPU 0
CUDA_VISIBLE_DEVICES=0 OLLAMA_HOST=127.0.0.1:11434 ollama serve
# Instance B — GPU 1
CUDA_VISIBLE_DEVICES=1 OLLAMA_HOST=127.0.0.1:11435 ollama serveRoute requests to the port for the model you want. Each card stays dedicated, so latency is predictable.
Keeping models resident
| Variable | Effect |
|---|---|
OLLAMA_MAX_LOADED_MODELS |
How many models stay in VRAM at once |
OLLAMA_KEEP_ALIVE |
How long an idle model lingers before unload (e.g. 30m, -1 = forever) |
OLLAMA_NUM_PARALLEL |
Concurrent requests per model (each adds KV-cache VRAM) |
OLLAMA_SCHED_SPREAD |
Spread one model across all GPUs instead of packing |
Watch the math
Resident VRAM ≈ model weights + (KV cache × num_parallel × context). Set num_ctx and OLLAMA_NUM_PARALLEL deliberately — see Ollama Parameters Guide. Overcommit and Ollama will evict a model mid-conversation or spill to host RAM.
nvidia-smi -l 2 # watch per-GPU memory live
ollama ps # which model is on which processorSee also: Ollama Cloud vs Local: Model Routing on the Fleet.