Pooling VRAM Across Two GPUs for a 32B Model

A single 16 GB card can't hold a 32B model at Q4 — the weights alone are ~18–20 GB before you reserve anything for the KV cache. On your-llm-host we sidestepped the cost of one big card by pooling two consumer cards into a single 32 GB compute target and letting Ollama split the model's layers across both.

The build is 2× ASUS DUAL-RTX5060TI (16 GB each), an ASRock X570 Taichi board, Ubuntu 24.04, and CUDA 13.0. The payoff: Qwen3 32B at Q4 runs fully local with room left over for context.

#01Get the Blackwell cards enumerating

The 5060 Ti is Blackwell (sm_120), and that detail bites first. The proprietary legacy driver module won't enumerate these cards properly — you need the NVIDIA open-kernel module. Install that variant before anything else, or nvidia-smi will come up empty or partial.

# open-kernel module — required for sm_120 / Blackwell
sudo apt install nvidia-driver-580-open
sudo reboot

# both cards should appear after reboot
nvidia-smi --query-gpu=index,name,memory.total --format=csv

#02Set BIOS for two cards on real lanes

Two cards in two slots only get full bandwidth if the board splits its lanes. In the X570 Taichi BIOS, enable PCIe slot bifurcation so the slots run x8/x8 instead of x16/x0 (the default that starves the second slot). While you're in there, turn on Resizable BAR (ReBAR) — it lets the CPU address the full framebuffer, which helps large-model loads.

# confirm each card actually negotiated x8
nvidia-smi --query-gpu=index,pcie.link.width.current --format=csv

# or check the link width from lspci
lspci -vv | grep -A3 NVIDIA | grep LnkSta

If a card reports x1 or x4, recheck the slot population and bifurcation setting before going further — a starved lane shows up later as ugly cross-GPU latency.

#03Let Ollama split the layers

This is the part that needs no special work. Ollama automatically offloads a model's layers across every visible GPU. There is no NVLink in this box and none is required — the cards talk over PCIe. You do not shard the model by hand; you just make both cards visible and pull the model.

# both GPUs visible to Ollama
export CUDA_VISIBLE_DEVICES=0,1

# optional — spread layers evenly across cards
export OLLAMA_SCHED_SPREAD=1

ollama pull qwen3:32b          # Q4 by default, ~18–20 GB
ollama run  qwen3:32b "Summarize the CAP theorem."

Use CUDA_VISIBLE_DEVICES to pin which cards a process may touch — handy if one GPU is busy serving something else. With OLLAMA_SCHED_SPREAD set, the scheduler spreads layers across the visible cards; left unset, the default layer split still uses both.

#04Confirm both cards are actually working

A model can technically "fit" while quietly living on one card. The proof that pooling worked is both GPUs showing memory and utilization during inference. Watch them live while a prompt runs:

watch -n1 nvidia-smi

You want to see ~9–10 GB resident on each card (the layer split), and util ticking up on both during generation. If only GPU 0 lights up, the model fit on a single card or the second card isn't visible — recheck steps 01 and 03.

#05Mind power and thermals

Two cards drawing together stresses the PSU and the case airflow, not just the lanes. Keep an eye on per-card power and temperature under sustained load — the same nvidia-smi watch shows it. Adequate cooling matters more in a two-card box than people expect, because the cards sit close and heat each other.

#Verify it took

• Both cards appear in nvidia-smi at 16 GB each, on x8 links.
• qwen3:32b loads and generates coherent output — not OOM, not gibberish.
• During inference, memory and util are visible on both GPUs, not just one.