Jetson Orin as an Ollama Host

A Jetson Orin makes a capable little Ollama node, but the unified-memory architecture and JetPack's CUDA stack introduce failure modes you won't see on a discrete-GPU box.

#The GPU-discovery OOM

On your-edge-device (a Jetson running Ollama), GPU discovery can OOM the board. Ollama probes the GPU on model load; on a memory-constrained Jetson that probe — combined with the model's own footprint — can exhaust unified memory and get the process killed.

When this happens, local models fall back to CPU instead of crashing outright, so the symptom is "it works but it's slow," not a hard error. Watch for it.

#Diagnosing

# Is Ollama actually using the GPU?
journalctl -u ollama -f          # look for "no compatible GPUs" / CUDA OOM
tegrastats                       # live RAM + GPU usage on Jetson
ollama ps                        # PROCESSOR column shows CPU vs GPU per model

If ollama ps shows 100% CPU for a model you expect on GPU, you've hit the fallback.

#Mitigations

• Cap concurrency. Set OLLAMA_NUM_PARALLEL=1 and OLLAMA_MAX_LOADED_MODELS=1 so the board only ever holds one model.
• Stay small. Keep models in a quant tier the board's unified RAM can hold with headroom for the CUDA context — see GGUF Quantization Tiers Compared.
• Add swap carefully. Swap prevents OOM kills but does not make GPU inference viable — see Linux Swap Memory Guide. It buys stability, not speed.
• Pre-pull and warm the model once at boot so the first user request doesn't pay the discovery cost.

Rule: on the Jetson, treat CPU fallback as a bug to investigate, not a feature. If a model won't fit on the GPU, pick a smaller quant rather than living on the CPU path.

See also: Ollama Cloud vs Local: Model Routing on the Fleet.