Local AI load balancing distributes inference work among compatible backends. Failover sends eligible new requests elsewhere when a backend cannot serve them. Neither mechanism guarantees that different models will produce equivalent answers.
This article covers routing design, not a claim that every mechanism described is enabled in theLAB. For deciding when to use hosted models, see Ollama Cloud vs Local: Model Routing on the Fleet.
Route by capability before load
A backend is eligible only if it satisfies the request's requirements: model identity, context budget, tool support, structured output, modality, and data policy. Two OpenAI-compatible endpoints can still differ in important behavior.
Request -> authenticate -> apply privacy and capability policy
-> select eligible healthy backend -> stream responseDo not silently send a private local-only request to a cloud provider during an outage. Make any permitted change of model or execution location visible to the caller.
Readiness is more than an open port
| Signal | What it tells you |
|---|---|
| TCP connection | A listener exists |
| API health response | The service can answer a basic request |
| Model inventory | The intended model is available |
| Small representative inference | The actual request path works |
| Queue and memory pressure | Capacity may be constrained |
Use cheap checks frequently and representative inference sparingly. Large health-check prompts can create their own overload. Separate a cold model that needs loading time from an unreachable host, and require a few successful checks before returning a failed backend to service.
Choose a simple scheduling policy
Round-robin works for comparable workers and requests. Least-active-request routing helps when request duration varies, but a single large-context request may consume much more memory than several small ones. Token-aware estimates can improve placement when workload measurements justify the complexity.
Bound queues and concurrency. Reject excess work with a clear retryable error rather than allowing requests to wait indefinitely. Record queue time separately from prompt processing and generation time.
Streaming and retry safety
Set distinct deadlines for connecting, receiving the first response, stalled streaming, and the overall request. Preserve cancellation so closing a browser request does not leave expensive generation running indefinitely.
Before any response bytes are delivered, a bounded retry may be reasonable for an idempotent inference request. Once tokens have reached the client, switching backends cannot transparently continue the same generation. Report the interruption and let the caller start again.
Tool execution is a separate side-effect boundary. A retried agent run must not duplicate a payment, note, email, or deployment. Use operation IDs, server-side deduplication, and explicit confirmation where appropriate.
Observe and exercise failure
Log request ID, selected backend, resolved model, routing reason, queue delay, latency, and error class. Avoid logging private prompts by default. Measure p50 and p95 latency as well as success rates.
Test backend shutdown, missing model, queue saturation, stalled streaming, and cancellation. Confirm that failures do not bypass privacy policy and recovery does not cause a burst of repeated requests.