Local LLM performance includes responsiveness, throughput, memory use, and behavior under concurrent load. A single tokens-per-second number cannot describe all of them.
This article measures serving performance. Evaluating LLMs with Local Benchmarks covers answer quality; a fast configuration that fails the task is not an improvement.
Measure the stages separately
| Metric | Meaning |
|---|---|
| Time to first token (TTFT) | Client-observed delay until the first generated token |
| Prompt processing rate | Input tokens processed per second |
| Generation rate | Output tokens generated per second |
| End-to-end latency | Time from request start to completed response |
| Aggregate throughput | Work completed across simultaneous requests |
| Queue time | Time waiting before processing begins |
Client TTFT can include network transit, queuing, model loading, and prompt processing. A runtime's internal timing fields usually do not include every one of those stages. Record both when diagnosing the user experience.
Make a reproducible workload
Record the model identifier and digest, quantization, runtime version, hardware, context setting, sampling parameters, prompt, and expected output length. Keep these constant when comparing one tuning change.
Use a short interactive prompt, a long-context prompt, and a representative task with tools or structured output. Token counts differ between tokenizers, so comparisons across models need care.
Run cold-start tests separately from warm tests. Warming a model before measuring every request hides startup latency that users may actually encounter. Likewise, prefix caching can change repeated-prompt results; note whether it is present.
Inspect runtime timings
Ollama's non-streaming generation response includes counters and timing fields that can help separate prompt evaluation from generation:
curl --fail-with-body http://localhost:11434/api/generate \
-H 'Content-Type: application/json' \
-d '{"model":"YOUR_INSTALLED_MODEL","prompt":"Explain a database index in one paragraph.","stream":false,"options":{"num_predict":128,"temperature":0}}'For timing fields documented in nanoseconds:
generation tokens/s = eval_count / (eval_duration / 1,000,000,000)That calculation is not client-observed TTFT. Measure streaming arrival times with a monotonic clock when first-token latency matters. Handle missing fields and zero durations rather than fabricating a rate.
Test realistic concurrency
Start with one request, then increase concurrency gradually. Record per-request latency, completion rate, errors, memory, and total throughput. Stop if the machine becomes unstable or the queue grows without bound.
A higher aggregate throughput can come with worse individual latency. Report medians and tail latency such as p95, together with the sample count. A p95 calculated from only a handful of requests is not a reliable service target.
Read memory and utilization correctly
Observe CPU, system RAM, swap, GPU memory, GPU utilization, and temperatures. More context and simultaneous sequences increase KV-cache requirements. Quantizing model weights does not make those costs disappear.
An out-of-memory failure, CPU offload, thermal throttling, or model eviction can explain large timing changes. Collect the evidence before increasing concurrency or context limits.