Evaluating LLMs with Local Benchmarks

Picking a local model on "vibes" fails the moment you change quant or prompt. Evaluation replaces gut feel with numbers, so you can compare models, quant tiers, and prompts objectively. Complements Evaluating a Fine-Tuned Model.

#Two kinds of eval

• Benchmarks — standardized datasets (MMLU, GSM8K, HumanEval) scored automatically. Good for capability comparison, weak on your task.
• Task evals — a hand-built set of your prompts with expected answers or a rubric. The only thing that predicts real-world performance.

#lm-evaluation-harness

pip install lm-eval
lm_eval --model local-completions \
  --model_args base_url=http://localhost:11434/v1 \
  --tasks gsm8k,mmlu --limit 100

#LLM-as-judge

For open-ended outputs, use a strong model to score responses against a rubric — fast and surprisingly consistent, but watch for bias (it favors verbose, its-own-style answers).

Rule: never ship a model swap or quant change without re-running your eval set.

Related: Evaluating a Fine-Tuned Model · GGUF Quantization Tiers Compared · Temperature & Sampling: Determinism vs Creativity