The Free Encyclopedia
Log in
Article History

Evaluating a Fine-Tuned Model

Revision as of Jun 25, 2026 23:03 by migration.

Once a distilled student is live in Ollama, the question is no longer "did it train?" but "is it actually better, and better at the thing we wanted?" At theLAB we answer that with a fixed eval set and four escalating levels of scrutiny — from a 30-second smoke test to an auto-graded benchmark.

The single most important move comes before training: build a fixed held-out eval set of 30–50 representative prompts and set it aside. It must not overlap with the training data. Without it you can't compare the base against the fine-tune fairly, and you'll be tempted to eval on the training set — which only ever tells you the model memorized.

Hold it out, freeze it early The eval set is a contract you sign before you see results. Lock those 30–50 prompts before the first training run so the base and every later iteration are judged on the exact same questions. Never eval only on training data.

01Smoke tests — does it even work?

The cheapest level catches broken conversions. The model should appear in ollama list at the expected size, load, and run without crashing. Then read a few outputs: they should be coherent and non-repeating — no gibberish, no token loops, no degenerate "the the the." A bad GGUF conversion or a botched merge usually shows up here as garbage on the very first prompt.

ollama run adi-qwen3.5-4b-glm5.2-general \
    "Explain the CAP theorem in two sentences."
# looking for: coherent, on-topic, ends cleanly — not looping

02Side-by-side on the held-out set

Now run the base and the fine-tuned model on the same held-out prompts and read the diffs. This is where you confirm the intended behavior actually transferred — better structure, the house style, tighter instruction-following. If base and tuned answers look identical, the fine-tune didn't take; if the tuned one is just longer but no more correct, that's a warning sign (see below).

03LLM-as-judge

Reading every diff by hand doesn't scale. Hand the pair to a strong judge model — we use glm-5.2 — and score each answer against a rubric: correctness, structure, style adherence, instruction-following. It's cheaper and far faster than human grading and good enough to rank models. Average the scores, then go read the losers: the low-scoring answers are where the real signal is.

import requests, json

def chat(model, prompt):
    r = requests.post("http://localhost:11434/api/chat", json={
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": False,
    }, timeout=300)
    return r.json()["message"]["content"]

RUBRIC = """Score the answer 1-5 on correctness, structure,
style, and instruction-following. Reply ONLY as JSON:
{"correctness":n,"structure":n,"style":n,"following":n}"""

for p in held_out:                          # the frozen 30–50 prompts
    base  = chat("qwen3.5:4b", p)           # the unmodified base
    tuned = chat("adi-qwen3.5-4b-glm5.2-general", p)
    judge = chat("glm-5.2",
        f"{RUBRIC}\n\nPROMPT:\n{p}\n\nANSWER:\n{tuned}")
    score = json.loads(judge)               # average these across the set

04Task benchmarks

For models with a checkable output, stop trusting opinions and run the work. For coders, actually execute the generated code against tests or a runner and measure pass@1 — the code either passes or it doesn't. For general models, keep a small QA set with known answers you can auto-grade by string or numeric match. This is the only level that's immune to a confident-but-wrong judge.

05What to watch for

Three failure modes hide behind nice-looking averages:

  • Overfitting / parroting — the model echoes training phrasing and sounds smarter without being more correct.
  • Catastrophic forgetting — a regression on general ability the base model used to have. Your held-out set should include a few off-topic prompts to catch this.
  • Format drift — the style mutates in ways you didn't intend (stray markdown, dropped sections).
Don't trust loss alone Track training loss, but never ship on it. A low final loss can simply mean the model memorized the dataset — it looks great on training data and parrots phrasing on the eval set. Loss tells you training converged; only the held-out evals tell you it learned the right thing.

06The practical loop

Putting it together: fixed eval set → generate base + tuned answers → judge with glm-5.2 → human spot-check the disagreements (where judge and your gut differ) → decide ship or iterate. The spot-check is small because the judge already narrowed it down.

One tie-in worth weighting your rubric around: distillation mainly moves style and reasoning, not net-new facts. So weight correctness toward reasoning on familiar tasks and structure, not obscure factual recall — grading a 4B student on trivia punishes it for something fine-tuning was never going to fix. See Does distillation add knowledge or just style? for why.