When a small student starts answering like its frontier teacher, it's tempting to claim it got smarter. This is the skeptical companion to our end-to-end distillation pipeline — the part where we stop admiring the output and actually measure what moved.
The claim under test: does distilling a teacher into a small student add net-new knowledge, or does it just transfer style and reasoning structure? Our honest answer, from running theLAB's ADI models through this exact test: mostly style and reasoning, not facts. Roughly 2,000 distilled examples cannot install an encyclopedia into a 4B model that never had the capacity or coverage for it in the first place.
Split the eval into three buckets
A single aggregate score hides everything interesting. The fix is to partition a held-out eval set by what each prompt actually demands, then watch the three buckets move at very different rates.
| Bucket | What it tests | Expected outcome |
|---|---|---|
| (a) Style / format adherence | Does it answer in the teacher's structure and tone? | Big improvement |
| (b) Reasoning on familiar topics | Step-by-step quality on things the base already half-knew | Clear improvement |
| (c) Factual recall on unseen facts | Niche facts the base genuinely didn't hold | Little to no change |
Method: build the held-out set first
The discipline is to write the eval prompts before training, so they can't drift toward whatever the model happens to do well. Then run base and distilled side by side on the same prompts and score with an LLM-as-judge rubric — correctness, structure, style — backed by human spot checks. glm-5.2 makes a fine judge.
# held-out prompts written BEFORE training, tagged by bucket
EVAL = [
{"bucket": "style", "q": "Explain backpressure in a message queue."},
{"bucket": "reasoning", "q": "Why does adding an index sometimes slow writes?"},
{"bucket": "facts", "q": "What year did the Treaty of Westphalia conclude?"},
]
# run base vs distilled on the SAME prompts
for row in EVAL:
base_ans = ask("Qwen3.5-4B", row["q"])
dist_ans = ask("adi-qwen3.5-4b-glm5.2-general", row["q"])
# judge scores correctness / structure / style, 1-5 each
verdict = judge("glm-5.2", row["q"], base_ans, dist_ans)
record(row["bucket"], verdict)Watch for the false positive
The trap is style leakage masquerading as knowledge. A distilled student often parrots the teacher's training phrasing — it sounds more authoritative, structures the answer beautifully, and is exactly as wrong as the base was. On bucket (c), a fluent confident wrong answer should score no better than a blunt wrong answer.
What this means in practice
The two techniques solve different problems and don't compete. Use distillation to make a small model reason and respond like a big one across its existing competence. Use RAG (retrieval) when you need facts the model doesn't hold. They're complements, not substitutes — distillation shapes how the model answers, retrieval supplies what it answers with.
This is exactly what we saw with the ADI general models: they picked up the teacher's structured-reasoning style on familiar topics and reasoned visibly better there. They did not become fact oracles, and we never expected them to. When a question needs a fact the 4B never learned, that's a retrieval job, not a fine-tuning job.