Knowledge Distillation, End to End

This is the full pipeline behind theLAB's ADI distilled-model line: take a small open student, teach it to answer like a much larger frontier teacher, and ship the result as a 2–3 GB GGUF that runs fully local in Ollama. Every GPU-heavy step runs on a single 16 GB card (an RTX 5060 Ti); only the teacher generation reaches out over the network.

The reference build here is adi-qwen3.5-4b-glm5.2-general — a Qwen3.5-4B student distilled from glm-5.2. The same six steps produced the coder line (Qwen2.5-Coder-7B from Kimi K2.7 Code) with only the teacher, dataset, and base swapped.

#01Pick the teacher

The teacher defines the ceiling: the student can only imitate behavior the teacher demonstrates. Match the teacher to the purpose, not just raw size — a general-knowledge student wants a strong generalist (glm-5.2), a coder wants a coding specialist (kimi-k2.7-code). The full decision framework lives in Picking a Teacher Model.

#02Build the seed prompts

You need a diverse set of questions to ask the teacher. Rather than hand-writing them, pull human instructions from an existing instruction set (Dolly-15k for general knowledge), drop anything that needs an attached context passage, dedupe, and keep ~2,000.

from datasets import load_dataset

ds = load_dataset("databricks/databricks-dolly-15k", split="train")
skip = {"closed_qa", "information_extraction", "summarization"}

seen, prompts = set(), []
for row in ds:
    if row.get("category") in skip:
        continue
    q = row["instruction"].strip()
    if not q or len(q) < 15 or len(q) > 400 or q.lower() in seen:
        continue
    seen.add(q.lower()); prompts.append(q)
    if len(prompts) >= 2000:
        break

#03Generate the dataset from the teacher

Send each prompt to the teacher through Ollama's native /api/chat endpoint with thinking disabled — a small student should imitate clean final answers, not chain-of-thought it's too small to reproduce. Write the result as chat-format JSONL.

payload = {
    "model": "glm-5.2",
    "messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user",   "content": prompt},
    ],
    "think": False,          # native API reliably disables thinking
    "stream": False,
    "options": {"num_predict": 2048, "temperature": 0.7},
}
r = requests.post("http://localhost:11434/api/chat", json=payload, timeout=300)
answer = r.json()["message"]["content"].strip()

#04Fine-tune the student (LoRA)

Train a small rank-16 LoRA adapter on the distilled pairs with Unsloth; the base stays frozen, so only a fraction of a percent of parameters move. Whether you load the base in bf16 or 4-bit depends on the architecture — covered in depth in bf16 LoRA vs 4-bit QLoRA. Qwen3.5 needs bf16 (its delta-net/Mamba layers quantize poorly during training); a dense base like Qwen2.5-Coder is happy with 4-bit QLoRA.

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name     = "unsloth/Qwen3.5-4B",
    max_seq_length = 4096,
    load_in_4bit   = False,   # bf16 — NOT 4-bit for Qwen3.5
    load_in_16bit  = True,
)
model = FastModel.get_peft_model(
    model, r=16, lora_alpha=16, lora_dropout=0,
    use_gradient_checkpointing="unsloth",
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)
# 3 epochs · lr 2e-4 · 777 steps · final loss 0.93

#05Merge, convert, and quantize to GGUF

Merge the adapter into the base as 16-bit safetensors, convert to GGUF with llama.cpp's own converter, then quantize to q4_k_m. The quant-tier trade-offs are in GGUF Quantization Tiers Compared and the conversion gotchas in Converting HF Models to GGUF.

# merge adapter → 16-bit safetensors (Unsloth), then:
python3 llama.cpp/convert_hf_to_gguf.py ./merged \
    --outfile adi-qwen3.5-4b-glm5.2-general-bf16.gguf --outtype bf16

# quantize bf16 → q4_k_m (9 GB → 2.7 GB)
./llama.cpp/build/bin/llama-quantize \
    adi-qwen3.5-4b-glm5.2-general-bf16.gguf \
    adi-qwen3.5-4b-glm5.2-general-q4_k_m.gguf q4_k_m

#06Serve from Ollama

A Modelfile points Ollama at the quantized GGUF. After ollama create, the model is live in the fleet alongside every other ADI model, reachable through the standard OpenAI-compatible endpoint.

# Modelfile
FROM ./adi-qwen3.5-4b-glm5.2-general-q4_k_m.gguf
PARAMETER temperature 0.7
PARAMETER num_ctx 4096
SYSTEM """You are ADI, a precise and knowledgeable assistant."""

ollama create adi-qwen3.5-4b-glm5.2-general -f Modelfile
ollama run  adi-qwen3.5-4b-glm5.2-general "Explain the CAP theorem in two sentences."

#Verify it took

• The model shows up in ollama list at the expected size (2.7 GB for a 4B q4_k_m).
• Smoke-test prompts return coherent, well-structured answers — not repeated tokens or gibberish.
• Answers visibly adopt the teacher's structure on familiar topics — that's the distillation showing.

Build a small held-out set and score it properly in Evaluating a Fine-Tuned Model.