Unsloth is our default harness for fine-tuning on a single card. It's a drop-in replacement for the usual Hugging Face load-and-train path — roughly 2× faster and using far less VRAM — exposed through FastModel / FastLanguageModel, and it pairs cleanly with TRL's SFTTrainer. The whole loop is three moves: load a base, attach a LoRA adapter, train.
The one decision that actually matters is how you load the base — bf16 or 4-bit — and that depends entirely on the model's architecture. Get it wrong on Qwen3.5 and the run technically completes but the student comes out worse than the base. This article is the practitioner's version of that lesson.
01Load the base model
Pick the precision deliberately. load_in_4bit=True gives you QLoRA — the base weights live in 4-bit, which slashes VRAM and is the right call for standard dense transformers. load_in_16bit=True keeps the base in bf16, which costs more memory but is mandatory for some newer architectures (next section).
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
model_name = "unsloth/Qwen3.5-4B",
max_seq_length = 4096,
load_in_4bit = False, # 4-bit QLoRA — off for Qwen3.5
load_in_16bit = True, # bf16 LoRA — required here
)02Attach the LoRA adapter
The base stays frozen; we train a small rank-16 adapter over the attention and MLP projections. r and lora_alpha at 16 is our standard starting point, dropout at 0, and Unsloth's own gradient-checkpointing mode to keep activation memory down.
model = FastModel.get_peft_model(
model,
r = 16,
lora_alpha = 16,
lora_dropout = 0,
use_gradient_checkpointing = "unsloth",
target_modules = ["q_proj","k_proj","v_proj","o_proj",
"gate_proj","up_proj","down_proj"],
)03Train with TRL's SFTTrainer
Format the dataset with the model's own chat template, then hand it to SFTTrainer. Our defaults — 3 epochs, lr 2e-4, rank 16 — carry most jobs. For reference, theLAB's Qwen3.5-4B run came out to 777 steps in ~2h53m with a final loss of ~0.93 on a single 16 GB card.
from trl import SFTTrainer, SFTConfig
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset, # chat template already applied
args = SFTConfig(
num_train_epochs = 3,
learning_rate = 2e-4,
per_device_train_batch_size = 2,
max_seq_length = 4096,
),
)
trainer.train()load_in_4bit=False, load_in_16bit=True); 4-bit measurably hurts it. Qwen2.5-Coder is a plain dense transformer — 4-bit QLoRA (load_in_4bit=True) is fine and saves VRAM. The full reasoning is in bf16 LoRA vs 4-bit QLoRA.
transformers >= 5.2.0 is required just to recognize it — but Unsloth caps at transformers <= 5.5.0. The only version satisfying both is exactly transformers 5.5.0. Also pin numpy < 2.3 (numba breaks above that), and for qwen3next-style loaders strip the multi-token-prediction head (e.g. --no-mtp) or the loader fails outright. Standard dense bases — Qwen2.5-Coder, Llama, Gemma — need none of these pins.
04Export a merged checkpoint for GGUF
When you're ready to ship, have Unsloth merge the adapter back into the base and write a 16-bit checkpoint. That merged checkpoint is exactly what llama.cpp's converter consumes downstream.
model.save_pretrained_merged(
"merged",
tokenizer,
save_method = "merged_16bit", # feeds llama.cpp conversion
)