bf16 LoRA vs 4-bit QLoRA

When we fine-tune a model at theLAB, we almost always reach for LoRA: train a small low-rank adapter while the base model stays frozen, so only a fraction of a percent of the parameters ever move. The open question isn't whether to use LoRA — it's how to hold the frozen base in memory while we train. There are two choices, and picking the wrong one quietly costs you accuracy.

4-bit QLoRA keeps the frozen base quantized to 4-bit (NF4) and trains the adapter on top. It saves a lot of VRAM, but it introduces quantization error into the base during training — the gradients flowing into the adapter are computed against a lossy approximation of the real weights. bf16 LoRA keeps the base in full 16-bit. It costs more VRAM, but there's no quant error: cleaner weights, cleaner gradients.

#Dense transformers tolerate 4-bit well

Standard dense transformers — Qwen2.5 and Qwen2.5-Coder, Llama, Gemma — survive 4-bit quantization with minimal accuracy loss during training. The NF4 representation is a good enough stand-in for their linear layers that the adapter still learns what it needs to. For these bases, QLoRA is the default: take the VRAM savings, lose almost nothing.

from unsloth import FastModel

# Dense base (Qwen2.5-Coder-7B) — 4-bit QLoRA
model, tokenizer = FastModel.from_pretrained(
    model_name     = "unsloth/Qwen2.5-Coder-7B",
    max_seq_length = 4096,
    load_in_4bit   = True,    # QLoRA: frozen base in NF4
)

#Hybrid / SSM architectures do not

Architectures with delta-net, Mamba-hybrid, or other SSM layers — Qwen3.5 is the one we hit — quantize poorly during training. Those layers are sensitive in a way that ordinary attention/MLP stacks aren't, and 4-bit costs real, measurable accuracy. Here the VRAM savings are a trap. We keep the base in bf16 and accept the higher memory bill.

from unsloth import FastModel

# Hybrid base (Qwen3.5-4B) — bf16 LoRA, NOT 4-bit
model, tokenizer = FastModel.from_pretrained(
    model_name     = "unsloth/Qwen3.5-4B",
    max_seq_length = 4096,
    load_in_4bit   = False,   # no quant error in the frozen base
    load_in_16bit  = True,
)

Either way, the adapter config is identical — rank 16, alpha 16, no dropout, all linear projections targeted, with Unsloth's gradient checkpointing to keep the activation memory down:

model = FastModel.get_peft_model(
    model, r=16, lora_alpha=16, lora_dropout=0,
    use_gradient_checkpointing="unsloth",
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)

#What it costs in VRAM

The reason this is a real decision and not a free lunch: bf16 roughly doubles the memory the frozen base occupies. On a 16 GB card the trade-offs land like this — a 7B at 4-bit QLoRA sits around 6–8 GB (comfortable), a 4B at bf16 LoRA around 10 GB (comfortable), but a 7B at bf16 LoRA pushes to roughly 16 GB — tight enough that you're fighting OOM on every long batch.

#The decision, at a glance

#The theLAB lesson

We learned this the practical way running the same distillation pipeline twice. adi-qwen3.5-4b-glm5.2-general used bf16 LoRA because Qwen3.5 carries delta-net layers; adi-qwen2.5-coder-7b used 4-bit QLoRA because it's a dense base. Same scripts, same hyperparameters, same hardware — the architecture alone flipped the load flag. When you're setting up a new fine-tune, identify the base's architecture before you pick the quantization, not after the loss looks suspiciously flat.