Qwen3.5 GGUF to Ollama Conversion Gotchas

Converting a Qwen3.5 checkpoint to GGUF and loading it in Ollama has two specific traps that produce confusing failures. Both are easy once you know them.

#1. Tokenizer needs transformers 5.5.0

The conversion step reads the model's tokenizer through transformers. Qwen3.5's tokenizer requires transformers==5.5.0 — older versions fail to load it (unknown tokenizer class / missing config), which aborts the GGUF export before it starts.

pip install "transformers==5.5.0"
python llama.cpp/convert_hf_to_gguf.py /path/to/qwen3.5 --outfile qwen3.5.gguf

#2. Strip the MTP head or Ollama's loader fails

Qwen3.5 ships a Multi-Token-Prediction (MTP) head. If you leave it in the GGUF, Ollama's qwen3next loader chokes on the unexpected tensors and refuses to load the model. Strip it during conversion:

python convert_hf_to_gguf.py /path/to/qwen3.5 --no-mtp --outfile qwen3.5.gguf

Symptom → cause: if Ollama errors on load with a tensor/architecture mismatch for a qwen3next model, you forgot --no-mtp. Re-export with the flag — you do not need to re-download the checkpoint.

#Then quantize and import

# optional: quantize down from the f16 export
llama.cpp/llama-quantize qwen3.5.gguf qwen3.5-q4_K_M.gguf Q4_K_M

Modelfile:

FROM ./qwen3.5-q4_K_M.gguf
PARAMETER num_ctx 8192

ollama create qwen3.5 -f Modelfile
ollama run qwen3.5 "test"

For quant tier trade-offs see GGUF Quantization Tiers Compared and Converting HF Models to GGUF with llama.cpp.