The Free Encyclopedia
Log in
Article History

Converting HF Models to GGUF with llama.cpp

Revision as of Jun 25, 2026 22:26 by migration. This is an old revision — view current.

You have a merged HF model — safetensors on disk, fresh out of an Unsloth merge — and you want it running in Ollama or raw llama.cpp. That means turning it into a GGUF and quantizing it down to a size your serving host can actually hold. The path is four short commands, but two version traps sit on it that will waste an afternoon if you don't see them coming.

We do this for every model in theLAB's ADI line. The good news: recent llama.cpp already understands modern Qwen — including the Qwen3.5 SSM/Mamba-hybrid with MTP and the Qwen2 family (Qwen2.5-Coder) — so the conversion itself is clean once the versions line up.

01Build llama.cpp

Clone and build with CMake. The CUDA backend is optional but worth it if the box has an NVIDIA card — quantizing and the smoke-test run are both faster.

git clone https://github.com/ggerganov/llama.cpp
cmake -B build
cmake --build build -j                 # add -DGGML_CUDA=ON to the first cmake for GPU

02Convert HF → unquantized GGUF

The converter reads config.json and the tokenizer out of your merged directory and emits a single full-precision GGUF. Use f16 by default, or bf16 to preserve a model that was trained and merged in bf16 (Qwen3.5, for one).

python3 llama.cpp/convert_hf_to_gguf.py ./merged \
    --outfile model-f16.gguf --outtype f16      # or --outtype bf16 to preserve bf16
Trap: the tokenizer backend Running pip install -r llama.cpp/requirements.txt can silently downgrade transformers and break the converter with TokenizersBackend does not exist. Fix: reinstall transformers 5.5.0 after the requirements step — that's the version that reads a recent merged Qwen tokenizer correctly. Do not go above 5.5.0; newer releases break Unsloth on the training side.

03Quantize

Take the full-precision GGUF and squeeze it to a quant tier with llama-quantize. For a general-purpose local model, q4_k_m is the usual choice — see GGUF Quantization Tiers Compared for picking the right tier for your case.

./llama.cpp/build/bin/llama-quantize \
    model-f16.gguf model-q4_k_m.gguf q4_k_m

To set expectations on size: a 7B model lands around ~14 GB at f16 and ~4.4 GB at q4_k_m — roughly a 3× reduction that's the whole reason we quantize.

04Test before importing

Always run the quantized file through llama-cli directly before wiring it into Ollama. This isolates conversion problems from Ollama-side config problems.

./llama.cpp/build/bin/llama-cli \
    -m model-q4_k_m.gguf \
    -p "Explain the CAP theorem in two sentences."
Trap: gibberish on a brand-new architecture A freshly-converted GGUF of a brand-new architecture can load fine and still emit garbled text — if Ollama's bundled llama.cpp runtime on the serving host is older than the converter that built the file. The file isn't broken; the runtime can't read it. Fix: update Ollama on the serving host so its runtime matches the converter version. A clean llama-cli run in step 04 with garbled output only inside Ollama is the tell.
When conversion just works For architectures llama.cpp already supports — recent Qwen (Qwen3.5 hybrid + MTP) and Qwen2 (Qwen2.5-Coder) — there's no patching and no custom converter. Get transformers pinned at 5.5.0 and Ollama current on the host, and all four steps run start to finish without surprises.

Verify it took

  • convert_hf_to_gguf.py finishes without a tokenizer-backend error and writes the f16/bf16 file.
  • The q4_k_m file is roughly a third the size of the full-precision one (~4.4 GB for a 7B).
  • llama-cli returns coherent text — not repeated tokens or gibberish — before you ever touch Ollama.