https://huggingface.co/havenoammo/Qwen3.6-27B-MTP-UD-GGUF
https://www.reddit.com/r/LocalLLaMA/comments/1t5ageq/qwen3627b_with_mtp_grafted_on_unsloth_ud_xl_25x/
This is an UD XL quantized version of Qwen3-27B provided by Unsloth, with MTP draft heads grafted in Q8_0 format. The base model maintains the same low-bit quantization as before, while only 3 MTP layers are kept at Q8 to preserve accuracy.
I'm sharing the grafted GGUF file (UD XL base + Q8 MTP), the extracted original MTP layer source (MTP_Q8_0.gguf), and a grafting script (convert.py) modified by referring to this gist for those who want to apply the same work to other models. I also include a complete guide for building custom llama.cpp.
Qwen3 is trained with 3-step MTP, predicting 4 tokens at a time in each forward pass. Since the current llama.cpp main branch doesn't yet support MTP, I brought speculative decoding support from the open PR #22673, merged it into the master branch, and built llama-server from that source. Use the following options when running:
--spec-type mtp --spec-draft-n-max 3
Results: We observed approximately 2.5x token throughput improvement compared to the same UD XL GGUF without MTP, and recorded a stable acceptance rate with most draft tokens being accepted. This means the MTP head is not merely consuming computational resources but actually working effectively. Also, since the Q8 MTP layer takes up a very small proportion of the overall model, there is almost no additional VRAM overhead.
MTP is one of the biggest efficiency improvement factors achievable through speculative decoding, but currently it is virtually unsupported except for Qwen3 distributions officially supported by SGLang and vLLM. Through this work, MTP can now be used with GGUF and llama.cpp, and can be run in a local environment with the same tools you've been using. Once PR #22673 is merged, it is expected to be available out of the box soon. Until then, the merge process is simple (just 3 git commands).
If you have any questions or need help running it, I'm always happy to help. If you've tried it yourself, please let me know what speed you're getting!
The complete step-by-step guide is detailed in the HuggingFace repository, but here's the simplified version:
bash
# 1. Build llama.cpp with MTP support
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin
git fetch origin pull/22673/head:pr-22673
git checkout master
git reset --hard origin/master
git merge --no-ff pr-22673 -m "Merge PR #22673: llama + spec: MTP Support"
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release --target llama-server
# 2. Download GGUF files from HuggingFace
# https://huggingface.co/havenoammo/Qwen3.6-27B-MTP-UD-GGUF
# 3. Run with MTP options
./build/bin/llama-server -m your-model.gguf --spec-type mtp --spec-draft-n-max 3
My system is an RTX 3090 24GB system, so I made some changes.
ExecStart=/usr/local/bin/llama-server \
-m /home/models/havenoammo/Qwen3.6-27B-MTP-UD-GGUF/Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf \
--alias Qwen3.6-27B \
--host 0.0.0.0 \
--port 8000 \
--reasoning off \
--temp 0.7 \
--top-p 0.80 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 1.5 \
--repeat-penalty 1.0 \
--kv-unified \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--flash-attn on \
--fit on \
--ctx-size 163840 \
-n 32768 \
--api-key sk........ \
-ngl 99 \
--parallel 1 \
--mlock \
--threads-http 8 \
--spec-type mtp \
--spec-draft-n-max 3
By the way, you should use v=q8_0 to avoid quality loss. If you use v=q4_0, there's a speed advantage, but there may be quality loss, so it's not suitable for coding.
Update - I changed it to q4_0 and it seems fine. Try it in actual use first and adjust as needed
Update 2 - Since prefill and token generation speed have increased, coding feels more enjoyable. ^^
