(Qwen3.6-27B-MTP model, confirmed speed of 52.58 token/s when tested with 'hello')
I saw the Qwen3.6-27B-MTP model from unsloth yesterday, downloaded it, and tested it.
(Model address https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF )
Based on 1 RTX3090, the existing 27B model speed was around 20-30 Token/s,
After applying MTP, it seems to output around 52 token/s, about 1.5-2 times improvement as shown in the screenshot above (verified with simple 'hello')
I think it's very useful for personal use.
To use it, there's an issue where you need to recompile llama.cpp to a version that supports speculative decoding along with the MTP-applied model,
You can recompile it according to the instructions in the model address above.
Below are how to download the model and how to compile the new llama.cpp. (Based on my installation. It may vary for each person)
# How to download the model (I used Q4_K_M. 17.1GB model)
hf download unsloth/Qwen3.6-27B-MTP-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir /home/myfolder/models/unsloth/Qwen3.6-27B-MTP-GGUF
# How to install new llama.cpp (Official or unsloth introduction)
Related comment: MTP Support: https://github.com/ggml-org/llama.cpp/pull/22673
# Add CUDA (nvcc) path to current session
export PATH=/usr/local/cuda/bin:$PATH
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone -b mtp-clean https://github.com/am17an/llama.cpp.git
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
However, I got a linking error at the end with the above method during compilation, so I compiled using the method below.
cd ~/llama.cpp # Move to llama.cpp folder
# Completely delete build folder
rm -rf build
CUDACXX=/usr/local/cuda/bin/nvcc \
cmake -B build \
-DGGML_CUDA=ON \
-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
-DCUDAToolkit_ROOT=/usr/local/cuda/targets/x86_64-linux \
-DCMAKE_CUDA_ARCHITECTURES=native \
-DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64 -L/usr/local/cuda/targets/x86_64-linux/lib" \
-DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64"
cmake --build build --config Release -j$(nproc)
# For llama.cpp binaries, just create symlinks with ln -sf. (The reason is that compilation changes frequently)
sudo ln -sf /home/myfolder/git/llama.cpp/build/bin/llama-cli /usr/local/bin/llama-cli
sudo ln -sf /home/myfolder/git/llama.cpp/build/bin/llama-server /usr/local/bin/llama-server
sudo ln -sf /home/myfolder/git/llama.cpp/build/bin/llama-quantize /usr/local/bin/llama-quantize
At the bottom of the above execution options
-np 1 \
--spec-type mtp \
--spec-draft-n-max 3
This part is the key.
Below is the llama execution batch file
# cat qwen36_27b_mtp.sh
#!/bin/sh
export LD_LIBRARY_PATH=/home/myfolder/git/llama.cpp/build/bin:$LD_LIBRARY_PATH
/usr/local/bin/llama-server \
-m /home/myfolder/models/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-Q4_K_M.gguf \
--alias Qwen3.6-27B-MTP \
--host 0.0.0.0 \
--port 8000 \
--temp 0.6 \
--top-p 0.95 \
--top-k 40 \
--min-p 0.05 \
--repeat-penalty 1.0 \
--presence-penalty 0.5 \
--kv-unified \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--flash-attn on \
--ctx-size 196608 \
--n-gpu-layers 99 \
--batch-size 2048 \
--ubatch-size 512 \
--no-mmap \
--fit-target 1536 \
--api-key sk-kkkk \
-np 1 \
--spec-type mtp \
--spec-draft-n-max 3
(Change or delete the API key part. I set ctx-size to 262144 but it ran out of memory, so I lowered it a bit. Note: Even running at 196608 it ran out of memory, so I lowered it to 131072 again. I can't quite gauge it. It's sad.)
I'll stop here.
ps. Bonus: Qwen3.6-27B with MTP support and uncensored version model
https://huggingface.co/llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-GGUF
ps2. I don't know why the
tag appears when line breaking the options above. It's a backslash \ .
