Introducing the Bonsai27b model (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf), which is a ternary (ternary) model conversion of the Qwen 3.6 27b model. The size is only about 2~8 gigabytes. Nevertheless, its performance on benchmarks is quite good.
However, this requires a special llama.cpp.
Format | llama.cpp version | Status |
|---|
1-bit (Q1_0) | Upstream llama.cpp (latest) | ✅ Works with a normal build |
Ternary (Q2_0) | PrismML fork (prism branch) | ❌ Cannot be built normally |
How to run Ternary
# Use the PrismML fork's prism branch
git clone -b prism https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON # Or -DGGML_METAL=ON
cmake --build build -j
Why is this necessary?
Ternary encodes {-1, 0, +1} three values in the Q2_0_g128 format. Standard llama.cpp does not have this kernel. PrismML wrote a custom 2-bit hybrid attention kernel and put it in the prism branch.
Binaries are also available
If you don't want to build, you can get pre-built binaries for macOS/Linux/Windows from the PrismML llama.cpp releases.
Sources: llama.cpp Docs · Ternary-Bonsai-8B-gguf · Ternary-Bonsai-27B-gguf
