First, when TurboQuant came out, I checked the details and found that it compresses the KV cache incredibly. But since there's no degradation in speed or quality, and it can increase the context window that we've been desperately wanting, my eyes lit up. So I searched GitHub a bit and found that https://github.com/tonbistudio/turboquant-pytorch was already available.
So I thought I'd try applying it to the Ubuntu where I was making simple web apps and other applications, and I did some trial and error for a bit...
Oh! It's amazing! Now I can just use this for coding.
Honestly, I could barely fit a ctx window of 16384, so I couldn't even dream of coding, but it runs smoothly up to 110,000. Wait? How is this even possible... why exactly...
Anyway, I did some experiments and it actually works without any problems.
From code base code reviews to running everything I made simply, it doesn't crash.
I thought this was crazy.
Anyway, if you take this source directly, you'll definitely get errors.
So I'm publishing my troubleshooting results as a repo.
I'm confident that all of you AI experts who are much better than this novice will make something even better... but though embarrassing, I'm publishing my troubleshooting results. (Please don't curse...)
https://github.com/lowhillfoto/Ubuntu-vLLM-TurboQuant
If you go there, you'll find the installation procedure and records of my troubleshooting, so I hope others won't have to go through the same trial and error I did. I'm also posting a simple benchmark result.
vLLM + TurboQuant + OpenCode Configuration Documentation
Complete record of TurboQuant KV cache CPU offload patch and OpenCode integration settings for operating the Qwen3-Coder-30B-A3B MoE model in a single GPU environment with RTX 3090 (24GB).
Benchmark Summary
Measurement Environment: RTX 3090 24GB · vLLM 0.18.0 · compressed-tensors W4A16 · fp8 KV cache detailed results → BENCHMARK.md
Actual Coding Speed (Decode)
Output Scale | Output Tokens | Time Taken | Generation Speed |
|---|
Short function | ~200 t | 5.6 s | 35.7 t/s |
Medium class | ~500 t | 14.2 s | 35.2 t/s |
Long implementation | ~1,000 t | 28.8 s | 34.7 t/s |
Large refactoring | ~1,500 t | 42.0 s | 35.7 t/s |
Average 35.3 t/s — Consistent regardless of output length
Overall Performance Metrics
Item | Value |
|---|
Decode speed | 35.3 t/s (±0.5, very stable) |
TTFT — prefix cache hit | 0.06 s |
TTFT — cold start | 0.5 – 5 s (context dependent) |
Peak prefill | 4,371 t/s @ 10K ctx |
Maximum stable context | ~115,000 tokens (~88,000 words) |
Coding accuracy | 5/5 (100%) |
Tool call success rate | 10/10 (100%), 0.88–0.92 s/instance |
Practical Speed Reference
1 function (~100t) → ~3 seconds
Function + test (~300t) → ~9 seconds
Class implementation (~800t) → ~23 seconds
File refactoring (~1500t) → ~43 seconds
Table of Contents
System Configuration Overview
Fresh Installation Procedure
TurboQuant Patch
vLLM Service Configuration
OpenCode Configuration
Precautions When Updating vLLM
Troubleshooting Log
1. System Configuration Overview
Item | Value |
|---|
GPU | RTX 3090 24GB |
OS | Ubuntu (Linux 6.8) |
vLLM | 0.18.0 (/opt/vllm/venv) |
Model | Qwen3-Coder-30B-A3B-Instruct-W4A16-awq |
Model Path | /home/models/Qwen3-Coder-30B-A3B-Instruct-W4A16-awq
|
Weight Size | 16GB (compressed-tensors W4A16) |
GPU KV Memory | 5.38 GiB (0.18.0 baseline) |
max_model_len | 117,616 tokens |
Memory Layout (During Inference)
RTX 3090 24GB
├── Model weights (W4A16) : ~15.6 GB
├── GPU KV cache (fp8) : ~5.4 GB → 117,616 tokens
└── CUDA/PyTorch overhead : ~2.6 GB
Model Architecture (Qwen3-Coder-30B-A3B MoE)
Item | Value |
|---|
Total parameters | 30.5B |
Active parameters (inference) | 3.3B |
Number of layers | 48 |
Q heads | 32 |
KV heads | 4 (GQA) |
head_dim | 128 |
Number of experts | 128 (8 active per token) |
Native context | 262,144 tokens |
Quantization | compressed-tensors W4A16 |
