Google announced Turbo quant in a paper.
Below is Gemini's analysis
‐‐----
Breaking Through VRAM Limits: Google's Next-Generation KV Cache Compression Technology 'TurboQuant' Complete Guide
Recently announced by Google Research, TurboQuant is a 'game changer' technology for users running large language models (LLMs) in local environments or with limited resources. It particularly solves the critical VRAM shortage problem that occurs when processing long contexts.
1. What is TurboQuant?
An algorithm that extremely compresses the 'KV Cache (Key-Value Cache)', the area where LLMs store previous conversation content when generating responses. Rather than reducing the model itself (weights), it's a technology that optimizes the 'working memory space' that the model uses.
2. Core Technology: What Makes It Special?
Existing compression methods had the problem of models becoming increasingly 'dumb' as the number of bits decreased. TurboQuant solved this with two revolutionary mathematical approaches.
PolarQuant: Converts and stores data in polar coordinates rather than standard Cartesian coordinates. In this process, it removes additional computational constants that consume large amounts of memory, maximizing efficiency.
QJL (Quantized Johnson-Lindenstrauss): Corrects fine information loss that occurs during data compression with just 1 additional bit of data. Thanks to this, it maintains accuracy close to the original (16-bit) even at the low bit depth of 3 bits.
3. Key Benefits
Overwhelming Compression Ratio: Compresses data from FP16 (16-bit) to 3 bits, reducing the memory occupied by KV cache by approximately 6 times.
Minimal Performance Degradation: According to Google's benchmark results, despite extreme compression, the model's accuracy loss is almost negligible.
Speed Improvement: As data size decreases, GPU memory bandwidth is less occupied, allowing inference speed to be up to 8 times faster (H100 standard) compared to before.
Versatility: A 'Training-free' method that requires no retraining (fine-tuning) for specific models. Can be immediately applied to any LLM engine simply by implementing the algorithm.
4. What It Means for Local LLM Users
This technology especially benefits users with consumer GPUs like RTX 3090/4090.
Context Expansion: Context length that was limited to 8K due to VRAM constraints can now be significantly expanded to 32K, 64K or more in the same environment.
Running High-Parameter Models: Solves the problem where VRAM would overflow when running large models of 70B or more if the context length increased even slightly, providing a much more comfortable inference environment.
Application Through Engine Updates Only: Users don't need to change existing GGUF or EXL2 model files. Once execution engines like llama.cpp or vLLM are updated, benefits can be enjoyed immediately.
==================
Related to this, discussions are underway in each open source project to support TurboQuant functionality in LLM engines.
It seems that soon it will be possible to use larger models locally.
