Paper 2511.18643
Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-Wise Precision Boost
- Published
- Nov 2025
- Research lab
- Together AI
- Citations
- 7
- GitHub
- Not linked
01 In brief
Summary
The paper introduces Kitty, an algorithm-system co-design for 2-bit KV cache quantization that maintains accuracy close to FP16 while reducing memory by nearly 8x.
The authors observe that 4-bit quantization preserves accuracy, but 2-bit degrades it significantly.
They propose Dynamic Channel-wise Precision Boost, which identifies critical key-cache channels via magnitude-based heuristics and preserves a small fraction (12.5%-25%) in 4-bit precision while quantizing the rest to 2-bit.
This approach, combined with preserving initial tokens in full precision, recovers most accuracy loss.
The system design decomposes mixed-precision key pages into two unified 2-bit tensors, enabling coalesced memory access and efficient Triton-based dequantization kernels.
Evaluations on Qwen3 and LLaMA3 models across seven tasks show Kitty-Pro achieves near-parity with FP16 accuracy, with average drops under 1 point.
The system enables 8x larger batch sizes and 2.1x-4.1x higher throughput compared to FP16 baselines under the same memory budget.
The implementation is open-sourced at https://github.com/Summer-Summer/Kitty.
02 From the paper
Abstract
The KV cache is a dominant memory bottleneck for LLM inference. While 4-bit KV quantization preserves accuracy, 2-bit often degrades it, especially on long-context reasoning. We close this gap via an algorithm-system co-design for mixed-precision KV caching: Kitty. On the algorithm side, extensive experiments show that Dynamic Channel-wise Precision Boost -- which ranks Key-cache channels by sensitivity and keeps only a small fraction at higher precision -- maintains near-zero loss in accuracy drop while approaching 2-bit memory. The main challenge is handling dynamic 4-bit channel boosts while keeping the page layout coalesced and the dequantization uniform, with no scattered reads or hard-coded masks. Kitty addresses these issues by decompose each mixed-precision Key page into two tensors with unified 2-bit precision. Based on this, Kitty provides a page-centric KV layout, Triton-compatible page dequantization kernels, and a lightweight runtime pipeline that preserves coalescing and avoids divergence. Across seven tasks and two model families (Qwen3, LLaMA3), Kitty cuts KV memory by nearly 8x with negligible accuracy loss, enabling up to 8x larger batches and 2.1x-4.1x higher throughput under the same memory budget. We release the full implementation of Kitty at https://github.com/Summer-Summer/Kitty.