Author(s): Devavrat Samak

Originally published on Towards AI.

Part 2 of 2. If you have not read Part 1, it covers the KV cache problem and how TurboQuant works. This part is about what happened when I actually tested it.

Same algorithm. Same model. Same input text. Same bit-width.

In this second part, the author shows that TurboQuant’s results are highly seed-dependent: changing only the random seed can swing perplexity dramatically from near-failure to acceptable behavior, even when geometric distortion measures look similar. They explain why seeds matter—because TurboQuant applies a seed-derived random rotation before quantization, and different rotations interact differently with quantization error—then describe an experiment setup across multiple seeds, two model architectures, and multiple bit-widths. By focusing on more informative diagnostics than perplexity alone, they demonstrate that different metrics expose different failure modes: KL divergence reveals distribution-level distortion; layer-wise sensitivity identifies architectural outliers (notably Layer 0 for Qwen); norm/direction decomposition shows that “stored exactly” norms still produce significant error due to computation precision; token-position degradation is largely flat (challenging residual-window heuristics); and for GQA, cross-query consistency shows superlinear disagreement as more query heads share one KV head. Overall, the piece argues that model KV compression should be evaluated with targeted, stable metrics and careful seed- and architecture-aware testing, and it provides code and open questions for predicting catastrophic seed behavior.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI