Search publications
“All-KV was not the consistent winner in terms of accuracy across the various workloads in our benchmark. Instead, under most workloads, one of the compression techniques achieved even better results.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingTable 2 directly shows compressed methods outperforming All-KV on most model-dataset combinations.
“The results show that the compression ratio alone is a poor predictor of end-to-end performance.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingThe joint evaluation across task quality, TTFT, throughput, and realized compression ratio clearly demonstrates that compression ratio alone does not predict end-to-end system or quality outcomes.
“The B-bit integer quantization-dequantization process can be expressed as: Q(X) = ⌊(X - z_X)/s_X⌉, X' = Q(X) · s_X + z_X, where z_X = min X is the zero-point, s_X = (max X - min X)/(2^B - 1) is the scaling factor, and ⌊·⌉ is the rounding operation.” · Show in paper
“For KIVI [33], we use the official KIVI-style asymmetric KV-cache quantization implementation integrated into the Llama and Mistral attention modules. We evaluate both 2-bit and 4-bit KV-cache variants.” · Show in paper
“Overall, these results indicate that KV cache compression techniques introduce only limited additional latency for the first token, with the exception of TurboQuant, whose more complex quantization pipeline leads to noticeable overhead.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingTable 3 and Figure 2 support the claim for the evaluated datasets, though GovReport results for TurboQuant are omitted due to 3-day single-run execution times.
“Quality is essentially unchanged in the free zone. On Mistral at T=1024, the reconstruction-error gap between FlashJoLT and JoLT is |∆K| ≤ 0.0093 and |∆V | ≤ 0.0011 across 2–3× (Appendix A). Timing both compressors over the full compression pipeline, including layer grouping, gives end-to-end speedups of 1.04× to 3.97× depending on cache length and target ratio (Appendix A).” · Show in paper
Appendix A provides both the reconstruction-error parity measurements and the end-to-end wall-clock timing on A100.
“Experimental results validate our theoretical findings, showing that for KV cache quantization, we achieve absolute quality neutrality with 3.5 bits per channel and marginal quality degradation with 2.5 bits per channel.” · Show in paper
The experimental results in Figure 4 and Table 1 directly support the claimed quality neutrality at 3.5 bits and marginal degradation at 2.5 bits.
“In Figure 13, LCLMs achieve the highest accuracy across all compression ratios on GSM8K, with particularly strong gains over baselines at higher compression ratios.” · Show in paper
Table 9 and Figure 13 directly support the claim across all three compression ratios.
“Across diverse long-context language and vision-language reasoning, and multi-turn dialogue benchmarks, our method substantially reduces KV memory while matching or surpassing full-cache inference. These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.” · Show in paper
The experimental results across Tables 1, 2, 3, 7 and Figure 5 directly support that DBTrimKV reduces KV memory while matching or surpassing full-cache inference across all tested benchmark categories.
“We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks.” · Show in paper
The reported 400x compression ratio, 3.7x A100 speedup, 32.6% peak memory reduction, and near-Full-KV accuracy are directly supported by the experimental results in Section 4.2, Section 4.3, Figure 5, Figure 7, and Table 2.
“Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2–3× compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3× and declines by only 0.90 and 2.40 pp at 4× and 5×, respectively.” · Show in paper
The experimental results in Table 1, Tables 4-7, and Table 9 support the 2-3x near-lossless claim across the tested models and tasks, noting that LLaMA-3.1-8B and Qwen2.5-14B used FlashJoLT.
“These non-integer bit precisions result from our strategy of splitting channels into outlier and non-outlier sets, and applying two independent instances of TurboQuant to each, allocating higher bit precision to outliers.” · Show in paper
“Across five backbones and four benchmarks at a 10% GPU-resident KV budget, SEKV outperforms every method that fixes resolution at prefill, improving over the strongest semantic baseline by 2.9 points (5.5% relative).” · Show in paper
The averages in Table 1 and per-backbone breakdown in Table 7 directly support the 2.9-point (5.5% relative) improvement over SentenceKV and the superiority over all tested fixed-resolution baselines.
“As shown in Figure 3, RocketKV performs coarse-grain KV cache eviction in the first stage. The purpose of this stage is to remove KV tokens with low importance while keeping the majority of important tokens. In the second stage, it conducts fine-grain dynamic KV token selection on the remaining KV tokens, followed by top-k sparse attention.” · Show in paper
“our key idea to solve this problem is to group key cache every G tokens and quantize them separately. Because the number of tokens in X_K can be arbitrary, we split X_K into two parts, namely, the grouped part = X_K[: l - r] and residual part = X_K[l - r :], where l is the number of tokens inside the current key cache X_K, r is the number of residual tokens, where l - r can be divisible by G.” · Show in paper
“The first-order approximation adds O(d^2) storage per head (4.1 MB total, 6.4% of retained cache) and 1.6 ms latency, yet more than triples the zeroth-order gain by capturing query-dependent directional variation. Both costs are determined by d and do not grow with context length, making MOMENTKV practical for arbitrarily long sequences while reducing per-token latency by over 65% compared to the full cache.” · Show in paper
Table 4 confirms the 1.6 ms overhead over SnapKV and >65% latency reduction vs. Full Cache (9.8 vs 28.5 ms). Note that Table 5 lists 8.5 MB (6.4%) for LLaMA-3.1-8B whereas the text on page 9 says 4.1 MB (6.4%), which is a minor internal discrepancy in the reported MB figure.
“On LongBench and RULER with LLaMA-3.1-8B-Instruct and Qwen3-4B-Instruct, MOMENTKV outperforms all baselines at every cache budget, with the largest gains under aggressive compression.” · Show in paper
The experimental results in Tables 1 and 2 directly support the claim across all tested models, benchmarks, and cache budgets.
“From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using 2.6× less peak memory (including model weight). This reduction in memory usage enables up to 4× larger batch size, bringing 2.35× ∼ 3.47× throughput on real LLM inference workload.” · Show in paper
The headline claim overgeneralizes: Section 4.2.2 explicitly notes that Falcon-7B needs 4bit KIVI to maintain accuracy and suffers a large accuracy drop with 2bit KIVI, and the 2.6x memory reduction and throughput gains were only measured on Llama-2-7B.
“Our composable format design allows for the decomposition of the KV cache sparse matrix based on prior knowledge. For instance, if certain requests share a prefix, the corresponding rows and columns in the KV cache form a dense submatrix. We can then use a block sparse matrix with a larger B_r to store these submatrices efficiently.” · Show in paper
“Combined with KV cache sharing (Shazeer, 2019) and the reuse of keys as values in global layers (Kayyam et al., 2026), these optimizations reduce the global KV cache footprint by up to 37.5%.” · Show in paper
The reduction follows directly from the specified architectural parameterization ( -RoPE and key-value reuse) and is consistent with the KV cache footprints reported in Table 3.