Search publications
“For key cache, there are a few fixed channels whose magnitudes are very large, which is consistent with previous finding (Lin et al., 2023; Xiao et al., 2023a). Thus, as shown in Figure 1 right, key cache should be quantized per-channel, i.e., group elements along the channel dimension and quantize them together. In this way, it can confine the error to each individual channel, without impacting the other normal channels.” · Show in paper
Both the visualization in Figure 2 and the quantitative error metrics in Table 2 directly support that key cache has channel-wise outliers and suffers much lower error under per-channel quantization.
“SnapKV achieves the best throughput, on par with All-KV on Llama and even surpassing All-KV on Mistral.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingFigure 3 directly supports that SnapKV-0.75 achieves the highest average throughput among all evaluated methods on both models.
“We identify that a primary source of output degradation is not the residual attention mass on evicted tokens, which existing methods already minimize, but a directional mismatch between the retained and evicted token sets. Specifically, the evicted tokens in practice are often near-orthogonal to the retained ones. Thus, even a small evicted mass could have an oversized impact on the resulting direction distribution and amplify into substantial output error.” · Show in paper
The mathematical decomposition in Section 3 and empirical measurements in Figure 2 substantiate that directional mismatch drives the output error.
“For decode kernels, the performance gap between sparse and dense KV-Cache is negligible (within 1%). For prefill kernels, there is approximately a 10% performance gap.” · Show in paper
Figure 12 shows decode bandwidth utilization within 1% (83–85%) and prefill TFLOPs/s within ~5–15% (around 10% on average).
“SpecStream uses online softmax to maintain normalization state chunk by chunk [17, 18]. For each query and attention head, it stores three accumulators (m, ℓ,a).” · Show in paper
“Storage: spending the same CPU memory on exact KV for a subset of spans costs 2.9 to 3.5 points, because low-rank factors cover every span, and a fixed rank of 32 matches the learned rank in accuracy but needs 1.6× the CPU memory.” · Show in paper
Table 3 directly supports both the 2.9–3.5 point drop for exact KV at matched CPU bytes and the ~1.6x CPU memory increase for fixed rank 32.
“The fixed-pair estimate below combines polynomial evaluation, as in classical algebraic fingerprinting [32], with affine universal hashing [11].” · Show in paper
“Gψ,j(z) = Pj z̄ + bj + αj Dj σ(Uj z̄), z̄ = LN(z)” · Show in paper
“Indeed, our profiling results in Fig. 2 show that only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems.” · Show in paper
The profiling data in Figure 2 directly supports the 96.3% vs 20.4%-38.2% memory utilization comparison.
“CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingBoth Table 2 and Table 4 clearly demonstrate extreme workload sensitivity in CaM's accuracy and realized compression ratio.
“For the question answering tasks HotpotQA, 2WikiMQA, Qasper, and MultiFieldQA_en, we use the F1 score, which measures the overlap between predicted and ground-truth answers. For TriviaQA, we use the exact match (EM)” · Show in paper
“LCLMs achieve the fastest compression time at every context length and substantially lower peak memory at long contexts; all baselines run out of memory at 512K tokens, except Attention Matching, which fails at 512K due to numerical instability in its linear solver and runs out of memory at 1M.” · Show in paper
Figure 4 and Section 6.2 clearly document the scaling measurements and memory limits on the H200 GPU.
“As shown in Table 3, we observe that for the Llama and Mistral model, KIVI only has up to 2% accuracy drop despite the KV cache being stored in 2bit. ... Since Falcon-7B adopts multi-query attention and only has one head for KV cache, it is already highly compressed compared to Llama-based models. Thus, in Table 3, 4bit KIVI is needed to maintain the accuracy, while 2bit KIVI may have a large accuracy drop in this case.” · Show in paper
The reported tables and figures accurately reflect these accuracy bounds and the distinction between Llama/Mistral and Falcon-7B.
“We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage.” · Show in paper
The accuracy, TTFT, and peak memory benchmarks across RULER, LongBench, LongHealth5, and GSM8K support the claim that LCLMs improve the Pareto frontier over KV cache compression baselines.
“The difference between fake “2bit (K per-channel, V per-token)” quantization and KIVI is that KIVI maintains a full precision key and value cache sliding window for the local relevant tokens. This sliding window is crucial to maintaining accuracy for hard generation tasks such as mathematical reasoning.” · Show in paper
Table 3 clearly shows that fake 2bit (K-C, V-T) degrades sharply on GSM8K, while KIVI-2 recovers most of the 16bit baseline performance.
“For value cache, there is no obvious outlier pattern. Although value cache has no obvious outlier pattern, we experimentally show that it can only be quantized per-token because it is used to calculate the attention output, which is essentially a value cache mixer. As shown in Figure 1 left, the per-token quantization can confine the error inside each individual token and ensure that the quantization of one token does not adversely impact the others.” · Show in paper
The sparsity measurement, the 15x reduction in relative attention output error in Table 2, and the downstream task results in Tables 1 and 3 strongly support this claim.
“Notably, DBTrimKV with budgets of 1024, 512, and even 256 surpasses vanilla full-cache inference with up to 3.75%.” · Show in paper
Table 2 (103.73% at budget 512, i.e., +3.73%, and 103.26% at 1024) and Figure 5 support the claim.
“At inference time, we use the learned retention coefficients to rank all cached KV entries globally. For each cached token (ℓ, h, i) present at decoding step t, we assign the retention score G̃ℓ,h,i(t) = ∑T s=t+1 βs−i ℓ,h,i = βt+1−i ℓ,h,i (1 − βT−t ℓ,h,i)/(1 − βℓ,h,i), for βℓ,h,i < 1.” · Show in paper
“The Target KV Cache is partitioned into the following three regions accordingly, Sealed History: [0,H) (CPU) Active Tail: [H,Cr) (GPU) Speculative Frontier:[Cr ,Lr)(GPU) 0 ≤ H ≤ Cr ≤ Lr.” · Show in paper
“As shown in Table 1, both DBTrimKV and TrimKV consistently outperform visual token pruning baselines and match vanilla performance under all memory budgets.” · Show in paper
Table 1 directly supports the claim, showing 99.4%–99.9% relative performance compared to vanilla and higher scores than all visual token pruning baselines.