Search publications
“We conduct all our experiments using the open-weight Llama 3.1-Instruct model, which is an instruction-finetuned version of Llama 3.1 (Meta AI 2024). ... For our RAG experiments, we make use of an off-the-shelf and a specialised multilingual models – Jina-reranker (JinaAI) (Multilingual 2024) and paraphrase-multilingual-mpnet-base-v2 (MPNet) (Reimers and Gurevych 2020), respectively.” · Show in paper
“At sequence length 1024, ARCQuant reaches 21077.0 total tokens/s and 2341.9 decode tokens/s, corresponding to a 1.96× decoding speedup over FP16. At sequence length 2048, it achieves 21236.6 total tokens/s and 1249.2 decode tokens/s, yielding a 2.08× decoding speedup over FP16. In both settings, ARCQuant also substantially outperforms the FP8 reference, demonstrating that its efficiency advantages extend beyond prefill and remain effective in decoding generation scenarios.” · Show in paper
Table 4 shows decode throughput of 2342 vs 1195 (1.96x) at SeqLen 1024 and 1249 vs 600 (2.08x) at SeqLen 2048, outperforming both FP16 and FP8.
“Again, RocketKV shows robust performance and a clear advantage over all other methods across various token budget and sequence length settings. Overall, we can see that the accuracy loss of RocketKV is negligible under short sequence lengths and gradually becomes larger as sequence length increases.” · Show in paper
Figure 5(g-r) and the detailed RULER tables (Tables 6-9, 12-15, 18-21) confirm RocketKV's advantage across sequence lengths and token budgets.
“On LongBench and RULER with LLaMA-3.1-8B-Instruct and Qwen3-4B-Instruct, MOMENTKV outperforms all baselines at every cache budget, with the largest gains under aggressive compression.” · Show in paper
The experimental results in Tables 1 and 2 directly support the claim across all tested models, benchmarks, and cache budgets.
“We developed CUDA/CUTLASS (Thakkar et al., 2023) templates for FlashAttention, designed specifically for both dense and block-sparse matrices and compatible with NVIDIA GPU architectures from Turing to Hopper (sm75 to sm90a).” · Show in paper
“All stitching layers of a pair have the same shapes, so the batched implementation stacks their weights and runs the whole module in one pass.” · Show in paper
“The first-order approximation adds O(d^2) storage per head (4.1 MB total, 6.4% of retained cache) and 1.6 ms latency, yet more than triples the zeroth-order gain by capturing query-dependent directional variation. Both costs are determined by d and do not grow with context length, making MOMENTKV practical for arbitrarily long sequences while reducing per-token latency by over 65% compared to the full cache.” · Show in paper
Table 4 confirms the 1.6 ms overhead over SnapKV and >65% latency reduction vs. Full Cache (9.8 vs 28.5 ms). Note that Table 5 lists 8.5 MB (6.4%) for LLaMA-3.1-8B whereas the text on page 9 says 4.1 MB (6.4%), which is a minor internal discrepancy in the reported MB figure.
“Across all four scenarios and a range of sequence lengths, FlashInfer consistently outperforms FlexAttention, with especially large gains at longer sequence lengths.” · Show in paper
Tables 1–4 and Tables 9–11 in Appendix G directly substantiate these performance comparisons.
“Presented in the second row of Figure 5, RocketKV achieves near the Full-KV accuracy across all models, even with a token budget of 256. In fact, it reaches 100% accuracy on Llama3.1-8B-Ins under 256 token budget, which corresponds to a compression ratio of over 400× with a maximum sequence length of 109K tokens. In contrast, all other methods suffer from substantial accuracy drops.” · Show in paper
The NIAH scores in Figure 5(d-f), Table 2, Table 10, and Table 16 directly support the claim.
“The base LLM stays frozen, and the router adds under 0.1% parameters.” · Show in paper
Table 24 confirms parameter overhead is 0.054%–0.086% (<0.1%), and Table 3 confirms the 2.8–3.8 point drop when using training-free routing.
“The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers.” · Show in paper
The SGLang paired runs and 32B trace-shape runs support the generalization, and the authors appropriately note that 32B provides weaker evidence of history dependence than 8B.
“The finetuned models SWE-Llama 7b and 13b perform surprisingly poorly with BM25 retrieved context. As these models were fine-tuned using the “oracle” retrieval as context, we suspect this shift in context makes it difficult for the model to perform reliably. For instance, SWE-Llama was trained to edit every file included as context whereas in the BM25 setting many files provided in context are not expected to be changed.” · Show in paper
The performance drop from 3.97% (Oracle) to 0.70% (BM25) for SWE-Llama 13b is documented in Tables 5 and 18, and the explanation aligns with the training setup.
“BridgeKV reduces response time from 39.80 s to 9.13 s, a 4.36× speedup over Llama-3.1-Instruct-8B-only inference. Throughput increases from 25.81 tok/s to 113.5 tok/s, a 4.40× gain, because decode follows the Llama-3.2-Instruct-1B anchor. ... Energy per request drops from 11.98 Wh (i.e., watt-hour) for Llama-3.1-Instruct-8B-only inference to 2.78 Wh for BridgeKV, corresponding to a 4.31× reduction.” · Show in paper
Tables 5 and 6 directly report these exact simulation numbers.
“Thus, recomputation is more efficient when the block size is small, while swapping is more efficient when the block size is large, though recomputation overhead is never higher than 20% of swapping’s latency. For medium block sizes from 16 to 64, the two methods exhibit comparable end-to-end performance.” · Show in paper
Figure 19 (a) and (b) directly support the microbenchmark and end-to-end performance comparison across block sizes.
“Table 1 reports evaluation results on CMMLU and C-Eval (valid) with G = 64. FP4Quant consistently preserves downstream performance across model-parallel settings, achieving scores of 0.5287/0.4829 (TP), 0.5288/0.5342 (PP), and 0.5301/0.5253 (MP), compared with the BF16 baseline (0.5365/0.5215). Notably, under PP and MP configurations, FP4Quant matches or slightly exceeds BF16 on C-Eval, demonstrating strong capability preservation under ultra-low-bit compression.” · Show in paper
Table 1 confirms the reported scores and shows that Tahquant drops to near random-chance levels under TP and MP while FP4Quant remains close to BF16.
“Across diverse long-context language and vision-language reasoning, and multi-turn dialogue benchmarks, our method substantially reduces KV memory while matching or surpassing full-cache inference. These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.” · Show in paper
The experimental results across Tables 1, 2, 3, 7 and Figure 5 directly support that DBTrimKV reduces KV memory while matching or surpassing full-cache inference across all tested benchmark categories.
“All four are 4-bit GGUF builds served by llama-cpp-python with every layer on the GPU, and each file is pinned by its sha256” · Show in paper
“Table 1 reports perplexity at n=1024 (T=512) on all five models. Every free-zone cell stays below the 0.2% threshold, with the largest drift at +0.17% (Qwen at 3×). The free zone holds at context lengths 512 and 2048 as well (Appendix B).” · Show in paper
All reported 2x and 3x perplexity values in Table 1 and Tables 4-6 are strictly below +0.2% relative drift.
“3.2. Generational gains are driven mainly by reach and reliability rather than efficiency” · Show in paper
The regression results in Table 4 and curve summaries in Table 9 support that reach and reliability improve more consistently across generations than token efficiency.
“Table 2 shows that PagedWeight preserves long-context quality with much lower memory consumption than FP16. At the 10 GB budget, PagedWeight reaches the same 17.0% average score as FP16 while using only 9.86 GB. In contrast, APL reaches only 12.2% at a similar memory consumption.” · Show in paper
Table 2 directly supports the claim, showing PagedWeight-10GB matches FP16's 17.0% average score and substantially outperforms APL-3bit on Passage Retrieval (17.5% vs. 10.5%) and NarrativeQA (10.0% vs. 4.9%).