Search publications
“Theorem 3.3 (Geometric decay of retention (Informal)). Assume that, within an attention head, future query states evolve according to stable dynamics. If, whenever the query state is inside token 's relaxed top- region , there is probability at least of exiting this region within the next decoding steps, then there exists such that .” · Show in paper
The mathematical derivation in Appendix A.2 is sound under Assumptions A.1 and A.2, and Figures 2, 8, and 9 support the empirical plausibility of the assumptions and decay.
“Presented in the second row of Figure 5, RocketKV achieves near the Full-KV accuracy across all models, even with a token budget of 256. In fact, it reaches 100% accuracy on Llama3.1-8B-Ins under 256 token budget, which corresponds to a compression ratio of over 400× with a maximum sequence length of 109K tokens. In contrast, all other methods suffer from substantial accuracy drops.” · Show in paper
The NIAH scores in Figure 5(d-f), Table 2, Table 10, and Table 16 directly support the claim.
“Online latent compaction is nearly lossless, reaching 90.52% accuracy compared with 92.46% for full-context generation, while being highly efficient: it reduces the average KV length by 3.1x, the average per-problem peak by 3.6x, and the maximum peak by 7.9x: the longest full-context trajectories reach the 32,768-token limit, whereas no compacted trajectory exceeds 4,135 tokens.” · Show in paper
The full 5,000-problem MATH test set evaluation under matched prompts, context limits, and scoring directly supports the accuracy and KV-reduction claims.
“Once collapsed, throttling admissions recovered 2 of 8 runs after 21–23 minutes and latency-driven AIMD (which also suspends sessions) 5 of 7 after 14–25 minutes, while suspending admitted sessions at turn boundaries against a limit derived from the KV pool restored throughput and a 0.93–0.95 hit ratio in every collapsed run within 6 minutes.” · Show in paper
Matched-seed recovery experiments and Table 3 directly support the superiority of capacity-based session parking over admission-only throttling.
“The KV block manager also maintains block tables—the mapping between logical and physical KV blocks of each request. Each block table entry records the corresponding physical blocks of a logical block and the number of filled positions. Separating logical and physical KV blocks allows vLLM to dynamically grow the KV cache memory without reserving it for all positions in advance” · Show in paper
“while transferring 3× less data per decoding step than ShadowKV and using the least total memory among the methods we profile. At 128K context, it halves peak GPU memory and cuts total memory by one third relative to full KV caching.” · Show in paper
Tables 2 and 20 directly confirm the 3.1x transfer reduction vs. ShadowKV, lowest total memory (46.01 GB), ~50% peak GPU memory reduction, and ~34% total memory reduction vs. Full KV.
“DBTrimKV adds a small overhead compared to TrimKV due to variable-length, head-specific cache handling; however, it still outperforms vanilla inference at scale.” · Show in paper
Figures 7, 10, 11, and 12 demonstrate that DBTrimKV is substantially faster than Vanilla at scale while incurring a modest overhead relative to TrimKV.
“Query and output matrices are efficiently stored as ragged tensors (also known as jagged arrays) (Tensorflow Developers, 2018) without padding, which facilitates the compact packing of queries and outputs from diverse requests into a single tensor.” · Show in paper
“Inspect at most H2 + O(1) bits of the accuracy input. If its complete encoding has not been obtained, use exact alignment dynamic programming. The unread suffix is neither copied nor addressed.” · Show in paper
“A CPU offload tier delays the collapse but does not prevent it.” · Show in paper
The paired CPU-offload experiments directly demonstrate delayed but unprevented persistent degradation.
“For every plotted setting, BridgeKV improves over the fine-tuned decode anchor while keeping the same small model for generation. However, the magnitude of the gain depends on the task and model pair. As shown, BridgeKV improves the accuracy of the directly fine-tuned decode anchor by up to 8.4% relative (NarrativeQA, for Qwen2.5-Instruct-7B→Qwen2.5-Instruct-0.5B). In the worst case, the relative improvement is 0.2% (SAMSum, for Llama-3.1-Instruct-8B→Llama-3.2-Instruct-3B)” · Show in paper
Figure 3 confirms that BridgeKV outperforms the parameter-matched decode anchor in all 30 settings, with relative gains ranging from 0.2% (SAMSum, Llama 8B->3B: 45.3 to 45.4) to 8.4% (NarrativeQA, Qwen 7B->0.5B: 56.8 to 61.5).
“It matches exact-recall offloading caches on LongBench and InfiniteBench and trails on synthetic exact retrieval” · Show in paper
Table 1 supports the claim: SEKV is slightly ahead on LongBench (+0.54) and nearly level on InfiniteBench (-0.25), while trailing on RULER (-3.32) and NIAH (-4.28).
“For each action, the planner estimates accuracy impact by combining three factors: offline global sensitivity, online routing statistics, and runtime prompt residual.” · Show in paper
“Unless otherwise stated, the main system results use a 200-request trace with 1024 input tokens and 1024 output tokens per request. Request arrivals follow an exponential distribution with an arrival rate of 2.0 requests/s.” · Show in paper
“Dropping the residual entirely (rank-only compression) raises 2× perplexity from 6.54 to 7.27, a gap that additional rank cannot close.” · Show in paper
Table 11 directly supports both the necessity of the rotated residual and the saturation at 4 bits.
“Table 3 shows that DBTrimKV and TrimKV outperform heuristic eviction baselines by a wide margin, with the advantage widening as the KV budget shrinks. Notably, DBTrimKV at all budgets outperforms vanilla inference with full cache (by 14% at M = 512).” · Show in paper
Table 3 (and extended metrics in Table 6 on page 29) directly supports the claim.
“Assign independent uniform priorities in [0, 1] to the algorithms and run them in priority order, caching previously read bits.” · Show in paper
“The optimization below uses finite Gibbs variational duality: subtracting linear constraints from an entropy functional produces a normalized exponential law.” · Show in paper
“For each span and head, we factorize the pre-RoPE key matrix and the value matrix with a truncated SVD up to rank , and a lightweight predictor assigns each component a gate from the singular spectrum and the span summary: ” · Show in paper
“On Mistral-7B-v0.3 at context length 8,192, peak decode memory of the KV representation is 1.43×, 2.40×, 3.61×, and 4.73× below the dense bf16 cache at the 2×, 3×, 5×, and 8× compression targets.” · Show in paper
The memory savings and latency overhead are both transparently reported in Section 4.5.