Search publications
“We perform a controlled microbenchmark in which we fix a decode batch of size B, with each decode request having K cached tokens, where K corresponds to the median prompt length of the workload (Table 1). We then inject chunked prefill tokens, increasing the total tokens processed per batch to chi, and measure the resulting step latency increase delta T_step(chi) across five successive chunks.” · Show in paper
“On the 150M image encoder, quantizing activations and weights to 8-bit precision (W8A8) yields a 2x reduction in total forward-pass memory footprint (from 400 MB to 200 MB, including on-device compilation overhead) and a 44% reduction in on-device latency relative to Gemma 3n on newer hardware. On the audio encoder, we further reduce activation precision to 8 bits and weight precision to {2, 4, 8} bits, varying by layer cluster. Overall, we achieve a 78% reduction in on-disk footprint, from 390 MB in Gemma 3n to 87 MB in this version.” · Show in paper
While the 78% audio encoder on-disk footprint reduction (390 MB to 87 MB) is corroborated by Table 7, no benchmark table or hardware measurement details are provided to verify the 2x forward-pass memory reduction or 44% on-device latency reduction for the 150M image encoder.
“SpecStream achieves average throughput speedups of 1.40× and 1.23× over TriForce on Qwen3 and InternLM2.5, respectively.” · Show in paper
Figure 6 and Section 5.2.2 support both the throughput speedups over TriForce and the nuanced limitation regarding Draft KV Cache residency.
“Based on the figure, we can see that RocketKV consistently outperforms all other methods, especially under lower token budgets. For Llama3.1-8B-Ins, RocketKV achieves almost no accuracy loss with a token budget of 512 and above, and only 1.1% average accuracy drop with a token budget of 256.” · Show in paper
Table 2, Table 10, Table 16, and Figure 5(a-c) confirm RocketKV's consistent advantage at low token budgets and the 1.1% (52.2 to 51.1) drop at budget 256 on Llama3.1-8B-Ins.
“J̄(V) − J̄(U) ≤ Cm + (m/Nk)(C + B/h0) + B/q. When B ≤ d^2, the right side is at most C'd for an absolute constant C'.” · Show in paper
All integrability conditions (Lemma 6.2) and independence/conditioning steps are carefully verified.
“To verify that half-precision accumulation does not degrade quality in practice, we compare the LongBench scores obtained with FP16 and FP32 moment statistics on LLaMA-3.1-8B at L=128. The average score difference is less than 0.05 across all task categories, confirming that the numerical error introduced by half-precision accumulation is negligible relative to the approximation error O(\sigma^2).” · Show in paper
The reported experiment in Appendix C.2 supports the claim, though no per-category table is shown.
“Theorem A.1 (Fixed-length binary conversion). There is an absolute integer M0 such that, for every integer M >= M0 and every alphabet Gamma with 1 <= K = |Gamma| <= M, there exists a map c : Gamma -> {0, 1}^P with P = M^2 whose concatenation extension enc satisfies (P/804) ED(x, y) <= ED(enc(x), enc(y)) <= P ED(x, y) (x, y in Gamma^M).” · Show in paper
The probabilistic existence of the codebook (Lemma A.2), the span-trimming bound for unequal labels, and the bipartite multigraph decomposition for equal labels are all rigorous.
“Both terms come from the predictors in §3.3, which on held-out traces achieve MAPE < 10%. Predictions are O(1) via simple linear regressors.” · Show in paper
The paper consistently reports MAPE < 10% on held-out traces for the linear regression step-latency models.
“Our analysis confirms that, for compensated outlier channels, the dual-stage mechanism matches the worst-case error bounds of standard MXFP8.” · Show in paper
The mathematical derivation in Section 3.4 is sound and correctly shows that is bounded by , which is tighter than MXFP8's bound.
“Our local to global attention ratio patterns follow Gemma Team (2025a), that is, 4-to-1 local attention blocks for E2B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e. , values = keys. We encode position with p-RoPE with p = 0.25 on global attention layers and with RoPE on local attention layers, effectively reducing the global KV cache by 37.5%. The RoPE frequencies are set to 1M and 10k on global and local attention layers, respectively. Finally, we share the KV cache with ratios of 20/35 and 18/42 for the E2B and E4B model.” · Show in paper
“For integers p and q > 0, replace each weight by wp/q(e) = qw(e) − p. ... Choose K = 1 + ⌈log2(2(Win + 1)n^2)⌉, computable from integer bit lengths. For each vertex, start with the closed interval [−Win, Win] and make K bisections.” · Show in paper
“At a real root \alpha of a nonzero P, the associated weighted sum is S(\alpha) = \sum_{j=1}^d w_j \operatorname{sgn} b_j(\alpha).” · Show in paper
“The longer agentic profiles exceed the single-H100 HBM budget: coding agent exceeds the wall by C = 40 and reaches 155 GB at C = 80, while terminal reaches 126 GB (Appendix Figure 4b clips both at the 80 GB wall).” · Show in paper
The analytical CF model in Figure 4b and Table 14b directly supports the capacity saturation thresholds.
“Replace each schema by a path with 4(r + 1) private internal types, each of dimension one. Call this subdivided path a wire; its endpoints are the original real types, and its segments are links.” · Show in paper
“This Pareto analysis reveals a crucial asymmetry: element-level metadata is superior under a fixed shared scale due to its ability to capture dominant outliers without global adjustments, while subgroup-level metadata becomes preferable once an adaptive shared scale is incorporated.” · Show in paper
Figures 6 and 7 clearly demonstrate the crossover in MSE between Elem-EM and Sg-EM-2bit at 4.5–4.75 EBW when switching from fixed to adaptive shared scale.
“At 64k tokens, the 16x LCLM reaches the first token faster than the smallest uncompressed models (0.21 s, compared with 0.81 s for Llama 3.2 1B and 0.97 s for Qwen3-0.6B) while being far more accurate on LongHealth5 (66.25% vs. 22.50% and 32.75%). The 4x LCLM exceeds the uncompressed Qwen3-4B-Instruct-2507 on LongBench with a 6x lower TTFT.” · Show in paper
The vLLM-served benchmarks in Figure 5 and Section 6.2 directly substantiate the reported TTFT and accuracy comparisons.
“Approximate this pair in L^1 by rational-valued step functions on finite dyadic rectangular grids. For example, conditional expectations on the successive dyadic grids converge in L^1, and subsequent rational quantization preserves the range [0, 1]^2 and the convergence.” · Show in paper
“Thus, recomputation is more efficient when the block size is small, while swapping is more efficient when the block size is large, though recomputation overhead is never higher than 20% of swapping’s latency. For medium block sizes from 16 to 64, the two methods exhibit comparable end-to-end performance.” · Show in paper
Figure 19 (a) and (b) directly support the microbenchmark and end-to-end performance comparison across block sizes.
“In Table 1, we verify the assumption that simulated per-channel activation quantization successfully bridges the accuracy with the FP16 baseline, which echos the findings of Bondarenko et al.. However, per-channel activation quantization does not map well to hardware-accelerated GEMM kernels, that rely on a sequence of operations executed at a high throughput (e.g., Tensor Core MMAs) and do not tolerate the insertion of instructions with a lower throughput (e.g., conversions or CUDA Core FMAs) in that sequence.” · Show in paper
Table 1 supports the accuracy of simulated per-channel quantization, and the hardware GEMM constraint is a well-established architectural fact.
“To carry this table through another pair of splits, encode a payload v ∈ Ui and a work-register value m ∈ M as a single basis vector. Set Ui+1 = F Ui×M 2 with basis [v, m].” · Show in paper