Search publications
“KIVI4 provides the most stable quality across models” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingTable 2 confirms that KIVI4 avoids the large accuracy drops seen in TurboQuant, SnapKV, and CaM on summarization while maintaining competitive or superior scores on QA and few-shot tasks.
“The throughput comparison outlined in Figure 6 indicates that QQQ outperforms the alternative methods across a range of batch sizes in most cases. Specifically, QQQ attains a substantial speed increase, reaching up to 2.24×, 2.10×, 1.59×, and 1.25× faster than FP16, SmoothQuant, AWQ, and Marlin on the LLaMA-2-13B model, respectively.” · Show in paper
Figure 6(b) and Section 4.3 support the reported speedup numbers on LLaMA-2-13B.
“Although tailored for the fine-grained NVFP4 format, ARCQuant demonstrates robust extensibility to INT4 and MXFP4 formats. On Llama 3.1-8B, our method consistently outperforms RTN baselines across all metrics as detailed in Appendix B. Notably, ARCQuant reduces INT4 perplexity by 0.89 and improves zero-shot accuracy by 2.12%.” · Show in paper
Table 7 confirms that ARCQuant improves INT4 average accuracy from 65.78 to 67.90 (+2.12%) and reduces PPL from 8.84 to 7.95 (-0.89), and also improves MXFP4.
“Stage 0: Adapter warmup. We freeze both the encoder and decoder and update only the adapter. • Stage 1: Encoder training. We unfreeze the encoder while keeping the decoder frozen. • Stage 2: End-to-end continual pre-training. We unfreeze the decoder with a small learning rate. • Stage 3: Supervised fine-tuning. We train on the SFT mixture and increase the decoder learning rate.” · Show in paper
“The per-group QQQ demonstrates competitive performance with with W8A8 SmoothQuant, W4A16 GPTQ, and W4A16 AWQ across various models. For example, per-group QQQ only increases perplexity by up to 0.13 on LLaMA-2-13B compared with them. When benchmarked against W4A4 Atom, both the per-channel and per-group QQQ consistently outperform Atom on all tested models.” · Show in paper
Table 1 and Table 2 directly substantiate the perplexity and zero-shot accuracy comparisons against SmoothQuant, GPTQ, AWQ, and Atom.
“Storage: spending the same CPU memory on exact KV for a subset of spans costs 2.9 to 3.5 points, because low-rank factors cover every span, and a fixed rank of 32 matches the learned rank in accuracy but needs 1.6× the CPU memory.” · Show in paper
Table 3 directly supports both the 2.9–3.5 point drop for exact KV at matched CPU bytes and the ~1.6x CPU memory increase for fixed rank 32.
“Figure 2(d) confirms the amplification empirically: a network-wide average mass below 0.08 produces over 20% relative error on average, with peaks above 60% in the shallowest layers.” · Show in paper
Figure 2(a)-(d) directly reports these exact averages and peak values.
“gpt-oss-120b surpasses OpenAI o3-mini and approaches OpenAI o4-mini accuracy. The smaller gpt-oss-20b model is also surprisingly competitive, despite being 6 times smaller than gpt-oss-120b.” · Show in paper
The reported benchmark scores in Figures 1 and 2 and Table 3 directly support that gpt-oss-120b exceeds o3-mini and approaches (or slightly exceeds on HLE and Tau-Bench Retail) o4-mini, while gpt-oss-20b achieves competitive scores.
“SnapKV achieves the best throughput, on par with All-KV on Llama and even surpassing All-KV on Mistral.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingFigure 3 directly supports that SnapKV-0.75 achieves the highest average throughput among all evaluated methods on both models.
“Each MoE block consists of a fixed number of experts (128 for gpt-oss-120b and 32 for gpt-oss-20b), as well as a standard linear router projection which maps residual activations to scores for each expert. For both models, we select the top-4 experts for each token given by the router, and weight the output of each expert by the softmax of the router projection over only the selected experts.” · Show in paper
“FFT and QLoRA r = 8 retain OOD performance well but do not reach the highest paraphrase accuracy. Higher-rank QLoRA configurations achieve stronger paraphrase performance only with larger OOD losses. This suggests that the apparent robustness of low-rank adaptation to forgetting is actually partly due to limited plasticity.” · Show in paper
Table 2 directly supports the claim that low-rank QLoRA and FFT do not reach the highest acquisition targets, while higher ranks do so at the cost of OOD retention.
“Table 2 lists all completed pairs: in 9 of 12 only the burst run degraded persistently, in 0 only the no-burst run did, 2 degraded in both arms and 1 in neither (two-sided sign test over discordant pairs, p = 0.00391).” · Show in paper
The paired design, statistical test, and control experiments cleanly establish history-dependent persistent degradation on vLLM.
“Training at scale confirms the choices from the small-scale sweep: both the continual pre-training losses (Figures 9 to 11) and the downstream ablations (Appendix F.6) again favor an encoder window W = 1024 and causal masking. At scale, the attention-based adapter is competitive with the MLP adapter; we keep the MLP adapter, which adds less computation. The choice of pooling operator depends on the compression ratio: mean pooling has a small edge over concatenation at 16x compression, while concatenation slightly outperforms mean pooling at 4x.” · Show in paper
Both the from-scratch pre-training sweeps and the full-scale continual pre-training and downstream ablations support these architectural conclusions.
“our key idea to solve this problem is to group key cache every G tokens and quantize them separately. Because the number of tokens in X_K can be arbitrary, we split X_K into two parts, namely, the grouped part = X_K[: l - r] and residual part = X_K[l - r :], where l is the number of tokens inside the current key cache X_K, r is the number of residual tokens, where l - r can be divisible by G.” · Show in paper
“We use the SWE-Agent [16] scaffold. ... For open-source LLMs, we use vllm to host each model. Models are hosted on a single node, with 8 H100 Nvidia GPUs. ... Models have a maximum of 50 turns. We use the same prompt for all models, which is the default prompt from [16].” · Show in paper
“Figure 8 demonstrates the results on Llama3.1-8B-Ins across multiple different benchmarks. In all cases, HSA consistently outperforms Quest and SparQ, especially at low token budgets.” · Show in paper
Figure 8 and Tables 2, 6-9 confirm that standalone HSA outperforms Quest and SparQ across the evaluated benchmarks on Llama3.1-8B-Ins.
“Gψ,j(z) = Pj z̄ + bj + αj Dj σ(Uj z̄), z̄ = LN(z)” · Show in paper
“Throughput is the binding constraint: each query costs 3-20 s upstream and yields roughly 20-45 characters, so a complete text of this size needs on the order of tens of thousands of requests.” · Show in paper
The scaling estimate follows directly from the observed per-query character yield and the total book length.
“For methods that require calibration, we sample 128 sequences of 2,048 tokens from the first WikiText shard (Merity et al. 2016), following a widely used PTQ setup (Frantar et al. 2023).” · Show in paper
“We run a separate reasoning experiment on a 94k-example subset of OpenR1-Math-220k (Hugging Face, 2025) to test whether the OSM trend also appears in a larger skill-adaptation regime. We evaluate Pass@1 on MATH-500 (Hendrycks et al., 2021), AIME’24 and AIME’25 (Mathematical Association of America, 2024), AMC’23 (American Mathematics Competitions, 2023), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). OOD degradation uses the same five-benchmark average as the main experiment, relative to the base Qwen3-4B” · Show in paper