Search publications
“As shown in Figure 9 (top), compared to FasterTransformer’s FP16 implementation of OPT, SmoothQuant-O3 can further reduce the execution latency of OPT-13B and OPT-30B by up to 1.56× when using a single GPU. This is challenging since FasterTransformer is already more than 3× faster compared to the PyTorch implementation for OPT-30B. Remarkably, for bigger models that have to be distributed across multiple GPUs, SmoothQuant achieves similar or even better latency using only half the number of GPUs (1 GPU instead of 2 for OPT-66B, 4 GPUs instead of 8 for OPT-175B).” · Show in paper
Figures 8 and 9 and Table 8 directly report the measured speedups, memory reductions, and GPU count reductions.
“LCLMs achieve the fastest compression time at every context length and substantially lower peak memory at long contexts; all baselines run out of memory at 512K tokens, except Attention Matching, which fails at 512K due to numerical instability in its linear solver and runs out of memory at 1M.” · Show in paper
Figure 4 and Section 6.2 clearly document the scaling measurements and memory limits on the H200 GPU.
“As shown in Figure 5, with similar maximum memory usage, KIVI enables up to 4× larger batch size and gives 2.35× ∼ 3.47× larger throughput.” · Show in paper
Figure 5 and Section 4.2.4 support the 4x batch size and 2.35x ~ 3.47x throughput improvement for Llama-2-7B on an A100 80GB GPU.
“Finding 3: LoRA significantly reduces training memory requirements: for Qwen3.5 9B at an 8K sequence length, rank-16 LoRA reduces GPU memory usage by 2.9×. However, LoRA underperforms full-parameter QAD, and increasing the adapter rank does not close the gap.” · Show in paper
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMsMemory measurements in Table 4 and rank-scaling evaluations in Tables 5 and 6 fully support the claim.
“while transferring 3× less data per decoding step than ShadowKV and using the least total memory among the methods we profile. At 128K context, it halves peak GPU memory and cuts total memory by one third relative to full KV caching.” · Show in paper
Tables 2 and 20 directly confirm the 3.1x transfer reduction vs. ShadowKV, lowest total memory (46.01 GB), ~50% peak GPU memory reduction, and ~34% total memory reduction vs. Full KV.
“QLoRA INT8 and NF4 configurations achieved 52.8% and 53.2% exact-match accuracy respectively, compared to 53.4% for the FP16 LoRA r=8 baseline with the same {q,v} module set. The accuracy cost of quantization is therefore minimal (0.2–0.6 points), while the memory savings are substantial: both QLoRA configurations require only 0.60 GB peak training memory, a 62% reduction relative to FP16 LoRA (1.59 GB) and a 74% reduction relative to full fine-tuning (2.31 GB). Inference latency is higher for INT8 (921 ms/query) than NF4 (614 ms/query) and FP16 LoRA (around 380 ms/query), reflecting the overhead of de-quantization on a T4 GPU. NF4 represents the best quantization trade-off: near-FP16 accuracy at 0.60 GB memory with 34 tok/s throughput.” · Show in paper
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter ModelTable 2 and Section 5.5 support the accuracy, memory reduction, and latency comparisons.
“SpecStream achieves average throughput speedups of 1.40× and 1.23× over TriForce on Qwen3 and InternLM2.5, respectively.” · Show in paper
Figure 6 and Section 5.2.2 support both the throughput speedups over TriForce and the nuanced limitation regarding Draft KV Cache residency.
“In early experiments, we consider directly training the full model end-to-end from the beginning, but find that this approach underperforms the multi-stage recipe... We also consider parameter-efficient variants that keep the decoder fully frozen (Ge et al., 2023; Cheng et al., 2024; Yen et al., 2024) or use LoRA (Liao et al., 2025; Feldman and Artzi, 2025), but we find them to substantially underperform compared to full-parameter training in early experiments; we therefore focus on end-to-end training. With compression disabled, the trained decoders show minimal drift from the original model (Appendix D.3).” · Show in paper
Table 6 quantitatively verifies minimal decoder drift when compression is disabled; the early-experiment comparisons against direct end-to-end, frozen-decoder, and LoRA training are reported qualitatively in Section 4.2.
“On Mistral-7B-v0.3 at context length 8,192, peak decode memory of the KV representation is 1.43×, 2.40×, 3.61×, and 4.73× below the dense bf16 cache at the 2×, 3×, 5×, and 8× compression targets.” · Show in paper
The memory savings and latency overhead are both transparently reported in Section 4.5.
“All experiments also use a set of shared levers, applied to every method: SDPA attention, LoRA dropout 0, length-grouped dynamic batching, and a fused AdamW.” · Show in paper
“Experiments show that SpecStream maintains task quality close to SGLang speculative decoding while supporting more concurrent requests under limited GPU memory. Across different datasets, it achieves average throughput speedups of 1.41× and 1.32× over the offloading baseline for Qwen3 and InternLM2.5, respectively. Compared with parallel speculative decoding on separate Target and Draft GPUs, SpecStream improves output throughput per GPU by an average of 55.4%.” · Show in paper
All components of the headline claim—task quality preservation, concurrency under limited GPU memory, throughput speedups over TriForce, and per-GPU throughput gains over separate-GPU parallel speculation—are directly supported by the reported experimental results.
“The improvement of vLLM over Orca (Oracle) on OPT-13B and the Alpaca dataset goes from 1.3× in basic sampling to 2.3× in beam search with a width of 6. Fig. 15 plots the amount of memory saving, computed by the number of blocks we saved by sharing divided by the number of total blocks without sharing. We show 6.1% - 9.8% memory saving on parallel sampling and 37.6% - 55.2% on beam search. In the same experiments with the ShareGPT dataset, we saw 16.2% - 30.5% memory saving on parallel sampling and 44.3% - 66.3% on beam search.” · Show in paper
Figures 14 and 15 directly support the Alpaca throughput and memory saving numbers. The ShareGPT memory saving numbers (16.2%-30.5% and 44.3%-66.3%) are reported in text without a corresponding figure, but are consistent with ShareGPT's longer prompts.
“Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU.” · Show in paper
Tables 6 and 7 substantiate the 99.3% relative score and top open-source ranking on the Vicuna benchmark.
“The attention rows compare BF16 SDPA with the NVFP4 Q/K body; QKV and output projections are counted in the complete-step table. Ours is 1.27× faster and uses 51.1%/52.9% less incremental allocation.” · Show in paper
All quality, error-analysis, and native-hardware speedup/memory claims for Q-Attn match the reported tables and figures.
“Across six model pairs and five datasets, BridgeKV outperforms fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings. It closes up to 80% of the quality gap to using the larger model for both phases. On one A40 GPU, Llama-3.1-Instruct-8B→Llama-3.2-Instruct-1B achieves 3.0× faster requests than the larger model alone, with latency close to the smaller model alone. BridgeKV can improve the hardware and system solutions that separate prefill and decode phases across GPUs. A separate serving simulation models request arrivals, queues, and cache transfer. For Llama-3.1-Instruct-70B→Llama-3.2-Instruct-1B, it predicts up to 33.86× faster requests and 27.5× lower energy use than using the larger model for both prefill and decode phases.” · Show in paper
All components of the overarching claim are directly supported by Figure 3, Figure 1, Section 4.3, and Table 7.
“LoRA-SVD parameterizes a training-time weight as W = Wres + βL2L1. It preserves a compact principal subspace and its activation path in BF16 while computing the full-shape residual in NVFP4.” · Show in paper
“To address these challenges, we propose ARCQuant, a framework that boosts NVFP4 performance via Augmented Residual Channels. Distinct from methods that compromise block isolation or hardware uniformity, ARCQuant maintains a strictly unified NVFP4 format by augmenting the activation matrix with quantized residual channels. ... Extensive experiments on LLaMA and Qwen models demonstrate that ARCQuant achieves state-of-the-art accuracy, comparable to full-precision baselines in perplexity and downstream tasks. Furthermore, deployment on RTX 5090 and RTX PRO 6000 GPUs confirms practical benefits, achieving up to 3× speedup over FP16.” · Show in paper
The experimental results in Table 1, Figure 6, and Table 10 directly support the accuracy and speedup claims on LLaMA and Qwen models across RTX 5090 and RTX PRO 6000 GPUs.
“As shown in Fig. 18a, this leads to 20–26% higher attention kernel latency, compared to the highly-optimized FasterTransformer implementation. We believe the overhead is small as it only affects the attention operator but not the other operators in the model, such as Linear. Despite the overhead, PagedAttention makes vLLM significantly outperform FasterTransformer in end-to-end performance (§6).” · Show in paper
Figure 18a and Section 6 end-to-end results support this trade-off.
“The per-component decomposition reveals why: projection GEMMs become compute-bound in prefill (OI up to 2,956 at C=80), while attention, elementwise, and all decode components stay bandwidth-bound (Table 14). The resulting prefill-to-decode OI gap spans 8-28x across workloads (Table 13).” · Show in paper
Tables 13 and 14a and Figure 2a provide the exact component-level OI values, latency shares, and prefill-to-decode gaps.
“We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks.” · Show in paper
The reported 400x compression ratio, 3.7x A100 speedup, 32.6% peak memory reduction, and near-Full-KV accuracy are directly supported by the experimental results in Section 4.2, Section 4.3, Figure 5, Figure 7, and Table 2.