Search publications
“(2) Double Quantization, a method that quantizes the quantization constants, saving an average of about 0.37 bits per parameter (approximately 3 GB for a 65B model).” · Show in paper
The mathematical derivation on page 5 is exact, and Figure 3 confirms that Double Quantization preserves zero-shot accuracy.
“Table 3 compares the training cost of the KV-stitching module with that of the parameter-matched PEFT baseline, which fine-tunes the decode anchor alone with LoRA at the same number of trainable parameters as the stitched model. Training the module is cheaper in every setting.” · Show in paper
In all 16 rows of Table 3, the sum of Stage 1 and Stage 2 GPU-hours is strictly less than the PEFT baseline GPU-hours.
“For moderate levels of parallel generation (4 ≤ n ≤ 32), FlashInfer’s composable formats yield consistent speedups for both ITL and TTFT. Peak speedups occur at n = 4, where ITL decreases by 13.73% for the 8B model and 17.42% for the 70B model, while TTFT is reduced by 16.41% for the 8B model and 22.86% for the 70B model.” · Show in paper
Figure 10 confirms the exact percentage reductions at n=4 and consistent speedups for 4 <= n <= 32.
“We use Qwen3-4B (Yang et al., 2025a) as the base model and compare full fine-tuning (FFT) with QLoRA adapters of rank r ∈ {8, 16, 32, 64}. Each training example consists of a question and its gold answer, with the autoregressive loss applied only to answer tokens. All runs are repeated over five random seeds.” · Show in paper
“LoRA-SVD reduces the error of outputs, input gradients, and weight gradients throughout the observed training trajectory. In a separate paired-forward check on real text, quantized and unquantized forwards from the same checkpoint and input show 29.8–33.0% lower final-output MSE with LoRA-SVD across d = 512–2048.” · Show in paper
Figure 2 visually supports the ~10% relative MSE reduction across the 1B trajectory, and Section 3.2 reports the 29.8–33.0% final-output MSE reduction.
“At ≈ 2.2 bit (group-size 128; using FP16 scale and 2-bit zero point per group) the perplexity increase is already less than 1.5 points, while dropping to 0.6 - 0.7 at ≈ 2.6 bit (group-size 32), which is only slightly worse than vanilla 3-bit and might be interesting for practical kernel implementations. Further, if we reduce group size to 8, we can apply ternary (-1, 0, +1) quantization, which achieves 9.20 WikiText2 PPL on OPT-175B, a less than 1 point drop.” · Show in paper
Table 7 directly supports the 2-bit grouped perplexity numbers; the ternary group-size 8 result (9.20 PPL) is reported in the text.
“At 64k tokens, the 16x LCLM reaches the first token faster than the smallest uncompressed models (0.21 s, compared with 0.81 s for Llama 3.2 1B and 0.97 s for Qwen3-0.6B) while being far more accurate on LongHealth5 (66.25% vs. 22.50% and 32.75%). The 4x LCLM exceeds the uncompressed Qwen3-4B-Instruct-2507 on LongBench with a 6x lower TTFT.” · Show in paper
The vLLM-served benchmarks in Figure 5 and Section 6.2 directly substantiate the reported TTFT and accuracy comparisons.
“For the second moment, direct NVFP4 gives 0.0116, whereas the complete pipeline gives 0.0053, a 54.3% reduction. ... In separate 1B native-state ablations, retaining NVFP4 state iteration while removing any required preprocessing stage caused rapid instability or divergence, so no comparable endpoint loss existed. To our knowledge, this is the first demonstration of stable NVFP4 AdamW momentum-state storage in an integrated 1B pretraining run.” · Show in paper
The reconstruction error reductions, bit-width accounting, and 1B/20B-token pretraining stability are all directly supported by the reported data.
“We use this feature to allocate paged memory for the optimizer states which are then automatically evicted to CPU RAM when the GPU runs out-of-memory and paged back into GPU memory when the memory is needed in the optimizer update step.” · Show in paper
“In Figure 12 (a), we see that increasing the size of the decoder is much more beneficial in terms of pre-training loss than increasing the size of the encoder.” · Show in paper
Figure 12(a) and Table 7 directly support both the pre-training loss reduction and the mixed downstream results.
“We implement SpecStream on SGLang 0.5.17 [21] and evaluate two Target/Draft model pairs, Qwen3-32B/0.6B [14] and InternLM2.5-20B-Chat/1.8B-Chat [15]. The main experiments use two NVIDIA A800 80GB PCIe Gen4 GPUs and 240GB of host memory.” · Show in paper
“From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using 2.6× less peak memory (including model weight). This reduction in memory usage enables up to 4× larger batch size, bringing 2.35× ∼ 3.47× throughput on real LLM inference workload.” · Show in paper
The headline claim overgeneralizes: Section 4.2.2 explicitly notes that Falcon-7B needs 4bit KIVI to maintain accuracy and suffers a large accuracy drop with 2bit KIVI, and the 2.6x memory reduction and throughput gains were only measured on Llama-2-7B.
“We find that in some settings SWE-Llama 13b is competitive with Claude 2 and is capable of processing contexts exceeding 100,000 tokens.” · Show in paper
In the Oracle retrieval setting (Table 18), SWE-Llama 13b achieves 3.97% resolved vs. Claude 2's 4.80%, and Table 4 confirms its >=100,000 token context capacity.
“Comprehensive kernel-level and end-to-end evaluations demonstrate FlashInfer’s ability to significantly boost kernel performance across diverse inference scenarios: compared to state-of-the-art LLM serving solutions, FlashInfer achieve 29-69% inter-token-latency reduction compared to compiler backends for LLM serving benchmark, 28-30% latency reduction for long-context inference, and 13-17% speedup for LLM serving with parallel generation.” · Show in paper
All three headline percentage ranges match the empirical results reported in Sections 4.1, 4.3, and 4.4.
“We follow the same training procedure as [Bui et al., 2025] described in Section 2.2 but replace the local capacity constraint by a global counterpart Lcap = ∑T t=1 max(0, ∑ℓ,h ∑t i=1 βt−i ℓ,h,i − Mglobal), where Mglobal is the target global KV budget. We only train retention gates and freeze LLM weights.” · Show in paper
“Across the two model pairs, SpecStream achieves an average throughput speedup of 1.036× over SPECTRE+Offload. Because SPECTRE uses one additional GPU, normalizing by GPU count gives SpecStream an average improvement of 55.4% in output throughput per GPU.” · Show in paper
Figure 10 and the GPU count normalization (3 GPUs vs. 2 GPUs, 1.036 * 3/2 ≈ 1.554) directly support the 55.4% per-GPU throughput improvement.
“Figure 6 confirms substantial gains over FP16: Qwen2.5-7B achieves 2.0×–2.5× speedup (PRO 6000) and Llama 3.1-8B reaches 3.5× (RTX 5090), while memory usage drops by 1.5×–2.8×. Crucially, residual overhead is marginal: compared to uncompensated NVFP4, latency increases by only 3%–9%.” · Show in paper
The latency and memory numbers in Figure 6, Figure 8, and Table 10 support the reported speedup, memory reduction, and 3%–9% overhead over uncompensated NVFP4.
“For each span and head, we factorize the pre-RoPE key matrix and the value matrix with a truncated SVD up to rank , and a lightweight predictor assigns each component a gate from the singular spectrum and the span summary: ” · Show in paper
“In vLLM, we adopt the first-come-first-serve (FCFS) scheduling policy for all requests, ensuring fairness and preventing starvation. ... we implement an all-or-nothing eviction policy, i.e., either evict all or none of the blocks of a sequence. Furthermore, multiple sequences within one request (e.g., beam candidates in one beam search request) are gang-scheduled as a sequence group.” · Show in paper
“The first stage trains only ψ, with both anchors frozen, and combines two losses: an alignment loss that keeps the stitched cache close to the native cache of Md, and a distillation loss that makes Md, reading the stitched cache, match the next-token distribution of Mp.” · Show in paper