Search publications
“Newer models generally benefit less from parallel sampling. On all five benchmarks, the gain from parallel sampling decreases with model release date (Table 7; Figure 4B): TerminalBench (τ = −0.73), FrontierMath (−0.60), HealthBench (−0.47), HLE (−0.47), and most weakly SWE-Bench Pro (τ = −0.33).” · Show in paper
The tau values in the text match Table 7, though as the authors note elsewhere, each correlation is computed over only six models.
“To summarize, QLORA has one storage data type (usually 4-bit NormalFloat) and a computation data type (16-bit BrainFloat). We dequantize the storage data type to the computation data type to perform the forward and backward pass, but we only compute weight gradients for the LoRA parameters which use 16-bit BrainFloat.” · Show in paper
“Finding 1: QATFactory converges stably across model families, parameter counts from 8B to 230B, and NVFP4, MXFP4, and Q4_K. In vLLM and llama.cpp, the exported QAD checkpoints achieve higher average benchmark accuracy and lower mean KL divergence than the RTN and GPTQ post-training quantization baselines.” · Show in paper
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMsAll reported training curves, benchmark accuracies, and KL divergence metrics in Figures 3-4 and Tables 2-3 directly support this claim.
“An LCLM consists of an encoder with a pooling method that maps token chunks to soft tokens, an adapter to project to the decoder's embedding dimension, and a decoder that consumes the soft tokens as latent context.” · Show in paper
“We decompose each decoder layer into constituent operations and compute component-level FLOPs and DRAM bytes from model dimensions, precision, tensor-parallel degree, active token count, and batch size. For component j, the operational intensity is OIj = FLOPsj/DRAMBytesj, and the roofline throughput ceiling is: Proof_j = min (Ppeak, OIj x BWpeak), where Ppeak is peak compute throughput (FLOP/s) and BWpeak is peak memory bandwidth (byte/s) for the target GPU. We convert the ceiling into a component latency lower bound, troof_j = FLOPsj/Proof_j, and aggregate component costs to obtain a layer-level roofline bound.” · Show in paper
“Table 2 assigns NVFP4 to the QK⊤ and dS paths and retains the softmax-sensitive P V , P⊤dO, and dOV ⊤ products in BF16. Quantized views are reused between forward and backward to avoid precision mismatch and repeated memory traffic.” · Show in paper
“Combining Hadamard preprocessing with SoScale is most useful for MXFP4, where it gives the best average score for both Qwen3-8B and Qwen3-14B, and it also gives the best NVFP4 result on Qwen3-14B. However, the best row is not always the Hadamard variant: Qwen3-8B under NVFP4 and Qwen3-14B under HiF4 are best with SoScale alone.” · Show in paper
Every comparison in the claim is directly supported by the bolded and underlined average scores in Table 2.
“It matches exact-recall offloading caches on LongBench and InfiniteBench and trails on synthetic exact retrieval” · Show in paper
Table 1 supports the claim: SEKV is slightly ahead on LongBench (+0.54) and nearly level on InfiniteBench (-0.25), while trailing on RULER (-3.32) and NIAH (-4.28).
“Across all three models and tasks, PagedWeight always reaches high quality with the lowest memory consumption (Figure 5). These results show that selecting routed expert pages to reduce is more effective than using one static quantization precision policy.” · Show in paper
Figure 5 clearly demonstrates that PagedWeight achieves lower perplexity and higher reasoning accuracy at equivalent or lower memory consumption across all three MoE models and four benchmarks.
“Adding recovery to the MSE objective gives mixed gains over ScaleSearch: it is marginal under MXFP4, decreases NVFP4, and improves HiF4. In contrast, full SoScale is best for all three formats, indicating that recovery is most effective when coupled with Hessian-aware scale-code selection rather than a local MSE objective.” · Show in paper
The numerical results in Table 4 match the authors' characterization for all three formats.
“We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme.” · Show in paper
Figure 6 and Table 4 on SCBench directly demonstrate that RocketKV-MT outperforms all non-oracle baselines and closely matches Exact-TopK across all tested token budgets.
“Table 4 reports speed over the thirteen kept sessions. AQLoRA-s averages 11.3% faster than QLoRA with a between-session standard deviation of 2.3, and it is faster in every session, by 7.0 to 15.2%; the smallest margin exceeds every floor.” · Show in paper
The 11.3 ± 2.3% speedup and 7.0% to 15.2% range are directly supported by Table 4 and sensitivity analysis in Table 20.
“Table 1 lists all evaluated configurations; each changes one variable relative to the LoRA r=8, {q,v}, FP16 reference.” · Show in paper
“FlashInfer’s kernel significant outperforms FlashAttention kernels in uniform and skewed sequence length distributions because of our load-balanced dynamic scheduler (section 3.3.1). FlashInfer’s decode attention outperforms FlashAttention kernels because our versatile tile size selection (section 3.2.2) and FlashAttention use suboptimal tile size for decoding.” · Show in paper
Figure 8 and the ablation in Tables 6–7 support the performance advantage and attribute it to the load-balanced scheduler and tile size selection.
“Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 ×, 2.10×, and 1.25× compared to FP16, W8A8, and W4A16, respectively.” · Show in paper
The reported perplexity, zero-shot accuracy, and vLLM throughput benchmarks on LLaMA models support the claimed parity with SOTA quantization methods and the specific speedup figures.
“In both PyTorch Huggingface and FasterTransformer frameworks, we implement INT8 linear modules and the batched matrix multiplication (BMM) function with CUTLASS INT8 GEMM kernels. We simply replace the original floating point (FP16) linear modules and the bmm function with our INT8 kernels as the INT8 model.” · Show in paper
“We also compare our rule against the only deployed alternative we know of: Unsloth’s released dynamic-4bit selection for Qwen2.5-7B, a hand-curated list covering 34 of the 196 linear layers. Substituting their list for ours in an otherwise identical run gives 89.0 on the 8-task average, matching our 89.0.” · Show in paper
The direct substitution experiment on Qwen2.5-7B supports the parity claim.
“Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94× throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.” · Show in paper
The reported figures match the experimental tables: (35.25 - 9.86)/35.25 = 72.0% memory savings, 130.1/67.1 = 1.94× throughput, (17.0 - 12.2)/12.2 = 39.3% quality gain, and (401.9 - 385.4)/401.9 = 4.1% max throughput loss.
“For an overall compression ratio of c, we define a split factor r so that c is split into c^r for the first stage and c^(1-r) for the second stage, where 0 <= r <= 1. We use the following formula to adaptively determine r: r = min(0.2 + 0.06 * log2(c), 0.8)” · Show in paper
“The benefit of BridgeKV is more pronounced when there is a large gap between the prefill anchor and decode anchor, especially for output-heavy workloads. As it is shown in Table 7, the benefit can go from 1.75× (1.76×) to 33.86× (27.5×) in terms of response time and energy savings.” · Show in paper
Table 7 directly supports the reported range of response speedups and energy savings.