Search publications
“Only when we choose α from the sweet spot region (0.4-0.6) can we get small quantization errors for both weights and activations, and maintain the model performance after quantization.” · Show in paper
Figure 10 directly demonstrates the accuracy drop outside the 0.4–0.6 sweet spot for OPT-175B.
“We quantize the inputs and weights of compute-heavy operators like linear layers and BMM in attention layers with INT8, while keeping the activation as FP16 for other lightweight element-wise operations like ReLU, Softmax, and LayerNorm.” · Show in paper
“The approximation error scales as O(\sigma^2) where \sigma = \max_{i\in\mathcal{E}} |\delta_i|, and a formal bound is given in Appendix A.” · Show in paper
The derivation in Appendix A.1-A.2 is mathematically sound and establishes both the O(\sigma^2) error bound and O(d^2) storage complexity.
“Concretely, we apply the algorithm to B = 128 columns at a time, keeping updates contained to those columns and the corresponding B × B block of H−1 ... For smaller models, applying dampening, that is adding a small constant λ (we always choose 1% of the average diagonal value) to the diagonal elements of H ... Algorithm 1 Quantize W given inverse Hessian H−1 = (2XX> + λI)−1 and blocksize B.” · Show in paper
“The unquantized Qwen3-1.7B control shows the same qualitative monotonic trade-off: paraphrase EM rises from 76% at r = 8 to 79% at r = 16 and 86% at r = 32, while average OOD performance falls from 57.0% to 52.0% and 40.2%, respectively. This suggests that quantization is not required for the qualitative rank trend, although this reduced control changes model scale.” · Show in paper
The reported numbers on Qwen3-1.7B demonstrate the same monotonic trade-off without quantization, and the authors appropriately qualify the difference in model scale.
“Overall, FP4Quant enables efficient low-bit communication with substantial acceleration under TP while incurring negligible runtime overhead under PP, demonstrating its effectiveness across diverse parallel configurations.” · Show in paper
Table 2 supports both the TP speedup and the negligible overhead under PP.
“For each action, the planner estimates accuracy impact by combining three factors: offline global sensitivity, online routing statistics, and runtime prompt residual.” · Show in paper
“Higher mean performance is associated with smaller variation (Figure 3(a)): Spearman’s ρ = −0.76, or −0.67 excluding Qwen. Full delivery remains sensitive, however: GPT Sol has the highest mean and an SD of only 1.93, yet its full-credit rate varies by 9.12 percentage points. Both GLM variants have full-credit gaps of 13.50 points.” · Show in paper
Table 9 and Figure 3(a) directly support both the negative correlation between mean score and SD and the persistent sensitivity of R1.
“We run vLLM 0.20 (ROCm build 0.20.2rc1.dev253) with automatic prefix caching on single AMD MI300X GPUs, one engine per GPU, BF16 weights and KV cache, default scheduler (chunked prefill, FCFS), 40K maximum context, greedy decoding with a fixed output length per turn... Both runs of a pair share one base arrival stream: every base session arrives at the same time with the same identity and shape. The burst run additionally receives an independent stream of extra sessions (rate 1.5λ for 5 minutes, i.e. 2.5× total arrivals) starting at minute 15.” · Show in paper
“Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, our approach reduces P95 TTFT by upto 81% and raises SLO attainment by upto 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.” · Show in paper
The evaluation results in Section 6 (Figures 4-6) and Section 4.4/5.3 support the improvements in P95 TTFT, SLO attainment, and sub-millisecond routing overhead, though Section 9 (Conclusion) states 'upto 71%' SLO attainment improvement rather than 79%.
“In 12 of the 30 settings, the selected configuration uses only the first stage of training (Appendix C), so Md decodes with exactly the weights of its fine-tuned baseline, and BridgeKV still scores higher in all 12. In these settings, the gain comes from the stitched cache alone, not from extra trainable parameters on the decode side.” · Show in paper
Table 9 confirms 12 settings use S1 only, and Figure 3 confirms higher scores for BridgeKV in all of them.
“Yes, the underlying principles are well-known in LLM systems engineering and agentic orchestration, though the specific name in your text ("O(n) Segmented State-Machine Streaming") is a bespoke, prompt-level branding of several established patterns. In academic literature and production AI engineering, this exact strategy does not live under a single unified name—instead, it is a composite of four formalized concepts:” · Show in paper
The mapping between the protocol's mechanisms (schema compression, checkpoint markers, coordinate indexing, and chunking) and established LLM engineering patterns is clearly articulated and supported by the text.
“In this setting, we find that models generally perform worse at this task than when generating patch files; for instance, Claude 2 scores at 2.2% compared to 4.8% in the main table for “oracle” retrieval. Even when controlling for instance length, generating on the shorter half of the task instances by input tokens yields 3.9% compared to 7.8% for generating patches with Claude 2.” · Show in paper
The reported comparison for Claude 2 (2.2% vs. 4.8% overall, and 3.9% vs. 7.8% on shorter instances) supports the claim.
“Table 4 shows that the full PagedWeight system achieves the lowest perplexity. Removing routing statistics or prompt residuals degrades perplexity to 7.26/10.13 and 7.31/10.19, showing that both online measurements help. Removing page movement turns PagedWeight into a static mixed-precision plan and further increases perplexity to 7.43/10.33. Removing global sensitivity falls back to uniform quantization and performs worst.” · Show in paper
Table 4 directly substantiates that removing any single component increases perplexity on both Wikitext2 and C4.
“Overall, these results indicate that KV cache compression techniques introduce only limited additional latency for the first token, with the exception of TurboQuant, whose more complex quantization pipeline leads to noticeable overhead.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingTable 3 and Figure 2 support the claim for the evaluated datasets, though GovReport results for TurboQuant are omitted due to 3-day single-run execution times.
“In NVFP4, each fused matrix uses a single tensor-level weight scale and a common input-activation scale. To reproduce this inference-time behavior, QATFactory ties the corresponding scales during training: it computes the shared weight scale from the concatenated projection weights and tracks one activation scale for their common input.” · Show in paper
“FlashInfer guarantees both attention and contraction stage are compatible with CUDAGraphs (Gray, 2019; Nguyen et al., 2021). Both attention and contraction stage use persistent kernel and the grid size is fixed once compiled, which means the kernel is launched with the same grid size for each generation step. We set the fixed offset for each section of the workspace buffer to store partial outputs and plan information to make sure the pointers passed to the kernel are the same for each generation step, meeting the requirement of CUDAGraphs” · Show in paper
“Dropping the residual entirely (rank-only compression) raises 2× perplexity from 6.54 to 7.27, a gap that additional rank cannot close.” · Show in paper
Table 11 directly supports both the necessity of the rotated residual and the saturation at 4 bits.
“We identify an outlier-induced scale mismatch in hierarchical FP4 quantization: default range-preserving scales can reserve scarce FP4 levels for regions that receive no assignments, reducing resolution for dense weight values.” · Show in paper
Figure 1a-c directly illustrates the unoccupied extreme levels under default scaling, the reduced element-wise error in the dense region, and the perplexity reduction achieved by SoScale compared to ScaleSearch and RTN.