Search publications
“We address this problem by developing a quantized-matrix full-precision-vector product kernel which performs a matrix vector product by dynamically dequantizing weights when needed. Most notably, this does not require any activation quantization.” · Show in paper
“The next lemma is the constructive finite-dimensional Carathéodory argument for preserving an expectation. Its connection with moment matching is classical [3]; we include the exact elimination and its bit bound because both are needed by the algorithm.” · Show in paper
“Only when we choose α from the sweet spot region (0.4-0.6) can we get small quantization errors for both weights and activations, and maintain the model performance after quantization.” · Show in paper
Figure 10 directly demonstrates the accuracy drop outside the 0.4–0.6 sweet spot for OPT-175B.
“For each linear layer ℓ with weights Wℓ, flattened into blocks b of 64 elements (matching bitsandbytes), εℓ = 1/|Wℓ| ∑_b ||b − absmax-NF4^-1(absmax-NF4(b))||_2^2, sℓ = Pr[|Wℓ,ij| < τ], with τ=0.01 and the true 16-level NF4 quantile grid. The pass runs on CPU over the fp16 checkpoint, is data-free, and requires no forward pass.” · Show in paper
“In Fig. 7, streaming access for the full query group yields the largest throughput gain. With 16K inputs, K2 improves throughput over K1 by an average of 83.2% on Qwen3 and 117.1% on InternLM2.5.” · Show in paper
The ablation results in Figure 7 and Section 5.3.1 directly substantiate the individual and cumulative gains of streaming verification.
“As shown in Table 3, SmoothQuant can match the FP16 accuracy on all evaluation datasets with all quantization schemes. LLM.int8() can match the floating point accuracy because they use floating-point values to represent outliers, which leads to a large latency overhead (Table 11). The W8A8, ZeroQuant, and Outlier Suppression baselines produce nearly random results, indicating that naively quantizing the activation of LLMs will destroy the performance.” · Show in paper
Table 3 provides direct empirical evidence that SmoothQuant-O1/O2/O3 closely match FP16 accuracy and perplexity on OPT-175B while the three baselines degrade to near-random performance.
“Table 2 lists all completed pairs: in 9 of 12 only the burst run degraded persistently, in 0 only the no-burst run did, 2 degraded in both arms and 1 in neither (two-sided sign test over discordant pairs, p = 0.00391).” · Show in paper
The paired design, statistical test, and control experiments cleanly establish history-dependent persistent degradation on vLLM.
“Proposition 4.2 (Uniform scheduling bound). There is one polynomial p, independent of the instance, such that preprocessing uses at most p(L) bit operations and serving request t uses at most p(L + ⌈log2(t + 1)⌉) bit operations, for every reachable history and internal state. Activation occurs at a finite deterministic time t∗ depending only on (n, k, d).” · Show in paper
The capped simulation schedule and output-size bound S = O(g(t)+1) cleanly link the constructor's arbitrary finite runtime T_P to the uniform polynomial per-request bound.
“Lemma 12.4 (Bounded-displacement matching). Let Λ be a full-rank lattice in R^2 and let T be invertible with |det T| = 1. There is a bijection ψ : Λ → Λ and a finite bT such that |ψ(z) − Tz| ≤ bT (z ∈ Λ).” · Show in paper
“Table 2 shows that conventional PTQ methods yield diminishing returns on NVFP4. Notably, QuaRot (Hadamard transform) results in performance regression compared to RTN on Llama 3.1-8B, which confirms our motivation (Figure 2) that rotation disrupts the isolation benefits of fine-grained blocking. Similarly, SmoothQuant offers marginal gains due to the limited capacity of 4-bit weights. In contrast, ARCQuant consistently achieves the best results in all metrics, effectively suppressing outlier errors in the activation matrices.” · Show in paper
Table 2 confirms that NVFP4+QuaRot regresses on Llama 3.1-8B (70.18 avg, 6.99 PPL vs 70.45 avg, 6.95 PPL for RTN), while ARCQuant achieves the highest average accuracy and lowest PPL on both models.
“Repeated projection reduces dimensions j ≥ 4 to dimensions two and three. In the remaining dimensions, the sampling argument uses independent Poisson batches. A moment argument controls the size of the decoded set; its geometric input is a rich-line estimate with the field characteristic kept larger than the relevant polynomial degrees.” · Show in paper
“Training at scale confirms the choices from the small-scale sweep: both the continual pre-training losses (Figures 9 to 11) and the downstream ablations (Appendix F.6) again favor an encoder window W = 1024 and causal masking. At scale, the attention-based adapter is competitive with the MLP adapter; we keep the MLP adapter, which adds less computation. The choice of pooling operator depends on the compression ratio: mean pooling has a small edge over concatenation at 16x compression, while concatenation slightly outperforms mean pooling at 4x.” · Show in paper
Both the from-scratch pre-training sweeps and the full-scale continual pre-training and downstream ablations support these architectural conclusions.
“Procedure \mathcal{L}(A, D, a, b). Execute the first applicable base case. If neither applies, execute Steps 3–6 in order.” · Show in paper
“Given a protection budget ρ ∈ [0, 1] (sensitive-frac), the K=⌈ρL⌉ layers with largest εℓ are loaded in fp16; all others in NF4 with double quantization; lm_head stays fp16.” · Show in paper
“Our results show that LoRA with r=16 recovers within 11.6 percentage points of full fine-tuning accuracy (59.6% vs. 71.2% exact-match) while training fewer than 1% of parameters and consuming 31% less peak GPU memory.” · Show in paper
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter ModelThe numbers in Table 2 and the Pareto plots in Figure 1 directly support the accuracy, parameter count, and memory reduction claims for seed 42.
“On a closing edge, its input and required endpoint are known. A butterfly updating each coordinate at most once has at most one route between those endpoints; the decoder fills the unknown bits along that route with the forced values, rejecting inconsistencies.” · Show in paper
“Under that control the ranking has no measurable advantage (Appendix N lists every setting). Across five models and three budgets, top-K beats the median matched draw in 11 of 15 settings, and 50 of the 150 draws beat top-K where chance predicts 75. A sign test does not separate that from chance (p=.12), and the median gap is 0.01 perplexity.” · Show in paper
The parameter-matched control in Table 19 and downstream comparisons in Section 7.2 directly support this negative result.
“From the filtered lists, enumerate only pairs whose signed reduced weights add to T F as integers and whose signed unreduced weights do not add to t F s as integers. Both sums count mask positions with their multiplicities, including a position selected by both masks.” · Show in paper
“f(n) = ‖qy˜(n)‖2 2 , κ(n) = ( (V (yi(n)))D i=1, ⌊f(n)⌋ ) , 𝒦 = {κ(n) : n ∈ G}.” · Show in paper
“At ≈ 2.2 bit (group-size 128; using FP16 scale and 2-bit zero point per group) the perplexity increase is already less than 1.5 points, while dropping to 0.6 - 0.7 at ≈ 2.6 bit (group-size 32), which is only slightly worse than vanilla 3-bit and might be interesting for practical kernel implementations. Further, if we reduce group size to 8, we can apply ternary (-1, 0, +1) quantization, which achieves 9.20 WikiText2 PPL on OPT-175B, a less than 1 point drop.” · Show in paper
Table 7 directly supports the 2-bit grouped perplexity numbers; the ternary group-size 8 result (9.20 PPL) is reported in the text.