Search publications
“Overall, these findings suggest that frontier capability cannot be fully characterized by a single benchmark score measured under a single inference-time protocol. Observed performance depends not only on the model, but also on how much inference-time compute it is given and how that compute is allocated.” · Show in paper
The claim is well-supported by the multi-benchmark experiments showing that token budgets, repeated submissions, feedback, and parallel vs. serial allocation all substantially affect measured performance.
“Our evaluations on various models and workloads show that vLLM improves the LLM serving throughput by 2-4× compared to the state-of-the-art systems [31, 60], without affecting the model accuracy at all. The improvements are more pronounced with longer sequences, larger models, and more complex decoding algorithms (§4.3).” · Show in paper
The experimental results across Sections 6.2-6.5 consistently demonstrate 1.67x-3.58x (and higher against Orca Max/FasterTransformer) throughput improvements at comparable normalized latencies, and PagedAttention is mathematically equivalent to standard attention.
“Indeed, our profiling results in Fig. 2 show that only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems.” · Show in paper
The profiling data in Figure 2 directly supports the 96.3% vs 20.4%-38.2% memory utilization comparison.
“Specifically, KV entries are stored in fixed-size pages, and each head maintains a block table for its currently active pages. Attention is then computed with variable-length kernels [Dao, 2023] using the resulting per-head sequence lengths.” · Show in paper
“Fig. 17 shows that vLLM can sustain 2× higher request rates compared to the three Orca baselines.” · Show in paper
Figure 17 directly supports the 2x higher sustainable request rate.
“We evaluate all models using vLLM (Kwon et al., 2023), an LLM serving system with efficient KV cache memory management. For all models, we run the inference in BFloat16 on 8 NVIDIA A100 GPUs with greedy decoding.” · Show in paper
“For each layer, we compute the top eight token-mode singular vectors of its key cache and score consecutive layers by the mean squared cosine between their bases. A new group starts whenever the score falls below the median score across the network.” · Show in paper
“Our extensive experiments with an open-weight large language model (LLM) reveal a pronounced gap between high- and low-resource languages, particularly for tasks requiring the model to aggregate multiple facts or predict the absence of information. We also find that, in multilingual settings, LLMs effectively utilize less than 30% of their claimed context length. Although off-the-shelf Retrieval Augmented Generation helps alleviate this to a certain extent, it does not solve the long-context problem.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksThe experimental results in Tables 1, 5, 6 and Figures 5-8 consistently support the claimed gaps across languages, task types, and effective context length, as well as the partial mitigation by RAG.
“SpecStream uses NVIDIA Multi-Process Service (MPS) [19] to enable concurrent execution of the Target and Draft models in separate processes. ... The system applies a TPC mask in the separate Draft process so that its compute kernels run only on the designated set of TPCs [16], while preserving the Target model’s original access to compute resources.” · Show in paper
“At each decoding step, each head scores every span with an independent gate rather than a softmax, since expansion is a per-span decision: ” · Show in paper
“We observe that, at 4 bits, GPTQ models reach only ≤ 0.25 lower perplexity than the full-precision versions, with a large gap to RTN results on OPT-175B. At 3-bit, RTN collapses, while GPTQ is still able to maintain good performance on most tasks, losing only 0.3 − 0.6 points for more than 5× compression.” · Show in paper
Table 5 confirms 4-bit perplexity differences (e.g., OPT-175B PTB: 12.26 vs. 12.01; BLOOM-176B PTB: 14.75 vs. 14.59) and 3-bit differences of 0.34-0.67 on WikiText2/C4/PTB (note BLOOM-176B PTB is 15.57 vs. 14.59, slightly above 0.6, which aligns with 'on most tasks').
“On Llama 3.1-70B, ARCQuant consistently improves over the NVFP4+RTN baseline across all reported metrics, with gains of 0.65 in average zero-shot accuracy, 0.23 lower WikiText2 perplexity, and 0.50 higher MMLU, while maintaining a competitive margin to the FP16 baseline. We also validate ARCQuant on the MoE model Mixtral 8x7B-Instruct, where it again achieves consistent improvements over NVFP4+RTN, remaining close to FP16 performance.” · Show in paper
Table 9 confirms the exact reported gains on Llama 3.1-70B and Mixtral 8x7B-Instruct.
“We segment the input into fixed-size chunks of 1,024 encoder tokens, compress each chunk, and assign it a segment identifier. The model receives the entire compressed sequence in a single prompt, together with one expand tool. At each turn, the model can issue a native tool call to expand with a segment identifier (Figure 6), and the tool returns the original text of that segment.” · Show in paper
“We address this problem by developing a quantized-matrix full-precision-vector product kernel which performs a matrix vector product by dynamically dequantizing weights when needed. Most notably, this does not require any activation quantization.” · Show in paper
“we compute during prefill and mark positions with as boundaries, where and are the sequence mean and standard deviation, and boundaries closer than tokens are merged, keeping the highest-surprisal one.” · Show in paper
“FlashInfer’s fused kernel can yield 28 − 30% latency reduction under different settings (by changing the recent window size of Streaming-LLM). Original implementation is included as a baseline reference. We also show the kernel-level performance comparison between FlashInfer’s fused RoPE kernel and the combination of FlashAttention’s RoPE kernel and FlashAttention’s attention kernel. FlashInfer’s fused RoPE kernel achieves 1.6-3.7x higher bandwidth utilization compared to not fusing attention with RoPE” · Show in paper
Figure 9 data supports both the 28–30% end-to-end ITL reduction and the 1.6–3.7x kernel bandwidth utilization improvement.
“Our results show that the effective context size is approximately 16k tokens for the best prompt-only and 32k tokens for the best RAG method. This suggests that the model is able to effectively use only up to 25% to 30% of its advertised context length.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksTable 1 directly supports the 16K and 32K ECW values under the authors' 30% degradation threshold.
“We apply three deliberately simple inference-scaling techniques uniformly across every (benchmark, model, condition) cell: 1. Expanded total token budgets of 5M–30M tokens per trajectory (Table 2), sized to task complexity and one to three orders of magnitude above typical benchmark defaults. 2. Context compaction, which replaces earlier turns with a model-generated summary to enable serial scaling beyond the nominal context-window size. We adopt Inspect AI’s default summary-based strategy (UK AI Security Institute, 2024a), triggered when the running context exceeds 130k tokens (about 65% of the smallest 200k context window across models). 3. Iterative resubmission with a hard cap of 999 submissions per trajectory, allowing the model either to refine its previous answer or to try a substantially different approach between submissions. To reduce unproductive loops of near-identical submissions, we also employ a lightweight repetition guard that terminates a trajectory when a separate LLM judge finds three or more consecutive submissions semantically equivalent (Section A.1.3).” · Show in paper
“On GSM8K [41] with 8-shot chain-of-thought prompting [42] and exact-match scoring (n=1319 per cell), JoLT is near-lossless at 2× and 3× on both Mistral and LLaMA-2 (Table 7 in Appendix C). All free-zone deltas fall within their 95% confidence intervals of zero, with the largest shift at −1.21 pp (Mistral 3×).” · Show in paper
Tables 7 and 10 support the claims with the stated caveats (single model and single seed for LongBench).
“The FlashJoLT decode kernel evaluates attention directly over the stored factors and packed residual, so no dense KV buffer is ever written to global memory.” · Show in paper