Search publications
“Our local to global attention ratio patterns follow Gemma Team (2025a), that is, 4-to-1 local attention blocks for E2B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e. , values = keys. We encode position with p-RoPE with p = 0.25 on global attention layers and with RoPE on local attention layers, effectively reducing the global KV cache by 37.5%. The RoPE frequencies are set to 1M and 10k on global and local attention layers, respectively. Finally, we share the KV cache with ratios of 20/35 and 18/42 for the E2B and E4B model.” · Show in paper
“When using the standard practice of applying LoRA to query and value attention projection matrices [28], we are not able to replicate full finetuning performance for large base models. As shown in Figure 2 for LLaMA 7B finetuning on Alpaca, we find that the most critical LoRA hyperparameter is how many LoRA adapters are used in total and that LoRA on all linear transformer block layers are required to match full finetuning performance.” · Show in paper
Figure 2 clearly supports that QLoRA-All matches the 16-bit baseline while QLoRA-Attention falls short.
“Since a single physical block can be mapped to multiple logical blocks, we introduce a reference count for each physical block. ... vLLM implements a copy-on-write mechanism at the block granularity for the physical blocks that need modification by multiple sequences, similar to the copy-on-write technique in OS virtual memory” · Show in paper
“The batched implementation adds 1.6–6.7% to the prefill time of the prefill anchor across the four pairs, whereas the per-layer implementation adds 5–51% (Figure 4b). ... The request takes 2.96 s, close to the 2.80 s of Llama-3.2-Instruct-1B alone and 3.0× faster than the 8.83 s of Llama-3.1-Instruct-8B alone (Figure 1).” · Show in paper
Measured median latencies on one A40 GPU in Figure 1, Figure 4, and Section 4.3 directly support the reported percentages and speedups.
“After the first 768 generated tokens, we encode the oldest 512 generated tokens into latent memory tokens at a 16x compression ratio, resulting in 32 latent tokens, and keep the most recent 256 tokens uncompressed. Generation then continues from the compacted context and latest raw tokens, and the procedure repeats after every further 512 generated tokens.” · Show in paper
“Our entire GPTQ calibration data consists of 128 random 2048 token segments from the C4 dataset (Raffel et al., 2020) ... Specifically, we always load one Transformer block, consisting of 6 layers, at a time into GPU memory and then accumulate the layer-Hessians and perform quantization. Finally, the current block inputs are sent through the fully quantized block again to produce the new inputs for the quantization of the next block.” · Show in paper
“On 40 CPU threads with the checkpoint in the page cache, it takes 5 s on the 1B models, 12 to 13 s at 3 to 4B, 25 s at 7B and 37 s at 14B; reading the checkpoint from disk adds up to 21 s (released as profile_timing.json).” · Show in paper
Directly reported measurement from the authors' profiling pass.
“Experiments on NVIDIA H100 GPUs with Qwen-7B demonstrate up to 1.56× end-to-end throughput improvement over BF16, revealing a trade-off between throughput and model quality.” · Show in paper
Table 2 directly supports the 1.56× speedup at G = 256 under TP4 and demonstrates the throughput-quality trade-off across group sizes.
“To address this limitation, FlashInfer offers a head-group fusion strategy: different KV heads are mapped to individual threadblocks, while query heads are fused with the query length dimension.” · Show in paper
“The rollout server loads the policy in the target low-bit format and uses an inference engine such as vLLM to generate responses with native low-precision kernels. Verifiers score these responses and send the resulting trajectories and rewards to the training server. The training server maintains high-precision latent weights and optimizer states in PyTorch.” · Show in paper
“Figure 1 shows the loop. Algorithm 1 (Appendix A) lists its six steps: diagnose, locate, write, predict, verify by replay and by agent evaluation, and score. The author is an LLM with shell access to the model weights, stored activations and logged episodes. Steps 1–4 run on a CPU; only verification uses a GPU.” · Show in paper
“To verify that half-precision accumulation does not degrade quality in practice, we compare the LongBench scores obtained with FP16 and FP32 moment statistics on LLaMA-3.1-8B at L=128. The average score difference is less than 0.05 across all task categories, confirming that the numerical error introduced by half-precision accumulation is negligible relative to the approximation error O(\sigma^2).” · Show in paper
The reported experiment in Appendix C.2 supports the claim, though no per-category table is shown.
“The routing summary projects token keys into a -dimensional space with a learned per-head matrix and averages them with surprisal weights : .” · Show in paper
“Overall, PagedWeight matches the throughput of the uniform APL-style baseline with similar or lower memory consumption. The largest throughput drop is 3.3% at batch size 1 and 4.1% at batch size 4.” · Show in paper
Table 3 confirms the throughput drops: at B=1, max drop is (134.5 - 130.1)/134.5 = 3.27% (3.3%); at B=4, max drop is (401.9 - 385.4)/401.9 = 4.11% (4.1%).
“We train a LoRA of rank 16 and α = 32 on every linear layer of the language model (43.6M parameters) with ms-swift [22], keeping the Whisper encoder, its projector and the speech decoder frozen.” · Show in paper
“SEKV gains more from extra budget than ShadowKV, so its gap to ShadowKV narrows from 3.4 points at 5% to 1.0 at 20% on RULER and from 2.8 to 0.3 on NIAH, and on LLAMA-3.1-8B SEKV matches ShadowKV on NIAH at 20%.” · Show in paper
Table 4 directly supports all stated numbers.
“1. Inference scaling is substantial but highly benchmark-dependent. Some benchmarks continue to improve well beyond typical published token budgets – including FrontierMath, TerminalBench, and Humanity’s Last Exam (HLE) – while others show weaker marginal gains under our protocol.” · Show in paper
Table 3 and Figure 1 directly substantiate both the substantial gains on FrontierMath and HLE and the weak marginal gains on SWE-Bench Pro, TerminalBench, and HealthBench under the tested protocol.
“The longer agentic profiles exceed the single-H100 HBM budget: coding agent exceeds the wall by C = 40 and reaches 155 GB at C = 80, while terminal reaches 126 GB (Appendix Figure 4b clips both at the 80 GB wall).” · Show in paper
The analytical CF model in Figure 4b and Table 14b directly supports the capacity saturation thresholds.
“As shown in Fig. 16 (a), vLLM achieves 1.67× higher throughput than Orca (Oracle) when the one-shot prefix is shared. Furthermore, when more examples are shared (Fig. 16 (b)), vLLM achieves 3.58× higher throughput than Orca (Oracle).” · Show in paper
Figure 16 directly supports the 1.67x and 3.58x throughput improvements over Orca (Oracle).
“Combined with KV cache sharing (Shazeer, 2019) and the reuse of keys as values in global layers (Kayyam et al., 2026), these optimizations reduce the global KV cache footprint by up to 37.5%.” · Show in paper
The reduction follows directly from the specified architectural parameterization ( -RoPE and key-value reuse) and is consistent with the KV cache footprints reported in Table 3.