Search publications
“KAIROS is implemented on top of vLLM v0.18.1 [8] which supports disaggregated prefill/decode through a KV transfer API. KAIROS adds approximately 2000 lines of new code across the profiling for step latency models, the Deflection Dispatcher (which contains the analytical models and the deflection algorithm), the modifications to the scheduler on the decode nodes and plugins to report node state to the Deflection Dispatcher.” · Show in paper
“Chat stays below it, peaking at OIeff = 181 (Table 12), remaining memory-bandwidth-bound across the full concurrency sweep. The other agentic workloads cross the ridge: computer-use reaches OIeff = 886 (3.0x the ridge), terminal reaches 1,809 (6.1x), and coding agent reaches 2,409 (8.2x) at C = 80, all compute-bound. Peak separation between coding agent and chat is 13x.” · Show in paper
Table 12 and Figure 4a directly substantiate the exact OI values and ridge multiples on H100 for LLaMA-3.1-8B.
“On average, prompt-based methods lose more than 50% of their baseline accuracy at longer context windows (≥ 32k), and nearly all prompting methods begin to deteriorate sharply after 8k to 16k tokens.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksTable 1 shows substantial accuracy drops for all four prompt-only baselines between 0k and 32k-128k.
“As shown in Table 3, SmoothQuant can match the FP16 accuracy on all evaluation datasets with all quantization schemes. LLM.int8() can match the floating point accuracy because they use floating-point values to represent outliers, which leads to a large latency overhead (Table 11). The W8A8, ZeroQuant, and Outlier Suppression baselines produce nearly random results, indicating that naively quantizing the activation of LLMs will destroy the performance.” · Show in paper
Table 3 provides direct empirical evidence that SmoothQuant-O1/O2/O3 closely match FP16 accuracy and perplexity on OPT-175B while the three baselines degrade to near-random performance.
“Figure 7 shows the ITL and TTFT measured on both Llama 3.1 (Dubey et al., 2024) 8B (on 1xH100) and Llama 3.1 70B (on 4xH100) models. Compared to SGLang with Triton backend, FlashInfer backend shows consistent speedup in all settings.” · Show in paper
Figure 7 clearly shows lower median ITL and TTFT for SGLang (FlashInfer 0.2) than SGLang (Triton) across all four model-workload combinations.
“On the ShareGPT dataset, vLLM can sustain 1.7×–2.7× higher request rates compared to Orca (Oracle) and 2.7×–8× compared to Orca (Max), while maintaining similar latencies. ... Compared to FasterTransformer, vLLM can sustain up to 22× higher request rates, as FasterTransformer does not utilize a fine-grained scheduling mechanism and inefficiently manages the memory like Orca (Max).” · Show in paper
The latency-throughput curves in Figure 12 and batch size measurements in Figure 13 directly substantiate the claimed speedups.
“Increasing rank from r=2 to r=16 produced consistent accuracy gains, from 38.6% to 59.6% exact-match (Table 2). Beyond r=16, accuracy plateaued sharply: r=32 achieved only 60.4%, a gain of just 0.8 points despite doubling trainable parameters from 589,824 to 1,179,648. Peak memory remained nearly flat across all rank configurations (1.59–1.61 GB), confirming that for a 60M-parameter model the adapter weights are negligible relative to the frozen base.” · Show in paper
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter ModelTable 2 and Table 3 confirm the accuracy plateau beyond r=16 and the negligible memory difference (1.59–1.61 GB) across rank configurations.
“Online latent compaction is nearly lossless, reaching 90.52% accuracy compared with 92.46% for full-context generation, while being highly efficient: it reduces the average KV length by 3.1x, the average per-problem peak by 3.6x, and the maximum peak by 7.9x: the longest full-context trajectories reach the 32,768-token limit, whereas no compacted trajectory exceeds 4,135 tokens.” · Show in paper
The full 5,000-problem MATH test set evaluation under matched prompts, context limits, and scoring directly supports the accuracy and KV-reduction claims.
“DBTrimKV adds a small overhead compared to TrimKV due to variable-length, head-specific cache handling; however, it still outperforms vanilla inference at scale.” · Show in paper
Figures 7, 10, 11, and 12 demonstrate that DBTrimKV is substantially faster than Vanilla at scale while incurring a modest overhead relative to TrimKV.
“We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.3 ± 2.3% faster than well-tuned QLoRA and gives up 0.1 to 1.5 accuracy points. It was faster in every one of thirteen independent timing sessions, at worst by 7%. The quality setting trains 4.8 ± 2.0% faster. Its accuracy is level with QLoRA on every model and within a point of fp16 LoRA. The memory premium depends on how the architecture packs its layers: 0.06 to 0.8 GiB on the 1–4B models, 0.5 GiB at 7B, 1.9 GiB at 14B.” · Show in paper
The accuracy, memory, and speed figures match the reported tables, and the authors appropriately scope the timing claims to the five timed models (1.4B to 7B).
“We present QATFactory, an open-source framework for deployment-aligned quantization-aware distillation (QAD) and reinforcement learning (QARL). QATFactory simulates deployment-time quantization while performing matrix multiplications in BF16, allowing models to adapt to quantization noise without requiring training hardware that natively supports the target format; for example, it supports NVFP4 training on H100 GPUs, which lack FP4 Tensor Cores. The framework supports NVFP4, MXFP4, and llama.cpp’s Q4_K format; dense and mixture-of-experts models; and both full-parameter and LoRA-based training. It exports checkpoints directly to vLLM and llama.cpp without an additional lossy conversion step or added inference overhead.” · Show in paper
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMsThe manuscript thoroughly documents the framework design, supported formats, training workflows, and direct evaluation of exported checkpoints in vLLM and llama.cpp.
“The difference between fake “2bit (K per-channel, V per-token)” quantization and KIVI is that KIVI maintains a full precision key and value cache sliding window for the local relevant tokens. This sliding window is crucial to maintaining accuracy for hard generation tasks such as mathematical reasoning.” · Show in paper
Table 3 clearly shows that fake 2bit (K-C, V-T) degrades sharply on GSM8K, while KIVI-2 recovers most of the 16bit baseline performance.
“Specifically, PagedAttention partitions the KV cache of each sequence into KV blocks. Each block contains the key and value vectors for a fixed number of tokens, which we denote as KV block size (B).” · Show in paper
“As shown in Figure 9, the best static split factor varies with different sequence lengths and token budgets, while adaptive compression decomposition provides comparable accuracy to the best static split factor in most cases.” · Show in paper
Figure 9 shows that adaptive RocketKV closely matches or exceeds the best static split factor across the tested benchmarks and token budgets.
“Based on the figure, we can see that RocketKV consistently outperforms all other methods, especially under lower token budgets. For Llama3.1-8B-Ins, RocketKV achieves almost no accuracy loss with a token budget of 512 and above, and only 1.1% average accuracy drop with a token budget of 256.” · Show in paper
Table 2, Table 10, Table 16, and Figure 5(a-c) confirm RocketKV's consistent advantage at low token budgets and the 1.1% (52.2 to 51.1) drop at budget 256 on Llama3.1-8B-Ins.
“The KV block manager also maintains block tables—the mapping between logical and physical KV blocks of each request. Each block table entry records the corresponding physical blocks of a logical block and the number of filled positions. Separating logical and physical KV blocks allows vLLM to dynamically grow the KV cache memory without reserving it for all positions in advance” · Show in paper
“We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage.” · Show in paper
The accuracy, TTFT, and peak memory benchmarks across RULER, LongBench, LongHealth5, and GSM8K support the claim that LCLMs improve the Pareto frontier over KV cache compression baselines.
“Adding a shared 1024-token synthetic prefix brings deep-turn E2EL MAPE to 3.2% and aggregate E2EL MAPE to 2.4%, within the observed noise scale. Hence synthetic distributional profiles in this paper use this prefix-aware configuration.” · Show in paper
Table 11 directly reports the ablation results confirming 2.4% aggregate and 3.2% deep-turn E2EL MAPE.
“AQLoRA-q is the quality setting: adapters on every layer (α=1.0). AQLoRA-s is the speed setting: depth placement with α=.85 and early backward stopping.” · Show in paper
“Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2–3× compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3× and declines by only 0.90 and 2.40 pp at 4× and 5×, respectively.” · Show in paper
The experimental results in Table 1, Tables 4-7, and Table 9 support the 2-3x near-lossless claim across the tested models and tasks, noting that LLaMA-3.1-8B and Qwen2.5-14B used FlashJoLT.