Search publications
“Figure 3 and Table 6 (Appendix A) show a monotone dose-response for all three edits. The submit rate rises from 0.15 to 0.41; the control gives 0.15. Thinking tokens fall from 659 to 399; the control gives 649. The rate at which the next call makes the first edit falls from 0.66 to 0.49. The four control adapters give 0.66, 0.62, 0.64 and 0.66.” · Show in paper
The replay results in Figure 3 and Table 6 support monotone dose-response for all three always-on edits while random controls remain near the unedited baseline.
“In summary, our results show that while LLMs are effective at retrieving isolated facts, their capabilities degrade substantially when required to reason over long contexts.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksFigures 1, 5, 6, and 8 clearly demonstrate the performance gap between retrieval tasks and reasoning tasks across all tested languages.
“A Long-Horizon-Terminal-Bench task follows the Terminal-Bench formulation [22]: it is specified as a Harbor task with (1) a natural-language instruction, (2) a Docker image that defines the terminal environment, (3) a task configuration file, and (4) an oracle implementation or simulator used for grading.” · Show in paper
“With the line the raccoon is fed in 78 of 80 draws, against 48 of 80 without it, and eating while not feeding him falls from 14 of 80 to 0” · Show in paper
The rates (78/80 vs 48/80; 0/80 vs 14/80) are clearly reported in §5.3, and the authors explicitly note that the comparison is between rates from an earlier run rather than draw-by-draw.
“Specifically, our approach involves two complementary measures: (1) exclusively selecting repositories distributed under strong copyleft licenses (GPL) to construct a public set (11 repositories) and a held-out set (12 repositories), and (2) acquiring commercial codebases from real startups to capture enterprise-grade problems in a commercial set (18 repositories).” · Show in paper
The claim accurately describes the repository selection and access-control strategy implemented by the authors.
“That is 30.3 against 25.3 over these six batches, higher in all six (Table 3; Figure 4 in Appendix E; sign test p = 0.031).” · Show in paper
The results in Tables 3 and 10 support the reported means, p-values, and the lack of significance after Holm correction.
“At 64k tokens, the 16x LCLM reaches the first token faster than the smallest uncompressed models (0.21 s, compared with 0.81 s for Llama 3.2 1B and 0.97 s for Qwen3-0.6B) while being far more accurate on LongHealth5 (66.25% vs. 22.50% and 32.75%). The 4x LCLM exceeds the uncompressed Qwen3-4B-Instruct-2507 on LongBench with a 6x lower TTFT.” · Show in paper
The vLLM-served benchmarks in Figure 5 and Section 6.2 directly substantiate the reported TTFT and accuracy comparisons.
“We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks.” · Show in paper
The reported 400x compression ratio, 3.7x A100 speedup, 32.6% peak memory reduction, and near-Full-KV accuracy are directly supported by the experimental results in Section 4.2, Section 4.3, Figure 5, Figure 7, and Table 2.
“Table 2 lists all completed pairs: in 9 of 12 only the burst run degraded persistently, in 0 only the no-burst run did, 2 degraded in both arms and 1 in neither (two-sided sign test over discordant pairs, p = 0.00391).” · Show in paper
The paired design, statistical test, and control experiments cleanly establish history-dependent persistent degradation on vLLM.
“The difference between fake “2bit (K per-channel, V per-token)” quantization and KIVI is that KIVI maintains a full precision key and value cache sliding window for the local relevant tokens. This sliding window is crucial to maintaining accuracy for hard generation tasks such as mathematical reasoning.” · Show in paper
Table 3 clearly shows that fake 2bit (K-C, V-T) degrades sharply on GSM8K, while KIVI-2 recovers most of the 16bit baseline performance.
“We compare results of various LLMs on this benchmark to results on SWE-bench Verified and show that performance of some language models might be inflated due to contamination issues.” · Show in paper
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering AgentsThe claim is appropriately hedged ('might be inflated') and supported by the empirical divergence in Table 2 between the two DeepSeek-V3 releases on SWE-bench Verified versus their near-identical performance on fresh SWE-rebench tasks.
- Claim Empirical results confirm that canonicalisation and hashing in Cogitator incur modest overhead.
“Empirical results confirm that witness roots are invariant across parallelism and CPU pinning, that deterministic chaos surfaces regressions without masking them and that canonicalisation and hashing incur modest overhead.” · Show in paper
Section 7 explicitly states that a comprehensive microbenchmark and systematic overhead study have not yet been performed and are left as future work, so no empirical results in the manuscript confirm modest overhead.
“1. Inference scaling is substantial but highly benchmark-dependent. Some benchmarks continue to improve well beyond typical published token budgets – including FrontierMath, TerminalBench, and Humanity’s Last Exam (HLE) – while others show weaker marginal gains under our protocol.” · Show in paper
Table 3 and Figure 1 directly substantiate both the substantial gains on FrontierMath and HLE and the weak marginal gains on SWE-Bench Pro, TerminalBench, and HealthBench under the tested protocol.
“Specifically, given a task instance (passage, query) and a corpus of background texts, we retrieve the top-k most semantically similar distractor passages based on cosine similarity. The value of k is adjusted dynamically to control the overall length of the prompt, allowing our setup to scale the long contexts to any arbitrary length as needed.” · Show in paper
“Training at scale confirms the choices from the small-scale sweep: both the continual pre-training losses (Figures 9 to 11) and the downstream ablations (Appendix F.6) again favor an encoder window W = 1024 and causal masking. At scale, the attention-based adapter is competitive with the MLP adapter; we keep the MLP adapter, which adds less computation. The choice of pooling operator depends on the compression ratio: mean pooling has a small edge over concatenation at 16x compression, while concatenation slightly outperforms mean pooling at 4x.” · Show in paper
Both the from-scratch pre-training sweeps and the full-scale continual pre-training and downstream ablations support these architectural conclusions.
“In addition, we linearly regress log(time horizon) against release date, finding that time horizon has doubled every 207 days with a 95% bootstrapped confidence interval 166-240 days (roughly ±19%).” · Show in paper
The data, hierarchical bootstrap confidence intervals, and comparison against linear and hyperbolic fits support the exponential doubling trend on this task suite.
“The unquantized Qwen3-1.7B control shows the same qualitative monotonic trade-off: paraphrase EM rises from 76% at r = 8 to 79% at r = 16 and 86% at r = 32, while average OOD performance falls from 57.0% to 52.0% and 40.2%, respectively. This suggests that quantization is not required for the qualitative rank trend, although this reduced control changes model scale.” · Show in paper
The reported numbers on Qwen3-1.7B demonstrate the same monotonic trade-off without quantization, and the authors appropriately qualify the difference in model scale.
“We evaluate representative MLLMs, including closed-source systems (Gemini-3 [5], ChatGPT-5[15]) and open-weight models (Qwen3 [1], Gemma-3 [4]), under a unified evaluation pipeline. All models are tested with standardized input formatting and identical decoding settings (temperature = 0.2, max tokens = 2048) to ensure fair comparison. For retrieval-augmented experiments, we employ ColPali as a multimodal retriever at the page level, retrieving top-k = 3 relevant pages per query. Retrieved content is concatenated with the prompt before generation. Then for result evaluation, we adopt Qwen-Plus (temperature = 0.2) as an LLM-as-a-judge to score final answer correctness and evidence alignment on a 0–5 scale. We further analyze reasoning behaviors using a structured error taxonomy. In addition, we report ROUGE [11] and BLEU [17] as complementary automatic metrics.” · Show in paper
“~115 tokens total (~85% compression vs. the full protocol), freeing almost the entire context window for payload generation.” · Show in paper
The live demonstrations and stage-by-stage audits in the manuscript support the claim, with token counts explicitly labeled as estimated BPE tokens.