Search publications
“The longer agentic profiles exceed the single-H100 HBM budget: coding agent exceeds the wall by C = 40 and reaches 155 GB at C = 80, while terminal reaches 126 GB (Appendix Figure 4b clips both at the 80 GB wall).” · Show in paper
The analytical CF model in Figure 4b and Table 14b directly supports the capacity saturation thresholds.
“We use the Qwen2.5-72B-Instruct model [Qwen et al., 2025] (prompt in Appendix B.2) to generate up to three candidate JSON recipes per task. If an error occurs during the subsequent scripted installation or test execution (derived from a recipe), the LLM attempts to refine that recipe by analyzing error logs and the original instructions (see correction prompt in Appendix B.4).” · Show in paper
“For Task Complexity (where 'high-score' implies >1 hour to solve; 100 high-score vs. 313 low-score examples in validation), our fine-tuned model achieved 81% accuracy and a weighted F1-score of 0.82. This is an improvement over the baseline Qwen-72B-Instruct, which achieved 68% accuracy. For Test Patch Correctness (180 high-score vs. 233 low-score examples), the model achieved 67% accuracy (weighted F1: 0.65). For Issue Clarity (84 high-score vs. 329 low-score examples), it achieved 79% accuracy (weighted F1: 0.76).” · Show in paper
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering AgentsAll reported accuracy and weighted F1 numbers match Table 4 exactly. Note that on Issue Clarity, the vanilla model achieved 80% accuracy (0.77 weighted F1) vs 79% (0.76) for fine-tuned, which is visible in Table 4.
“Higher mean performance is associated with smaller variation (Figure 3(a)): Spearman’s ρ = −0.76, or −0.67 excluding Qwen. Full delivery remains sensitive, however: GPT Sol has the highest mean and an SD of only 1.93, yet its full-credit rate varies by 9.12 percentage points. Both GLM variants have full-credit gaps of 13.50 points.” · Show in paper
Table 9 and Figure 3(a) directly support both the negative correlation between mean score and SD and the persistent sensitivity of R1.
“LCLMs achieve the fastest compression time at every context length and substantially lower peak memory at long contexts; all baselines run out of memory at 512K tokens, except Attention Matching, which fails at 512K due to numerical instability in its linear solver and runs out of memory at 1M.” · Show in paper
Figure 4 and Section 6.2 clearly document the scaling measurements and memory limits on the H200 GPU.
“Aggregate metrics (pass rates and scores) were identical, but Cogitator’s drift report flagged a divergence at tool call 3. Because the fault schedules are deterministic, the injected timeout occurred at the same step in both runs; thus the divergence is attributable to the agent change rather than random timing. A second pair of runs with fault injection disabled reproduced the same divergence, confirming that deterministic chaos neither hides nor amplifies agent regressions.” · Show in paper
The experimental comparisons with fault injection enabled and disabled support the claim that deterministic fault injection localises divergences without masking or amplifying regressions.
“A Terminal-Bench task consists of an instruction, a Docker image, a set of tests, an example solution, and a time limit (Figure 2).” · Show in paper
“We show that this makes prefix-cached agent serving metastable. With an identical base workload, a few minutes of extra arrivals leaves a vLLM server in a degraded regime—a ∼10% hit ratio, about half the throughput, and a backlog that grows for the remaining 70 minutes—while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction).” · Show in paper
The paired experiments, no-cache and session-cap controls, and reproductions across models and engines directly support the claim of metastability.
“Our main experiment follows a fully crossed design with six frontier models (Table 1) evaluated on five non-cyber benchmarks under two feedback conditions, using one shared ReAct-style scaffold (Yao et al., 2023) implemented in Inspect AI (UK AI Security Institute, 2024a). For each task in each (benchmark, model, condition) cell, we run 5 independent trajectories.” · Show in paper
“As shown in Figure 9, the best static split factor varies with different sequence lengths and token budgets, while adaptive compression decomposition provides comparable accuracy to the best static split factor in most cases.” · Show in paper
Figure 9 shows that adaptive RocketKV closely matches or exceeds the best static split factor across the tested benchmarks and token budgets.
“For each benchmark, we fit a linear probability model regressing on model generation, leave-one-model-out task difficulty, and their interaction, by ordinary least squares (OLS) with cluster-robust standard errors at the task level.” · Show in paper
“We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme.” · Show in paper
Figure 6 and Table 4 on SCBench directly demonstrate that RocketKV-MT outperforms all non-oracle baselines and closely matches Exact-TopK across all tested token budgets.
“with scaffold, language and run length held constant, testing after the last edit roughly doubles the odds of solving for all three models. The median failed run writes 1.6–2.0 times the completion tokens of the median solved run in the same model–scaffold pair (Figure 4c): a failing run is one that kept going, not one that gave up.” · Show in paper
Figure 4c and Appendix B support the completion-token ratios and classifier-based test-after-edit finding.
“Our results show that the effective context size is approximately 16k tokens for the best prompt-only and 32k tokens for the best RAG method. This suggests that the model is able to effectively use only up to 25% to 30% of its advertised context length.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksTable 1 directly supports the 16K and 32K ECW values under the authors' 30% degradation threshold.
“FrontierMath’s commonly used budget of 1M total tokens (Burnham, 2025) is itself an order-of-magnitude increase over the earlier 100k-token scaffold (Glazer et al., 2024), and extending from 1M to 10M adds a further +11.7 ± 11.0 percentage points on average. HLE also continues to improve well beyond its typical 64k-output-token budget, gaining +11.9 ± 10.4 points up to our 5M cap.” · Show in paper
The reported gains match Table 3 (rounded to one decimal place in the text) and are properly qualified as properties of the tested serial scaling protocol.
“Qwen2.5-Coder-32B-Instruct underperforms expectations, especially considering its strong code generation capabilities. Analysis of its trajectories reveals problems with instruction following; the model frequently hallucinated environment responses or enters loops of formatting errors, ultimately failing without producing a meaningful solution attempt.” · Show in paper
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering AgentsBoth the quantitative underperformance in Table 1 and the qualitative failure mode in Appendix J.2 are directly documented.
“Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch.” · Show in paper
The task counts, defect taxonomy, and repair pipeline are consistently documented in Figure 2, Table 1, and Sections 3.1–3.5.
“Under that control the ranking has no measurable advantage (Appendix N lists every setting). Across five models and three budgets, top-K beats the median matched draw in 11 of 15 settings, and 50 of the 150 draws beat top-K where chance predicts 75. A sign test does not separate that from chance (p=.12), and the median gap is 0.01 perplexity.” · Show in paper
The parameter-matched control in Table 19 and downstream comparisons in Section 7.2 directly support this negative result.
“Between the three reviews, the average task received approximately three hours of combined reviewer attention, implying multiple hundreds of person-hours went into reviewing alone, excluding the time spent creating the tasks.” · Show in paper
“Table 1 reports perplexity at n=1024 (T=512) on all five models. Every free-zone cell stays below the 0.2% threshold, with the largest drift at +0.17% (Qwen at 3×). The free zone holds at context lengths 512 and 2048 as well (Appendix B).” · Show in paper
All reported 2x and 3x perplexity values in Table 1 and Tables 4-6 are strictly below +0.2% relative drift.