Search publications
“With DeepSeek Flash, Codex scores 68.34 at $0.02022 per task, versus Nanobot’s 60.92 at $0.03272. Nanobot’s higher cost coexists with extensive cache reuse (85.65% of input tokens). With GPT Sol, Nanobot spends 2.97 times as much as OpenClaw while achieving lower full-credit attainment (24.82% versus 31.02%).” · Show in paper
Table 17 and Table 7 confirm the exact cost ratios, scores, and full-credit rates.
“For GLM-5.3 and Kimi K3, accuracy barely moves: only one of their six scaffold pairs differs significantly, Kimi K3’s Pi against OpenCode by 1.7 points. Inkling is the exception, with Pi trailing the other two scaffolds by 5.8–7.5 points.” · Show in paper
Table 3, Table 4, and Figure 4a directly substantiate the accuracy spreads, p-values, and token-cost differences.
“Splitting the variance of solved-or-failed over the 642 tasks every combination attempted, task identity accounts for 31.7%, the model for 2.65%, the scaffold for 0.29% and the model–scaffold interaction for 0.36%; the residual 65.0% is task-by-run variation.” · Show in paper
The numbers in Appendix B and Section 4.5 are consistent.
“Terminus 2 has a single tool, a headless terminal, and completes tasks using only Bash commands.” · Show in paper
“Choosing a separate harness per model on training tasks improves held-out full-credit attainment by 2.53–2.73 percentage points over a single shared harness. Mean-score gains remain within ±0.14 points, and R0.8 gains within 0.31 points. Matching harnesses to models thus matters more for complete delivery” · Show in paper
Table 10 and the accompanying text in Appendix C.1 confirm that full-credit gains are positive in all 20 repetitions and substantially larger than mean-score or R0.8 gains.
“Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2–3× compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3× and declines by only 0.90 and 2.40 pp at 4× and 5×, respectively.” · Show in paper
The experimental results in Table 1, Tables 4-7, and Table 9 support the 2-3x near-lossless claim across the tested models and tasks, noting that LLaMA-3.1-8B and Qwen2.5-14B used FlashJoLT.
“Across diverse long-context language and vision-language reasoning, and multi-turn dialogue benchmarks, our method substantially reduces KV memory while matching or surpassing full-cache inference. These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.” · Show in paper
The experimental results across Tables 1, 2, 3, 7 and Figure 5 directly support that DBTrimKV reduces KV memory while matching or surpassing full-cache inference across all tested benchmark categories.
“Our results show that teams outperform solo agents, but at a steep price: a team composed of all strong agents raises reward over a single agent working alone, yet costs 3 to 7 times as much in inference dollars and, in most configurations, takes longer to finish (Section 4).” · Show in paper
Figure 2, Section 4, Table 3, and Table 4 substantiate the reward gains, 3–7× cost increase, and 1.4×–3.8× wall-clock time increase on ProgramBench.
“We evaluate 17 frontier models and find that agents consume on average 9.8M tokens per task, with roughly 239 episodes and 88.9 minutes of execution time per run, making Long-Horizon-Terminal-Bench substantially more demanding than prior terminal-based benchmarks. Even the strongest tested model, Grok 4.5, achieves only 28.3% pass@1 at a partial-reward threshold of 0.95 and 19.6% at a perfect-reward threshold of 1.0, while the mean pass rate across models is just 6.4% and 3.2% under the two thresholds, respectively.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based GradingThe reported pass rates, token counts, episode counts, and execution times are directly supported by the quantitative results in Table 1, Figure 3, and Section 3.1.
“We evaluate models using the Harbor framework with the Terminus-2 agent harness [10], which interacts with each task environment through a single long-horizon terminal session. Additionally, for GPT-5.3, we instead use Codex [28] as the agent harness.” · Show in paper
“CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingBoth Table 2 and Table 4 clearly demonstrate extreme workload sensitivity in CaM's accuracy and realized compression ratio.
“Our additional ablations demonstrate that training on longer sequences does not always lead to better performance on RULER” · Show in paper
Both the LWM ablation in Figure 4 and the main results in Table 3 support that longer training context does not guarantee superior RULER performance.
“Cheap workers pay off only when directed toward planning (Figure 6). Relative to PIV, planning raises reward by 5.6% while lowering cost by 22.2% under blocking, and by 4.0% while lowering cost by 30.9% under asynchronous push. Directed toward implementation, by contrast, Luna workers reduce the team’s reward below that of the Sol solo agent while increasing its cost” · Show in paper
Figures 6 and 7, along with Table 3 and Table 4, support the stage-dependent payoff of cheap Luna workers.
“Across 19 agents, rankings remain highly correlated between TB2.0 and TB2.1 (Spearman ρ = 0.958), while individual scores shift by up to 12 percentage points and leaderboard positions move by up to three ranks.” · Show in paper
The correlation, confidence interval, and rank/score shifts are directly supported by Figure 4 and the 19-agent evaluation in Section 6.1.
“GLM-5.3 (thinking on), Kimi K3 (reasoning effort max) and Inkling (xhigh) ran with a 1M-token context window and no cap on thinking tokens or turns; the only cap besides the 50-minute budget is mini-swe-agent’s default 30 s per-command timeout, which is part of that scaffold.” · Show in paper
“Figure 4 (middle-right) shows that the 34B model is significantly better than the 6B model on RULER for both performance at length of 4K and the relative degradation, suggesting the benefit of scaling model sizes for better long-context modeling.” · Show in paper
Figure 4 (middle-right) clearly shows Yi-34B-200K outperforming Yi-9B-200K and Yi-6B-200K across all tested lengths.
“First-attempt success varies widely: Claude ranges from 96% in JavaScript and 86% in Python down to 36% in C++ and only 18% in Java, Codex spans 44% (C++) to 88% (Python and JavaScript), and Gemini runs from 32% (Python) to 84% (JavaScript). After repair, nearly every configuration converges: Claude reaches 100% in all four ecosystems, Codex falls short only in C++ (94%), and Gemini falls slightly short in Java (96%) and C++ (98%).” · Show in paper
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated SoftwareFigures 4-6 and Section 6.1 directly report the exact first-attempt and post-repair convergence rates across all 600 primary instances.
“For each task, the evidence agent produces a structured report containing information on the benchmark, the task specification, trace information, and community reports. ... An auditor applies the rubric to each task. ... A verifier reviews the evidence and the auditor’s label.” · Show in paper
“Greater communication, however, does not yield higher reward. Sol and Terra teams are unchanged within noise, and Luna teams lose 4.4–7.1 points (Figure 9).” · Show in paper
Figure 9, Appendix C.10, and Table 6 directly support the null/negative effect of inter-agent messaging on reward.
“We evaluate 19 agents from multiple providers and compare the accuracy in both TB 2.0 and TB 2.1, as illustrated in Fig. 4. The correlation is high (Spearman ρ = 0.958), with an OLS fit of TB2.1 = 2.1 + 1.04 · TB2.0.” · Show in paper