Search publications
“One agent finds bugs and writes them up as claims. The referee gets the claims and clones the code at the same commit. It reads the code, but it does not get the scripts, outputs or reasoning of the agent that found it. It writes its own input and works out the right answer by itself. Then it gives each claim a verdict, like CONFIRMED or REFUTED.” · Show in paper
“We show that frontier models and agents score less than 65% on the benchmark and conduct an error analysis to identify areas for model and agent improvement.” · Show in paper
The experimental results in Figure 1, Section 4, and Table 2 directly support that all evaluated frontier models and agents score below 65%.
“Applying the audit workflow to Terminal-Bench 2.0 surfaced issues in 28 tasks, roughly a third of the benchmark. ... Fig. 3 shows the breakdown of the task defects in the 28 tasks that needed repair. The most common failure mode was instruction–verification mismatch (present in 57% of the tasks). ... The second most common defect was task drift (43% of tasks). ... The third category, environment issues (29% of tasks), comes from resource budgets.” · Show in paper
The counts in Table 2 (16/28 = 57.1%, 12/28 = 42.9%, 8/28 = 28.6%) match the reported percentages, and Table 3 documents the exact repairs per task.
“Agentic LLM workloads occupy a hardware regime that no prior inference benchmark captures. Across 22 workload profiles, 9 models, and 14 GPU configurations (3,197 sweep rows, 148,077 NCU records), AGENTPERFBENCH identifies a dual pressure that separates agentic serving from chat.” · Show in paper
The benchmark comparison, saturation sweeps, and roofline/capacity analyses directly support the claim that agentic workloads exhibit this dual-pressure regime uncaptured by prior benchmarks.
“Cogitator is a harness for deterministic, auditable evaluation of autonomous agents. It treats each evaluation as an experiment whose artefacts can be replayed off-line and independently verified. Cogitator commits to the witnessed semantics of a run—agent trace entries and tool-call witness views—using canonical JSON (JCS) and a domain-separated BLAKE3 hash chain, yielding a witness root. In replay mode, the witness root recomputes identically from the recorded transcripts; in live mode, it recomputes identically when prompts/seeds and tool behaviour are held constant.” · Show in paper
The manuscript specifies the cryptographic commitment construction and provides empirical demonstrations of deterministic witness roots, replayability, and verification.
“To evaluate whether the workflow transfers beyond Terminal-Bench, we applied it to 34 randomly selected tasks from an 80-task snapshot during the development of Harbor-Index v0.1. ... The workflow flagged 14 tasks, of which manual review confirmed 12 as defective (86% precision).” · Show in paper
12/14 = 85.7% (~86%) precision is directly supported by the manual review of all 34 tasks.
“SnapKV achieves the best throughput, on par with All-KV on Llama and even surpassing All-KV on Mistral.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingFigure 3 directly supports that SnapKV-0.75 achieves the highest average throughput among all evaluated methods on both models.
“The result is a simplified taxonomy with three high-level classes – Execution, Coherence, and Verification —- each with subclasses.” · Show in paper
“We find that command error rates range from 9.2% (Grok 4) to 26.7% (GPT-OSS-120B).” · Show in paper
Figure 9 and the text on pages 9-10 support the reported error rate range and failure category percentages.
“The largest gains are concentrated among tasks with instruction–verification mismatches. In TB 2.1, five tasks increased from pass rates of at most 15% to over 50% after repairs centered on the verifier, accounting for roughly 62.7% of the total absolute pass-rate change on the modified-task subset (full breakdown in Appx. C).” · Show in paper
Table 7 confirms the five tasks increasing from <=15% to >50%, all of which are listed under instruction-verification mismatch in Table 2.
“Harness choice substantially affects performance within individual models. General-task score gaps across harnesses range from 5.0 to 9.5 points within nine models, and reach 39.2 for Qwen (Table 2). ... Multimodal gaps are larger for several models: Kimi ranges from 28.1 with Codex to 72.7 with OpenClaw, with preferred harnesses differing across backends (Table 3).” · Show in paper
The reported ranges match the values in Tables 2 and 3.
“The base LLM stays frozen, and the router adds under 0.1% parameters.” · Show in paper
Table 24 confirms parameter overhead is 0.054%–0.086% (<0.1%), and Table 3 confirms the 2.8–3.8 point drop when using training-free routing.
“Under the lock, the sandbox can reach only the model endpoint during the agent phase (the allowlist names that hostname alone), web tools that scaffolds execute outside the sandbox are disabled, and the images carry no fixing commit in their git history.” · Show in paper
“There is an extraction of a consistent tuple F′ of deterministic length ℓ′ = Θ(k), still in one of the two stream orientations, and a new discrete context C′ for which H(C′) ≤ Λ′ ≤ CqP log(w + 2)(σ + wP) + Cwσ, |domain of each slot of F′| ≤ Cq^d , Δ′ ≤ CK∗, 1 ≤ w ≤ Cσ^{1+η/2}/D.” · Show in paper
The standalone nature of the new context C' and the uniformity of all constants across admissible histories are carefully verified in Section 6.5 and the proof of Lemma 6.1 on page 47.
“Dropping the residual entirely (rank-only compression) raises 2× perplexity from 6.54 to 7.27, a gap that additional rank cannot close.” · Show in paper
Table 11 directly supports both the necessity of the rotated residual and the saturation at 4 bits.
“We evaluate the effective context length for two models with non-Transformer architectures: RWKV-v5 (Peng et al., 2023) and Mamba-2.8B-slimpj (Gu & Dao, 2023). We find that both models demonstrate significant degradation when extending context size to 8K, and both underperform the Transformer baseline Llama2-7B by large margins up till the length of 4K, beyond which Llama2 shows poor length extrapolation performance (Figure 4 right).” · Show in paper
Figure 4 (right) directly supports the performance gap up to 4K and the degradation at 8K, though Mamba-2.8B is smaller in parameter count (2.8B vs 7B).
No empirical token-footprint measurements are provided, and a fixed 4-tuple schema does not strictly guarantee tokens unless the length of text within each tuple field is also bounded.
“Among the models tied at one solved task at R >= 0.95, mean reward spans from 0.27 for GPT-5.4 to 0.32 for GLM 5.2. Pass rate and mean reward are positively but only moderately correlated (Spearman \rho = 0.74), and they can produce different model rankings because they measure different aspects of long-horizon performance: pass rate captures full task completion, whereas mean reward captures sustained partial progress.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based GradingFigure 3 confirms the exact mean reward values for the four models tied at one solved task at R >= 0.95.
“Execution errors dominate for Opus 4.5 and GPT-5.2, while coherence and verification errors occur at lower rates. Conversely, the open sourced model evaluated (Qwen Coder) displays a more balanced error pattern, with higher errors across all failure modes.” · Show in paper
Figure 8 clearly illustrates higher execution error bars relative to coherence and verification for Opus 4.5 and GPT-5.2, and elevated, more uniform bars across all three categories for Qwen Coder 480B.
“Figure 2 traces the tasks through three repair rounds, an engineer review and a final review. Round 1 repaired 40 container images whose reference patches carried test and fixture files or whose instructions lacked the standard template. Round 2 ran an automated audit of difficulty calibration, instruction–test alignment and reward hacking; it flagged 387 tasks (53%), found 326 clean and errored on 18.” · Show in paper