Search publications
“Without the requirements and interface, both models tested (GPT-5 and CLAUDE OPUS 4.1) show significantly degraded performance. In this setting, agents are less constrained and can submit more diverse solutions (particularly for feature additions). Since unit tests expect a narrow set of solutions, verifiers are prone to false negatives, resulting in lower pass rates.” · Show in paper
Table 3 quantitatively demonstrates the substantial performance drop when requirements and interface specifications are removed.
“Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K.” · Show in paper
The results in Table 3 and Tables 10-11 directly support the claim; 10 of 17 models (roughly half) pass the 85.6% threshold at 32K, while nearly all achieve near-perfect scores on vanilla NIAH.
“We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage.” · Show in paper
The accuracy, TTFT, and peak memory benchmarks across RULER, LongBench, LongHealth5, and GSM8K support the claim that LCLMs improve the Pareto frontier over KV cache compression baselines.
“Even under optimal configurations, the accuracy of RAG-based methods at longer context windows remains approximately 20% − 40% below the best short-context baseline. This indicates that while RAG improves stability and extends effective context size, there remains considerable room for further advancement in handling extreme context lengths.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksTable 1 and Figure 9 support both the improved stability of RAG and its remaining 20%-40% gap relative to the 0k CoT baseline (0.598).
“To initiate the dataset building process, we download approximately 450,000 pull requests linked to issues created before May 1, 2025. These originate from over 30,000 repositories that feature permissive licenses granting broad usage rights (see Appendix D for the list of included license types) and where Python constitutes over 75% of the codebase lines of code.” · Show in paper
“Online latent compaction is nearly lossless, reaching 90.52% accuracy compared with 92.46% for full-context generation, while being highly efficient: it reduces the average KV length by 3.1x, the average per-problem peak by 3.6x, and the maximum peak by 7.9x: the longest full-context trajectories reach the 32,768-token limit, whereas no compacted trajectory exceeds 4,135 tokens.” · Show in paper
The full 5,000-problem MATH test set evaluation under matched prompts, context limits, and scoring directly supports the accuracy and KV-reduction claims.
“Across all models, 79% of unresolved runs (518/660) end because the 90-minute budget expires while the agent is still actively working. However, these timed-out runs are not close to completion, with mean reward ranging only from 0.10 to 0.35 across models.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based GradingFigure 6 and Section 3.4 directly support that 79% of unresolved runs time out with mean rewards between 0.10 and 0.35.
“We show that fixed wall-clock and output-token budgets can materially change apparent agent performance: For example, extending the time limit closes a 20 pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp.” · Show in paper
Figure 6 directly shows the 20.3 pp gap on GPT 5.4 closing completely (74.9% vs 75.3%) and the 29.9 pp gap on GPT 5.4 mini shrinking to 5.3 pp (68.2% vs 62.9%) at 32x timeout.
“For each agent–model pair, we increase the wall-clock budget up to 32× the original task timeout and measure how many tasks are solved as the budget increases.” · Show in paper
“The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers.” · Show in paper
The SGLang paired runs and 32B trace-shape runs support the generalization, and the authors appropriately note that 32B provides weaker evidence of history dependence than 8B.
“Dense grading also reveals near-misses (0.75 <= R < 0.95) runs that are close to completion but fail some final checks. These near-misses occur nearly twice as often as passes (90 vs. 50), indicating that many runs get very close to fully solving the task, but fail at the last few steps or checks.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based GradingThe counts in Figure 4 (31 + 59 = 90 vs. 50) match the stated claim.
“The five leading harnesses have similar pooled mean scores, but differ in complete delivery and expenditure (Table 4). OpenCode leads mean score; OpenClaw leads full-credit attainment at the lowest API cost.” · Show in paper
Table 4 directly supports the stated ordering of OpenCode and OpenClaw.
“Chat stays below it, peaking at OIeff = 181 (Table 12), remaining memory-bandwidth-bound across the full concurrency sweep. The other agentic workloads cross the ridge: computer-use reaches OIeff = 886 (3.0x the ridge), terminal reaches 1,809 (6.1x), and coding agent reaches 2,409 (8.2x) at C = 80, all compute-bound. Peak separation between coding agent and chat is 13x.” · Show in paper
Table 12 and Figure 4a directly substantiate the exact OI values and ridge multiples on H100 for LLaMA-3.1-8B.
“3. Gains from inference scaling do not arise from one universal intervention. Benchmark scores depend partly on protocol choices about if and how models can iterate on their solutions, and whether compute is allocated to a single deep trajectory (“serial” scaling) or spread across multiple shallower ones (“parallel” scaling). Repeated submissions materially improve performance on all benchmarks, and feedback on submission correctness matters most where it can guide continued search (HLE and SWE-Bench Pro). Parallel scaling is strongest on stateless benchmarks (HealthBench and HLE, which do not involve a persistent interactive environment) and weakest on stateful ones.” · Show in paper
Tables 5, 7, and 12 directly support all three components of this claim.
“OPUS 4.1 primarily fails on semantic understanding, with wrong solutions accounting for 35.9% of failures and syntax errors at 24.2% ... SONNET 4 has context overflow as its primary failure mode (35.6%) and substantial endless file reading behaviors (17.0%) ... QWEN3 32B, as an open-source model, exhibits the highest tool error rate (42.0%)” · Show in paper
Table 4 provides the exact counts and percentages for each failure bucket across the evaluated models, supporting the characterization of distinct failure modes.
“Despite this wide variation in interaction patterns, we find essentially no correlation between the number of average turns per trial and model success rates (Figure 35). Similarly, while models vary significantly in their token generation patterns, our analysis shows that higher token count does not necessarily correlate with better performance (Figure 36).” · Show in paper
Both Appendix G.1 and G.2 show statistically non-significant correlations close to zero, supporting the claim despite minor numerical discrepancies between the text and figure legends.
“Table 3 quantifies this mismatch under a controlled single-turn comparison: matched on model, hardware, backend, tensor-parallel degree, and C=80, coding-agent planning calls have a median 2.8x higher TTFT and 1.3x higher TPOT than chat across 40 released trace_replay groups.” · Show in paper
Table 3 directly reports the 2.8x median TTFT ratio and 1.3x median TPOT ratio across 40 matched pairs at C=80.
“Across all evaluated models, Code A tasks have substantially higher pass rates than B* tasks. For example, Gemini reaches 34.0% pass@3 on Code A but only 4.0% on Code B*, while GLM-4.7 reaches 34.0% versus 6.0%. This gap supports the intended use of the metadata: A-category tasks provide cleaner learning signals, while B-category tasks identify settings where tests, specifications, or environments introduce confounding factors.” · Show in paper
Table 7 shows a consistent, large gap in pass@1 and pass@3 between Code A and Code B* tasks across all five evaluated models.
“Table A1 shows that larger declared dependency sets are associated with lower first-attempt success in Python, Java, and C++. The strongest negative association appears in C++, while JavaScript shows the same direction but not a statistically meaningful effect in the current benchmark.” · Show in paper
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated SoftwareTable A1 provides the exact Spearman correlation coefficients and p-values supporting the negative association in Python, Java, and C++.
“Directed toward planning, sixteen Luna workers come within 1.4 points of an all-Sol team at 70% lower cost (Figure 8), making them roughly 4× more efficient.” · Show in paper
The data in Figure 8 and Section 7 directly support the 1.4-point gap and 70% cost reduction, though Figure 8 notes that fixed-size team runs are single runs compared to three runs for dynamic teams.