Search publications
“Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1. These repairs change evaluation results: accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman ρ = 0.958). Beyond accuracy, we evaluate how resource allocation shapes benchmark conclusions. We show that fixed wall-clock and output-token budgets can materially change apparent agent performance” · Show in paper
All components of the headline claim are directly supported by the reported audit counts (28/89 tasks), the 19-agent evaluation (Spearman rho = 0.958, rank shifts up to 3), and the 32x timeout sweep results.
“We apply three deliberately simple inference-scaling techniques uniformly across every (benchmark, model, condition) cell: 1. Expanded total token budgets of 5M–30M tokens per trajectory (Table 2), sized to task complexity and one to three orders of magnitude above typical benchmark defaults. 2. Context compaction, which replaces earlier turns with a model-generated summary to enable serial scaling beyond the nominal context-window size. We adopt Inspect AI’s default summary-based strategy (UK AI Security Institute, 2024a), triggered when the running context exceeds 130k tokens (about 65% of the smallest 200k context window across models). 3. Iterative resubmission with a hard cap of 999 submissions per trajectory, allowing the model either to refine its previous answer or to try a substantially different approach between submissions. To reduce unproductive loops of near-identical submissions, we also employ a lightweight repetition guard that terminates a trajectory when a separate LLM judge finds three or more consecutive submissions semantically equivalent (Section A.1.3).” · Show in paper
“In contrast, for newer and more complex SE tasks such as ProgramBench and GitTaskBench, complex harnesses can provide greater benefits to stronger models (e.g., Qwen3.7-Max and DeepSeek-V4-Pro).” · Show in paper
Figures 7-10 clearly show that OpenCode underperforms or matches mini-SWE-agent on weaker models but outperforms it on stronger models in both benchmarks.
“By combining these factors in a unified three-dimensional experimental matrix, we evaluate each model under each harness across each task, allowing us to compare their individual effects and analyze how they interact in LLM-based SE agents.” · Show in paper
“The annotation is performed using the gpt-oss-120b model. In addition, for each task we automatically generate auxiliary interfaces extracted from the patches that are explicitly exercised in the test suite.” · Show in paper
“The model is inconsistent, not incapable. Evaluation is greedy. Yet over the 13 unedited runs of batches 3–15, a single run solves 22–30 tasks, 51 different tasks are solved at least once, and 9 are solved in every run.” · Show in paper
The observational tallies and independent recomputation support the claimed inconsistency and within-task associations.
“we collect and organize the available information from original sources, such as issue discussions, commit messages, or pull requests, and produce the final task description with two artifacts: (1) a problem statement, which captures the motivation for the change without extending beyond sources, and (2) a list of requirements and optionally interface, which provides the necessary details to fully understand and resolve the issue, grounded in the gold patch and test expectations when applicable.” · Show in paper
“On average, prompt-based methods lose more than 50% of their baseline accuracy at longer context windows (≥ 32k), and nearly all prompting methods begin to deteriorate sharply after 8k to 16k tokens.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksTable 1 shows substantial accuracy drops for all four prompt-only baselines between 0k and 32k-128k.
“Our extensive experiments with an open-weight large language model (LLM) reveal a pronounced gap between high- and low-resource languages, particularly for tasks requiring the model to aggregate multiple facts or predict the absence of information. We also find that, in multilingual settings, LLMs effectively utilize less than 30% of their claimed context length. Although off-the-shelf Retrieval Augmented Generation helps alleviate this to a certain extent, it does not solve the long-context problem.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksThe experimental results in Tables 1, 5, 6 and Figures 5-8 consistently support the claimed gaps across languages, task types, and effective context length, as well as the partial mitigation by RAG.
“(1) The marginal benefit of complex harnesses generally decreases as model capability improves: on SWE-bench Verified, matched-model gains over mini-SWE-agent drop from +9.60/+13.60 pp to +4.07/+5.47 pp, and in Qwen SWE-bench Pro experiments the OpenCode–mini-SWE gap narrows from 6.00 pp to 2.67 pp. (2) This convergence is not uniform across model families. Qwen follows a smooth narrowing trend, whereas DeepSeek shows a non-monotonic pattern, with OpenCode helping more on intermediate models (e.g., DeepSeek-V3.1, DeepSeek-V3.2) and nearly converging with mini-SWE on DeepSeek-V4-Pro (34.3% vs. 33.7%).” · Show in paper
The reported resolve rates and gaps in Table 3 and Figures 5-6 directly support both the general diminishing-returns trend and the family-specific scaling differences.
“1. An AI agent clones a pinned commit, reads the open and closed issues and PRs first, and looks for correctness bugs: numbers that are wrong on normal inputs. It checks each number against a reference it computes separately, using tiny inputs on a CPU.” · Show in paper
“Team reward closely tracks the capability of the lead: with identical Luna workers, upgrading the lead from Luna to Terra raises reward by 20.9%, and from Terra to Sol by a further 15.4%, and at every tier the team outperforms the solo agent built on the same model. Cheap workers, however, cannot compensate for a weak lead, as a Terra lead with Luna workers still trails a Sol agent working alone by 10.3%.” · Show in paper
Figure 7 and Table 3 support the reported percentage differences across lead tiers.
“Claim tested: the development-set gain of an edit transfers to tasks that played no part in building or selecting it. Neither test confirmed this claim.” · Show in paper
Both pre-registered tests clearly failed to reach their confirmation criteria, and the authors report the exact confidence intervals, p-values, and power limitations.
“Our tasks comprise three distinct task suites: 1. A subset of HCAST [8]: 97 diverse software tasks ranging from 1 minute to 30 hours. 2. RE-Bench [2]: 7 difficult ML research engineering tasks, all eight hours long. 3. Software atomic actions (SWAA): 66 single-step tasks representing short segments of work by software developers, ranging from 1 second to 30 seconds.” · Show in paper
“A key contribution of Long-Horizon-Terminal-Bench is dense, subtask-level grading rather than binary pass/fail, which is essential at this difficulty level. Under a strict threshold (R >= 1.0), 10 of the 17 models pass zero tasks (Figure 3); even at R >= 0.95, only 6.4% of runs pass while 30.8% score below 0.05 (Figure 4). The remaining 62.8% of runs make real partial progress, but binary grading would count them as complete failures by giving them the same score as a run that makes no progress.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based GradingThe leaderboard ties at R >= 1.0 and the reward distribution in Figure 4 directly substantiate that dense grading preserves ranking signal and partial-progress information lost under binary grading.
“The finetuned models SWE-Llama 7b and 13b perform surprisingly poorly with BM25 retrieved context. As these models were fine-tuned using the “oracle” retrieval as context, we suspect this shift in context makes it difficult for the model to perform reliably. For instance, SWE-Llama was trained to edit every file included as context whereas in the BM25 setting many files provided in context are not expected to be changed.” · Show in paper
The performance drop from 3.97% (Oracle) to 0.70% (BM25) for SWE-Llama 13b is documented in Tables 5 and 18, and the explanation aligns with the training setup.
“Based on these criteria, we develop four NIAH tasks. The "needle" in each of these tasks is a key-value pair inserted into the "haystack" (long distractor texts).” · Show in paper
“Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues.” · Show in paper
The experimental results in Tables 2 and 5 support the overarching finding that all evaluated models resolve only a tiny fraction of SWE-bench issues (under 2% for Claude 2, and 3.79% for Claude 3 Opus in the updated Table 5).
“In this setting, we see increases in performance, with GPT-4 jumping from 1.3% to 3.4% and Claude 2 from 4.8% to 5.9%.” · Show in paper
Table 6 and Table 18 confirm the performance gains when collapsing unedited context in the oracle setting.
“SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. ... In our evaluation of widely used coding models, under a unified scaffold, we observe that their performance on SWE-BENCH PRO remains below 45% (Pass@1).” · Show in paper
The dataset counts (1,865 problems across 41 repositories) and the <45% Pass@1 performance ceiling (max 43.6% in Table 1 and 17.8% in Table 2) are directly supported by the reported tables and dataset breakdown.