Search publications
“There is also a significant performance gap between the public and commercial set, where the best models score less than 20% in the commercial set, highlighting the difficulty of navigating enterprise codebases.” · Show in paper
Tables 1 and 2 directly show that all tested models score below 20% on the commercial set and significantly higher on the public set.
“We conduct all our experiments using the open-weight Llama 3.1-Instruct model, which is an instruction-finetuned version of Llama 3.1 (Meta AI 2024). ... For our RAG experiments, we make use of an off-the-shelf and a specialised multilingual models – Jina-reranker (JinaAI) (Multilingual 2024) and paraphrase-multilingual-mpnet-base-v2 (MPNet) (Reimers and Gurevych 2020), respectively.” · Show in paper
“To this end, we introduce SWE-bench, an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories.” · Show in paper
The dataset construction pipeline, repository breakdown (Figure 3, Table 10), and task statistics (Table 1) substantiate the existence, size, and automated collection of the 2,294-instance benchmark.
“For each benchmark, later model generations unlock a larger proportion of tasks. This is indicated by a positive, significant main effect of model generation on the probability of unlocking a task on all six benchmarks (Table 4), with the largest effect on FrontierMath and Cyber CTFs.” · Show in paper
All coefficients and significance levels in Table 4 match the stated claim.
“Drift-act traces take two main shapes, on the results lane's codes. Either the cost is named and argued past (158 of 185, mostly with the world stated correctly), or it is never mentioned (27, mostly with a fact of the scene stated wrongly).” · Show in paper
The counts (158/185, 27/185, 0/185, and 4/185 on blind re-rate) match §5.6, and the authors explicitly note that the blind second rater is from the same model family.
“Runs use a 200K-token input-context threshold, beyond which each harness applies its default context management, and a 150-turn interaction limit. ... Each task–model–harness combination is evaluated once.” · Show in paper
“On average, the reference solutions span 107.4 lines of code across 4.1 files. Every problem involves at least 10 lines of change, and over 100 tasks demand more than 100 lines of modification.” · Show in paper
The reported averages and distributions in Figures 1 and 2 support the claim of substantial multi-file modifications.
“However, Table 3 shows that, in our long multimodal document QA setting requiring multi-hop evidence, Colpali-RAG substantially degrades end-to-end answer quality even when Gemini is used as the downstream generator.” · Show in paper
Table 3 directly supports the substantial degradation in Audit, Accuracy, and Fidelity for Colpali-RAG relative to all three Gemini-only baselines.
“Each stage-specific guidance instructs the lead to complete the full workflow while using workers only in the designated phase. This design isolates the value of delegation at each phase. We then compose all three interventions into PIV, our heavy-guidance variant, to evaluate their joint effect when workers are used throughout the workflow” · Show in paper
“3.2. Generational gains are driven mainly by reach and reliability rather than efficiency” · Show in paper
The regression results in Table 4 and curve summaries in Table 9 support that reach and reliability improve more consistently across generations than token efficiency.
“The full NanoHarness configuration achieves the strongest performance among variants, improving over mini-SWE-agent on both models and outperforming any single component. Although NanoHarness still slightly trails product-level harnesses such as OpenCode and Claude Code, it closes most of the gap with a lightweight modular design, suggesting that a compact set of representative components can reproduce much of the benefit of complex harnesses.” · Show in paper
Table 6 and Figures 11-12 confirm that full NanoHarness outperforms all single-component variants and closes the gap to OpenCode and Claude Code to within 1.13–2.08 percentage points.
“Adding a shared 1024-token synthetic prefix brings deep-turn E2EL MAPE to 3.2% and aggregate E2EL MAPE to 2.4%, within the observed noise scale. Hence synthetic distributional profiles in this paper use this prefix-aware configuration.” · Show in paper
Table 11 directly reports the ablation results confirming 2.4% aggregate and 3.2% deep-turn E2EL MAPE.
“In general, there is a positive correlation (r = 0.436, p < 0.001) between human-predicted and empirical difficulty, and 93.3% of human-hard tasks are also empirically hard (Figure 7).” · Show in paper
Figure 7 and the accompanying text in Section 4.3 directly support the correlation and percentage breakdowns.
“Corollary 3.2 provides an intuition for why eviction can improve long-context behavior. Its condition, , only requires useful tokens to be retained at a higher logit-weighted average rate than distractors. Thus, any retention rule that suppresses distractor mass more than useful-token mass reduces distractor-induced dilution.” · Show in paper
The proofs of Proposition 3.1 and Corollary 3.2 in Appendix A.1 are mathematically sound, and Figure 1 provides consistent empirical support.
“As shown in Figure 6, our agent substantially improves performance over the same LCLM with 16x compressed context and no expansion.” · Show in paper
Table 36 and Figure 6 provide exact per-task and average accuracy numbers showing large gains (+18.50, +23.19, and +25.50 points at 4k, 8k, and 16k).
“For each problem, we build a complete but deliberately broken terminal-only project. ... Each task includes reproducible public asset-generation scripts, a weak baseline implementation, an official gold solution, a multi-step solve.sh, and a hidden verifier.” · Show in paper
“Most of the measured loss in general tool use is the prompt. Under a benchmark’s own prompt and call format, the tuned models look as if they had lost general tool use. Under the prompt they are deployed with, round 1 writes a valid call for 99% of the VoiceAgentBench requests that need a tool (§6).” · Show in paper
Table 4 and Section 6.1 directly substantiate the contrast between the benchmark prompt and the deployed prompt.
“The pooled edit exceeds the unedited model by 2.7 tasks over 12 batches (95% CI +0.3 to +5.0, p = 0.03; higher in 9 batches, sign test p = 0.15). It exceeds the mean of the random controls by 3.6 tasks in the 7 batches that contain both (CI +2.2 to +5.1, p < 0.001; higher in 7 of 7, sign test p = 0.016).” · Show in paper
The numbers in Table 2 support the claim, and the authors appropriately qualify that the 0.125–0.5 pool is post hoc and the development tasks are in-sample.
“Trace replay profiles preserve the exact recorded requests from the source traces. Distributional synthetic profiles sample multi-turn requests from per-turn ISL, OSL, and turn-count distributions fitted to those traces, producing representative workloads that can be reissued across models, hardware, backends, and concurrency settings, without the runtime cost of full replay.” · Show in paper
“On GSM8K [41] with 8-shot chain-of-thought prompting [42] and exact-match scoring (n=1319 per cell), JoLT is near-lossless at 2× and 3× on both Mistral and LLaMA-2 (Table 7 in Appendix C). All free-zone deltas fall within their 95% confidence intervals of zero, with the largest shift at −1.21 pp (Mistral 3×).” · Show in paper
Tables 7 and 10 support the claims with the stated caveats (single model and single seed for LongBench).