Search publications
“We employ the mini-SWE-agent v1.14.4 (Yang et al., 2024) scaffold with Qwen3-Coder-480B-A35B-Instruct as the underlying model to generate installation and test scripts.” · Show in paper
“On the 642 V2 tasks, frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%.” · Show in paper
Table 2 and Figure 1 directly support the reported pass-rate ranges and separation across systems.
“Second, interactive systems consistently outperform non-interactive ones: for example, an interactive agent using Qwen3-Coder-30B-A3B-Instruct can exceed a non-interactive pipeline even when the latter is driven by Qwen3-Coder-480B-A35B-Instruct. Third, increasing the number of setup attempts improves success probability: moving from one to ten runs substantially increases the fraction of installed repositories, in some settings approaching a twofold improvement.” · Show in paper
Table 2 directly confirms both that the 30B interactive agent outperforms the 480B non-interactive baseline at all pass@k values and that pass@10 is roughly double or more than pass@1 across interactive configurations.
“We introduce SWE-rebench V2, a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. ... Using this pipeline, we construct a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description.” · Show in paper
The pipeline design, stage-by-stage yield counts in Table 1, and dataset statistics directly support the claimed scale and language coverage.
“To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale.” · Show in paper
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering AgentsThe manuscript thoroughly documents the four-stage automated pipeline, provides concrete funnel metrics, and reports the exact counts of 21,336 tasks (3,468 repos) for the dataset and 294 tasks (169 repos) for the benchmark subset.
“When you run both prompts in fresh Claude sessions, compare them across four concrete failure metrics: 1. Truncation Cutoff Point 2. Early Token Bloat (Items 1–10 vs. Items 35–45) 3. Resumption Friction 4. Information Density” · Show in paper
“To quantify leakage in PR-derived tasks, we evaluated 509 PR-derived tasks constructed from SWE-bench Pro pull requests. An LLM judge compared each generated statement against the reference problem statement, requirements, and interfaces, and rated whether the statement introduced implementation details beyond the reference. Under this protocol, 392/509 tasks (77.0%) were clean, 117/509 (23.0%) contained some leakage, and only 12/509 (2.4%) contained explicit solution leakage.” · Show in paper
The reported counts (392/509 clean, 117/509 some leakage, 12/509 explicit leakage) directly support the claim under the stated LLM-judge protocol.
“Managed coding assistants, Claude Code (Opus 4.6) and Codex (GPT-5.4), agree on the primary defect category, including no defect, at κ = 0.80 over all 89 TB2.0 tasks. Lower-cost API auditors agree on the binary triage decision (flagged vs. clean) over all 89 tasks (Table 1), but less on attribution: per-category agreement (one-vs-rest, since a task can carry multiple labels) is moderate for instruction–verification mismatch (κ = 0.60) and environment issues (κ = 0.59), and low for task drift (κ = 0.21–0.41).” · Show in paper
Table 1 and Section A.2 support the reported kappa values for both managed coding assistants and lower-cost API auditors.
“We primarily use ProgramBench (Yang et al., 2026a), a from-scratch repository reconstruction benchmark comprised of 200 unique tasks ... Additionally, we also explore Commit0-Lite (Zhao et al., 2024) a benchmark comprised of 16 Python libraries that an agent is asked to reimplement given a repository skeleton and a set of unit tests.” · Show in paper
“As shown in Table 8, compared to an average gold patch, model generated patch files that apply correctly are less than half the total length (74.5 versus 30.1 lines) of gold edit patch files, and rarely edit more than a single file.” · Show in paper
Table 8 confirms that all evaluated models' applied patches average at most 30.1 total lines (less than half of All Gold's 74.5 lines) and 1.0-1.1 files edited.
“We replay whole-session shapes—initial context, per-turn output, per-turn context growth and inter-turn gaps—of 200 trajectories sampled from public mini-SWE-agent leaderboard runs on SWE-bench Verified by six 2026 models” · Show in paper
“In Figure 13, LCLMs achieve the highest accuracy across all compression ratios on GSM8K, with particularly strong gains over baselines at higher compression ratios.” · Show in paper
Table 9 and Figure 13 directly support the claim across all three compression ratios.
“Controlling for task length, we find models perform worse on tasks that have higher messiness scores. Notably, trends in AI agent performance over time are similar for the lower and higher messiness subsets of these tasks (see Figure 12). In particular, we find no evidence of plateaus in performance trends specific to the higher messiness subset.” · Show in paper
The regression results and split trend plots in Figures 12 and 13 support the claim.
“Claude's initial C++ failures are overwhelmingly due to incorrect system-library assumptions (SLAR of 87.5%), and every one of those failures is eventually repaired after feedback. Codex shows a lower but still meaningful assumption rate (17.9%), again with perfect recovery once the missing package is revealed. Gemini's C++ failures arise from other causes rather than missing system libraries.” · Show in paper
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated SoftwareTable 6 and Section 6.4 directly substantiate the SLAR and EGAR figures for all three agents.
“All evaluations on SWE-rebench are conducted by our team by using a fixed scaffolding, i.e., every model is assessed by using the same minimal ReAct-style agentic framework [Yao et al., 2023], identical prompts and default generation hyperparameters as recommended by model developers. We standardize the context length to 128K tokens for all evaluations, unless a model only supports a shorter context.” · Show in paper
“We segment the input into fixed-size chunks of 1,024 encoder tokens, compress each chunk, and assign it a segment identifier. The model receives the entire compressed sequence in a single prompt, together with one expand tool. At each turn, the model can issue a native tool call to expand with a segment identifier (Figure 6), and the tool returns the original text of that segment.” · Show in paper
“Finding 4: With the same training-token budget, QAD on fewer 32K sequences generalizes better than QAD on more 4K sequences, improving average accuracy across AIME25, BigCodeBench, and LiveCodeBench-v6 by 1.9 percentage points. QAD also generalizes across domains, with the strongest transfer observed from code training to mathematical reasoning.” · Show in paper
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMsThe token-matched experiments in Table 7 and analysis on page 20 substantiate both the 1.9-point gain and cross-domain transfer.
“This paper delivers cheap, precise, verifiable control of a coding agent’s decisions by LLM-written weight edits, and a measured account of when such control does and does not raise solve rate.” · Show in paper
The manuscript thoroughly substantiates both the precise, dose-dependent decision control and the nuanced, negative/unconfirmed solve-rate findings with appropriate controls and pre-registered tests.
“The unanimous consensus rate is 0% in every language, meaning Claude never converged to the same dependency declaration across all three trials for any task in any ecosystem. Intra-agent Jaccard scores are low throughout, with JavaScript again the worst case.” · Show in paper
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated SoftwareTable 5 provides complete data from the 400 stochastic trials of Claude Code confirming 0% UCR and low intra-agent Jaccard scores across all four languages.
“In this setting, we find that models generally perform worse at this task than when generating patch files; for instance, Claude 2 scores at 2.2% compared to 4.8% in the main table for “oracle” retrieval. Even when controlling for instance length, generating on the shorter half of the task instances by input tokens yields 3.9% compared to 7.8% for generating patches with Claude 2.” · Show in paper
The reported comparison for Claude 2 (2.2% vs. 4.8% overall, and 3.9% vs. 7.8% on shorter instances) supports the claim.