Search publications
“AI agents look for correctness bugs in open-source research software. A separate referee rebuilds each claim from scratch. Only bugs the referee confirms are listed here. Each bug was reported to the authors before it appears here.” · Show in paper
The summary table and the 14 detailed project sections consistently document the confirmed bugs, pinned commits, and independent referee/reproduction checks.
“We evaluate seven harnesses with ten backends on general tasks and seven on multimodal tasks, using paired comparisons on shared tasks. Our evaluation reveals substantial, model-dependent performance differences: no harness performs best across all ten backends, and GLM-5.3’s general-task full-credit rate varies by 13.5 percentage points across harnesses.” · Show in paper
The full-suite tables (Tables 2 and 3) directly show that no single harness leads across all ten backends on general tasks or all seven backends on multimodal tasks.
“To bypass GitHub API rate limits and enable large-scale processing, we clone repositories in distributed map-reduce jobs and extract pull request patches directly from local git histories, avoiding per-instance API requests.” · Show in paper
“Segmentation: fixed-length chunks of the same mean length cost 2.4 to 2.9 points and dropping anchors 3.0 to 3.9, so where spans begin and which tokens stay exact both matter.” · Show in paper
Table 3 directly supports the stated point drops.
“Planning workers improve reward at every team size: two workers already improve on the solo agent by 3.2%, and sixteen raise the gain to 7.1%. Implementation and verification workers, in contrast, improve from two to four workers but then plateau, with their reward changing by −0.015 and −0.011, respectively, when scaling to 16 workers.” · Show in paper
Figure 8 and Appendix C.8 (Figure 24) substantiate both the scaling behavior and the underlying mechanism.
“All-KV was not the consistent winner in terms of accuracy across the various workloads in our benchmark. Instead, under most workloads, one of the compression techniques achieved even better results.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingTable 2 directly shows compressed methods outperforming All-KV on most model-dataset combinations.
“We follow our data collection procedure and collect 19,000 issue-PR pairs from an additional 37 popular Python package repositories. In contrast to Section 2.1, we do not require that pull requests contribute test changes. ... To eliminate the risk of data contamination, the set of repositories in the training data is disjoint from those included in the evaluation benchmark. ... For memory efficiency, we fine-tune only the weights of the attention sublayer using LoRA Hu et al. (2022), and exclude training sequences with more than 30,000 tokens, reducing the effective size of the training corpus to 10,000 instances.” · Show in paper
“As shown in Table 5, using the token contextual embedding xt as input performs better, particularly under tight budgets, suggesting that xt provides a stronger long-horizon utility signal than the projected KV representations alone.” · Show in paper
Table 5 directly supports the claim.
“We conduct an LLM-as-a-judge analysis for failure modes of different models, utilizing GPT-5 as the judge. ... For each of the models in Table 4, we programmatically filter to only unresolved instances of SWE-BENCH PRO and collect the last 20 turns of each rollout. ... With a system prompt providing strict descriptions of the failure buckets and overall SWE-Agent format, we feed the trajectory input and prompt the GPT-5 judge to first produce a 1-paragraph reasoning and then an ultimate selection of one failure mode per instance.” · Show in paper
“As shown in Figure 3, RocketKV performs coarse-grain KV cache eviction in the first stage. The purpose of this stage is to remove KV tokens with low importance while keeping the majority of important tokens. In the second stage, it conducts fine-grain dynamic KV token selection on the remaining KV tokens, followed by top-k sparse attention.” · Show in paper
“On Full-Duplex-Bench v3, people speak the requests, and the tools are again unseen. Here fine-tuning costs nothing measurable (Table 5): all three models pass about half the requests, tool-selection F1 is 91–95%, and r2 matches round 1 (paired difference in F1 +0 [−3, +4]).” · Show in paper
Table 5 directly supports the claim, and the authors appropriately qualify that they ran it turn-based with script scoring rather than full-duplex with an LLM judge.
“We fine-tune Qwen 2.5-72B-Instruct using annotations from SWE-bench Verified. For each of the over 3,800 examples, the model receives the issue description, the canonical solution patch, and the test patch as input. It is then prompted to predict one of three binary quality labels: Issue Clarity, Task Complexity, or Test Patch Correctness.” · Show in paper
“The average costs span from about $3.6 to $26 per task. Although Grok 4.5 achieves the highest pass rate, it does so at only about $11 per task, well below the most expensive models. GPT-5.6-sol and GPT-5.5 form a second tier of strong but substantially pricier models at roughly $21 per task, and GPT-5.4 is the most expensive overall (about $26 per task) with a much lower pass rate.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based GradingTable 1 and Figure 5 directly support the cost-efficiency comparison and the conclusion that higher spending does not guarantee higher pass rates.
“Comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) at ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition–retention frontier. Low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at an increasing cost in performance on unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime.” · Show in paper
The five-seed results in Figure 1 and Table 5 directly support the full claim, including the relative positions of FFT and QLoRA ranks 8, 16, 32, and 64.
“For total budget B, we estimate pass@k by splitting B across k independent trajectories of B/k tokens each and counting a task as solved if any trajectory’s first correct submission falls within budget. pass@k is computed task-wise using the unbiased estimator of Chen et al. (2021).” · Show in paper
“The pipeline has five stages: 1. Preliminary Data Collection: Mining and filtering candidate PRs from global GitHub activity. 2. Setup Synthesis: Deploying an interactive agent to autonomously infer repository-level installation and testing scripts. 3. Execution-based Validation: Validating environments through dual-pass execution (pre- and post-fix) to extract test oracles. 4. Filtering By Issue Clarity: Removing underspecified tasks and excluding instances with potentially sensitive information. 5. Metadata Enrichment: Tagging instances with diagnostic features to enable flexible selection.” · Show in paper
“SEKV gains more from extra budget than ShadowKV, so its gap to ShadowKV narrows from 3.4 points at 5% to 1.0 at 20% on RULER and from 2.8 to 0.3 on NIAH, and on LLAMA-3.1-8B SEKV matches ShadowKV on NIAH at 20%.” · Show in paper
Table 4 directly supports all stated numbers.
“Presented in the second row of Figure 5, RocketKV achieves near the Full-KV accuracy across all models, even with a token budget of 256. In fact, it reaches 100% accuracy on Llama3.1-8B-Ins under 256 token budget, which corresponds to a compression ratio of over 400× with a maximum sequence length of 109K tokens. In contrast, all other methods suffer from substantial accuracy drops.” · Show in paper
The NIAH scores in Figure 5(d-f), Table 2, Table 10, and Table 16 directly support the claim.
“Our agent evaluation protocol divides this agent behavior into two skills, a generation skill, in which it produces code and a dependency manifest for a given task, and a repair skill, in which it iteratively revises both specified code and dependency manifest in response to runtime failures.” · Show in paper