Search publications
“We use the SWE-Agent [16] scaffold. ... For open-source LLMs, we use vllm to host each model. Models are hosted on a single node, with 8 H100 Nvidia GPUs. ... Models have a maximum of 50 turns. We use the same prompt for all models, which is the default prompt from [16].” · Show in paper
“The real loss came from a confound in public data. The part of the loss that remains came from a public tool dataset whose supervised split contains no calls at all, so its style of request taught the model to ask instead of act. We show this by ablation and give a fix (§6).” · Show in paper
The ablation in Table 4 and Section 6.2 supports the diagnosis of the confound; the authors also transparently note that their proposed fix has not yet been trained on.
“Specifically, RQ3.1 analyzes the differences in agent steps, tool calls, and token usage across different configurations of NanoHarness, while RQ3.2 measures the distribution patterns of probing behavior and uses case studies to illustrate shifts in failure modes.” · Show in paper
“The limit is hearing. After training, the model picks the right tool on 94% of held-out turns but hears names, numbers and codes right on 72%. With the caller’s words typed instead of spoken, the same model gets 99% of those details and handles 92% of turns exactly rather than 67%. Training the audio projector as well does not help (§5.3).” · Show in paper
The experimental results in Tables 1 and 3 and Section 5.3 support all numbers and conclusions in the claim.
“As shown in Figure 10(c), synthetic distractors lead to slightly lower performance than the other types. This suggests that in-distribution distractors, which are structurally similar to relevant content, make the task harder compared to other distractors.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacksTable 4 and Figure 10(c) directly show lower accuracy for synthetic distractors than for natural distractors or random noise.
“R3 achieves the highest consequential rate across all three venues (87.9–93.2%), compared to 65.5–77.5% for Gemini 3 Pro (all padj < .001, |d| ≥ 0.79), 73.7–81.4% for GPT-5.2 (all padj < .001, |d| ≥ 0.50), and 60.0–67.7% for humans (all padj < .001, |d| ≥ 1.00) (Figure 5A).” · Show in paper
Figure 5A and its caption support the reported consequential rates, significance levels, and effect sizes.
“Strategy-induced stylistic drift is visible even when judge metrics remain stable. For ChatGPT, CoT does not materially change audit/accuracy but substantially reduces ROUGE-L (0.247→0.192), indicating more paraphrastic or verbose generations. For Qwen, CoT/reflection improve judge-based correctness but decrease ROUGE-L/BLEU (e.g., ROUGE-L drops by 0.042 under CoT), suggesting that reasoning prompts yield answers that are judged more correct yet less surface-aligned with the GT wording.” · Show in paper
Table 2 confirms the divergence between ROUGE-L/BLEU and judge-based metrics for ChatGPT and Qwen.
“Across five backbones and four benchmarks at a 10% GPU-resident KV budget, SEKV outperforms every method that fixes resolution at prefill, improving over the strongest semantic baseline by 2.9 points (5.5% relative).” · Show in paper
The averages in Table 1 and per-backbone breakdown in Table 7 directly support the 2.9-point (5.5% relative) improvement over SentenceKV and the superiority over all tested fixed-resolution baselines.
“Among the 124 runs in which an agent terminates on its own before the budget expires, a different limitation becomes clear: weak self-verification. We define a false finish as an early exit at relatively high reward, where the agent stops despite having not yet satisfied the hidden verifier. We identify 14 such runs with R >= 0.75.” · Show in paper
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based GradingThe counts, examples, and per-model early-exit mean rewards in Figure 6 and Section 3.5 support the claim.
The counts (1/10, 2/10, 9/10) and position audit match §5.4, and the authors appropriately note that the other three base models hold cold as well.
“In Table 2 we see the frontier models solve about 99% of tasks, and each of the 642 tasks is solved by at least one of them, which independently confirms that every task is solvable.” · Show in paper
Both reference-patch checks and model evaluation results confirm that all 642 tasks are solvable.
“The suite contains 274 general and 26 multimodal tasks across six categories and 17 subcategories (Figure 1). ... Across the 300 tasks, 1,408 scored modules assess delivery requirements, with three to seven modules per task.” · Show in paper
“The results show that the compression ratio alone is a poor predictor of end-to-end performance.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context ServingThe joint evaluation across task quality, TTFT, throughput, and realized compression ratio clearly demonstrates that compression ratio alone does not predict end-to-end system or quality outcomes.
“Once we have agent success rates and time ratings for each task, we fit time horizons by performing the following logistic regression: p_success(agent,task) = σ((log h_agent − log t_task) · β_agent)” · Show in paper
“Stage 0: Adapter warmup. We freeze both the encoder and decoder and update only the adapter. • Stage 1: Encoder training. We unfreeze the encoder while keeping the decoder frozen. • Stage 2: End-to-end continual pre-training. We unfreeze the decoder with a small learning rate. • Stage 3: Supervised fine-tuning. We train on the SFT mixture and increase the decoder learning rate.” · Show in paper
“To evaluate a proposed solution, we apply the generated patch, using unix’s patch program, to the codebase and then execute the unit and system tests associated with the task instance. If the patch applies successfully and all of these tests pass we consider the proposed solution to have successfully resolved the issue. The metric for our benchmark is the percentage of task instances that are resolved.” · Show in paper
“For every plotted setting, BridgeKV improves over the fine-tuned decode anchor while keeping the same small model for generation. However, the magnitude of the gain depends on the task and model pair. As shown, BridgeKV improves the accuracy of the directly fine-tuned decode anchor by up to 8.4% relative (NarrativeQA, for Qwen2.5-Instruct-7B→Qwen2.5-Instruct-0.5B). In the worst case, the relative improvement is 0.2% (SAMSum, for Llama-3.1-Instruct-8B→Llama-3.2-Instruct-3B)” · Show in paper
Figure 3 confirms that BridgeKV outperforms the parameter-matched decode anchor in all 30 settings, with relative gains ranging from 0.2% (SAMSum, Llama 8B->3B: 45.3 to 45.4) to 8.4% (NarrativeQA, Qwen 7B->0.5B: 56.8 to 61.5).
“To find high-quality task instances at scale, we use a 3-stage pipeline as follows. Stage I: Repo selection and data scraping. We start by collecting pull requests (PRs) from 12 popular open-source Python repositories on GitHub, producing about ~ 90,000 PRs in total. ... Stage II: Attribute-based filtering. We create candidate tasks by selecting the merged PRs that (1) resolve a GitHub issue and (2) make changes to the test files of the repository... Stage III: Execution-based filtering. For each candidate task, we apply the PR’s test content, and log the associated test results before and after the PR’s other content is applied. We filter out task instances without at least one test where its status changes from a fail to pass (henceforth referred to as fail-to-pass test). We also filter out instances that result in installation or runtime errors.” · Show in paper
“We identify that a primary source of output degradation is not the residual attention mass on evicted tokens, which existing methods already minimize, but a directional mismatch between the retained and evicted token sets. Specifically, the evicted tokens in practice are often near-orthogonal to the retained ones. Thus, even a small evicted mass could have an oversized impact on the resulting direction distribution and amplify into substantial output error.” · Show in paper
The mathematical decomposition in Section 3 and empirical measurements in Figure 2 substantiate that directional mismatch drives the output error.
“The per-benchmark results in Appendix B show that degradation is broad on HumanEval, IFEval, MMLU-Redux, and BBH, while TruthfulQA remains comparatively stable.” · Show in paper
Table 5 clearly shows large drops on four of the five benchmarks and minimal change on TruthfulQA.