Search publications
“the original high-precision pretrained checkpoint serves as a frozen teacher to supervise a low-precision student initialized from the same weights. Given an input sequence of length L, the framework minimizes the token-averaged forward Kullback–Leibler (KL) divergence between the teacher distribution pT,t and the student distribution pS,t over their shared vocabulary V” · Show in paper
“Trace replay profiles preserve the exact recorded requests from the source traces. Distributional synthetic profiles sample multi-turn requests from per-turn ISL, OSL, and turn-count distributions fitted to those traces, producing representative workloads that can be reissued across models, hardware, backends, and concurrency settings, without the runtime cost of full replay.” · Show in paper
“Query and output matrices are efficiently stored as ragged tensors (also known as jagged arrays) (Tensorflow Developers, 2018) without padding, which facilitates the compact packing of queries and outputs from diverse requests into a single tensor.” · Show in paper
“Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K.” · Show in paper
The results in Table 3 and Tables 10-11 directly support the claim; 10 of 17 models (roughly half) pass the 85.6% threshold at 32K, while nearly all achieve near-perfect scores on vanilla NIAH.
“Each gate first computes a head-specific embedding, but the final scalar score is produced by a shared projection: gℓ,h(x) = σ (w⊤g Projℓ,h(x) + bg)” · Show in paper