Topic: prefill

A new method called Recursive Sparse Attention allows for efficient processing of up to one million tokens, significantly reducing computational costs compared to dense attention techniques, thus enabling better handling of longer documents in machine learning models. This breakthrough is crucial as it addresses the scarcity of long training data and improves the scalability of language models.

arxiv.org↗

BlockLock introduces a method for ensuring reproducible Large Language Model (LLM) inference by maintaining fixed batch geometry, addressing limitations of existing approaches that either constrain every operation or isolate prefilling. This technique is crucial for applications requiring consistent outputs across different requests in the same batch.

arxiv.org↗

Zyphra released open-source FP8 and BF16 prefill attention kernels for AMD's MI355X, achieving up to 1.76x faster performance with significantly lower error rates compared to AMD’s AITER, highlighting potential advancements in GPU efficiency for AI models.

vLLM released version 0.28.0 with significant optimizations, including Decode Context Parallel (DCP) support and fused FlashKDA decode and prefill kernels, benefiting from 584 commits by 270 contributors. This update enhances performance, particularly for Kimi-K3, highlighting the community's contribution to the project.

github.com↗