Topic: speculative

HyFlash is a new prefix-memory prior designed to enhance on-device speculative decoding by predicting multiple future positions during a single pass through the backbone, addressing limitations in drafter capacity and context storage under tight memory constraints. This advancement aims to accelerate parallel block drafting while managing limited device resources effectively.

arxiv.org↗

A new study by TraceSpec examines speculative decoding in large language models (LLMs) to better understand and improve how LLM agents make decisions during task execution, emphasizing the need for more detailed supervision.

arxiv.org↗

Polymathic AI and Princeton researchers have developed FlashMap, a speculative decoding drafter that can draft 16 tokens in one pass, achieving up to 6.14 times faster lossless speedup on Qwen3-8B, significantly enhancing LLM decoding efficiency.

A new method called SPECMEM aims to accelerate tool-calling agents by using speculative decoding and live memory management, reducing latency without altering accuracy. This advancement is crucial as it addresses the growing issue of deployment latency in tool-calling agents, making their use more efficient and practical.

arxiv.org↗