Topic: batch

BlockLock introduces a method for ensuring reproducible Large Language Model (LLM) inference by maintaining fixed batch geometry, addressing limitations of existing approaches that either constrain every operation or isolate prefilling. This technique is crucial for applications requiring consistent outputs across different requests in the same batch.

arxiv.org↗

The ggml/llama.cpp project released a new version that limits shared prefixes to the first required output position, ensuring decision output embeddings remain available. This update addresses issues with shared options in tasks and enhances the system's flexibility in handling multiple questions simultaneously.

github.com↗

The ggml/llama.cpp project added initial OpenCL support for IQ2, enhancing matrix multiplication methods specifically for Adreno GPUs. This update is crucial as it broadens the library's compatibility and performance on mobile devices with Adreno hardware.

The ggml/llama.cpp project has updated to include new models LiquidAI/d1-2B and LiquidAI/d1-omni-2B, expanding its model offerings. These updates are significant as they enhance the project's capabilities in handling speech tasks and provide more options for users interested in these specific models.