MFA

Headlines · 1

1. Oct 23
   Virtual Head Attention targets long-context LLM efficiency

   The r/LocalLLaMA post introduces Virtual Head Attention (VHA), an attention method using Q Premix and Linear Postmix to address tradeoffs among KV-cache size, compute, and quality. The authors report that VHA can be folded into a standard GQA-2 architecture for inference, beat a GQA baseline across four pretraining scales, reduce KV cache by 4× versus GQA-8 at dense scales and 2× versus GQA-4 at the MoE scale, and lower training and prefill FLOPs by 7.9% in a reported 1.7B, 4K-context run.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my MFA experienceAsk about MFA