GQA

Headlines · 2

1. Nov 3
   Trade-offs in Distributed Inference for Attention and MoE

   The report analyzes distributed-inference strategies across tensor, data, expert, context, and pipeline parallelism using weight, KV-cache, activation, communication, capacity, and latency budgets. Its author argues that GQA can shard KV heads up to the KV-head count, while MLA's shared latent cache must be replicated under tensor parallelism, and compares MoE tensor parallelism with expert parallelism in terms of communication, memory use, load balancing, and implementation complexity. The report title labels it installment 2, while the body calls it the third of eight installments in an inference-infrastructure methodology series. Its comparative figures and conclusions are the author's analysis and are not independently corroborated by the supplied evidence.

   掘金 · 人工智能 · 🔥 0
2. Oct 23
   Virtual Head Attention targets long-context LLM efficiency

   The r/LocalLLaMA post introduces Virtual Head Attention (VHA), an attention method using Q Premix and Linear Postmix to address tradeoffs among KV-cache size, compute, and quality. The authors report that VHA can be folded into a standard GQA-2 architecture for inference, beat a GQA baseline across four pretraining scales, reduce KV cache by 4× versus GQA-8 at dense scales and 2× versus GQA-4 at the MoE scale, and lower training and prefill FLOPs by 7.9% in a reported 1.7B, 4K-context run.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my GQA experienceAsk about GQA