Qwen2.5-0.5B-Instruct

Headlines · 1

1. 2d
   TKintegers K5 ternary weights claim 3.21x prefill, 3.10x decode over TorchAO FP8

   A developer on r/MachineLearning released TKintegers, a balanced-ternary weight representation with a converter, GPU inference kernels, benchmark scripts and recorded evidence, reporting that on Qwen2.5-0.5B-Instruct with a fixed matched workload on an NVIDIA L4 the K5 implementation reached 3,402.85 tokens/s prefill versus 1,061.58 for the TorchAO dynamic E4M3 FP8 baseline (3.21x) and 26.75 tokens/s decode versus 8.63 tokens/s (3.10x). K5 encodes each weight code as five balanced ternary digits giving 243 possible values stored in one byte, and the kernels expand bounded tiles while using FP16 tensor-core operands with FP32 accumulation, even though the L4 has native FP8 tensor-core support. The author states these numbers apply only to this model, workload and implementation, that the baseline was TorchAO dynamic E4M3 FP8 running in eager PyTorch, and that the FP16 control was faster than both.

   r/MachineLearning · 🔥 0

Experience and discussion from the community

Share my Qwen2.5-0.5B-Instruct experienceAsk about Qwen2.5-0.5B-Instruct