TKintegers K5 ternary weights claim 3.21x prefill, 3.10x decode over TorchAO FP8
A developer on r/MachineLearning released TKintegers, a balanced-ternary weight representation with a converter, GPU inference kernels, benchmark scripts and recorded evidence, reporting that on Qwen2.5-0.5B-Instruct with a fixed matched workload on an NVIDIA L4 the K5 implementation reached 3,402.85 tokens/s prefill versus 1,061.58 for the TorchAO dynamic E4M3 FP8 baseline (3.21x) and 26.75 tokens/s decode versus 8.63 tokens/s (3.10x). K5 encodes each weight code as five balanced ternary digits giving 243 possible values stored in one byte, and the kernels expand bounded tiles while using FP16 tensor-core operands with FP32 accumulation, even though the L4 has native FP8 tensor-core support. The author states these numbers apply only to this model, workload and implementation, that the baseline was TorchAO dynamic E4M3 FP8 running in eager PyTorch, and that the FP16 control was faster than both.
The release includes the converter, native kernels and benchmark scripts, so a ternary weight format that reportedly outperforms an FP8 baseline on existing FP8-capable hardware is testable, though the figures remain self-reported and await independent replication.
I built TKintegers, a balanced-ternary weight representation, converter, and GPU inference implementation.
The result that stood out to me: the NVIDIA L4 has native FP8 tensor-core support, but my K5 implementation achieved higher throughput than the TorchAO FP8 baseline—even though the GPU has no native K5 instructions.
On Qwen2.5-0.5B-Instruct, using a fixed matched workload:
- Prefill: K5 achieved 3,402.85 tokens/s versus FP8’s 1,061.58 — 3.21× throughput.
- Decode: K5 achieved 26.75 tokens/s versus FP8’s 8.63 — 3.10× throughput.
The baseline was standard TorchAO dynamic E4M3 FP8, running in eager PyTorch. Decode measured fixed-token model forwards with a carried KV cache.
#TKintegers K5#TorchAO#NVIDIA L4#Qwen2.5-0.5B-Instruct#PyTorch#NVIDIA