TKintegers K5三值权重推理自测超TorchAO FP8
一名开发者在 r/MachineLearning 发布 TKintegers，包含平衡三值权重表示、转换器、GPU 推理内核、基准脚本与录制证据。其称在 NVIDIA L4 上以 Qwen2.5-0.5B-Instruct 和固定匹配负载测试，K5 预填充达 3,402.85 tokens/s，对比 TorchAO 动态 E4M3 FP8 基线的 1,061.58 tokens/s（约 3.21 倍）；解码为 26.75 tokens/s 对 8.63 tokens/s（约 3.10 倍）。K5 用五个平衡三值位表示每个权重码，共 243 种取值、每码占一个字节，内核展开有界分块并以 FP16 张量核操作数配 FP32 累加。作者强调结果仅适用于该模型、负载与实现，基线是 eager PyTorch 下的 TorchAO FP8，且 FP16 对照组快于两者。
项目附带转换器、原生内核与基准脚本，使这一“在已有 FP8 硬件上超越 FP8 基线”的三值权重方案具备可验证性，但相关数据仍为开发者自测，尚待独立复现。
I built TKintegers, a balanced-ternary weight representation, converter, and GPU inference implementation.
The result that stood out to me: the NVIDIA L4 has native FP8 tensor-core support, but my K5 implementation achieved higher throughput than the TorchAO FP8 baseline—even though the GPU has no native K5 instructions.
On Qwen2.5-0.5B-Instruct, using a fixed matched workload:
- Prefill: K5 achieved 3,402.85 tokens/s versus FP8’s 1,061.58 — 3.21× throughput.
- Decode: K5 achieved 26.75 tokens/s versus FP8’s 8.63 — 3.10× throughput.
The baseline was standard TorchAO dynamic E4M3 FP8, running in eager PyTorch. Decode measured fixed-token model forwards with a carried KV cache.
#TKintegers K5#TorchAO#NVIDIA L4#Qwen2.5-0.5B-Instruct#PyTorch#NVIDIA