NVIDIA L4

Headlines · 2

1. 2d
   TKintegers K5 ternary weights claim 3.21x prefill, 3.10x decode over TorchAO FP8

   A developer on r/MachineLearning released TKintegers, a balanced-ternary weight representation with a converter, GPU inference kernels, benchmark scripts and recorded evidence, reporting that on Qwen2.5-0.5B-Instruct with a fixed matched workload on an NVIDIA L4 the K5 implementation reached 3,402.85 tokens/s prefill versus 1,061.58 for the TorchAO dynamic E4M3 FP8 baseline (3.21x) and 26.75 tokens/s decode versus 8.63 tokens/s (3.10x). K5 encodes each weight code as five balanced ternary digits giving 243 possible values stored in one byte, and the kernels expand bounded tiles while using FP16 tensor-core operands with FP32 accumulation, even though the L4 has native FP8 tensor-core support. The author states these numbers apply only to this model, workload and implementation, that the baseline was TorchAO dynamic E4M3 FP8 running in eager PyTorch, and that the FP16 control was faster than both.

   r/MachineLearning · 🔥 0
2. 3d
   AWS shows distributed PyTorch training with Ray Train DLCs on EKS

   AWS Machine Learning Blog provides a walkthrough for fine-tuning Qwen2.5-1.5B across eight NVIDIA L4 GPUs on two g6.12xlarge instances using Amazon EKS, KubeRay, Ray Train, PyTorch FSDP, and EFA. AWS says its Ray Train Deep Learning Container packages and patches the required CUDA, PyTorch, NCCL, EFA, and Ray components, while the sample uses one RayCluster manifest and requires no custom image build. AWS reports that enabling the EFA path doubled throughput over TCP with unchanged code, though the performance claim comes from the vendor.

   AWS Machine Learning Blog · 🔥 0

Experience and discussion from the community

Share my NVIDIA L4 experienceAsk about NVIDIA L4