Embedl int4 GEMM reaches 2.43× llama.cpp on Gemma 4 31B
Embedl’s developer reports that a custom int4-by-int4 WMMA GEMM ran Gemma 4 31B at 2.43× llama.cpp’s speed on Ryzen AI Max+ 395 hardware, with a measured instruction rate of 104 TOPS. Under identical benchmark settings, its prompt-processing results were 679 and 724 versus llama.cpp’s 287 and 276, while GSM8K, IFEval, and MMLU-Pro scores remained close. The implementation uses int4 weights and activations, rotation, GPTQ, and selective mixed precision.
The reported results show that exploiting AMD Strix Halo int4 WMMA instructions could substantially accelerate local Gemma 4 inference without a clear loss in the evaluated quality benchmarks.
The 8060S has an int4 x int4 WMMA instruction at 2x the FP16 rate (we measured 104 TOPS). Nothing in ROCm or llama.cpp uses it. We wrote a GEMM for it and run Gemma 4 with int4 weights and int4 activations. To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision.
llama-bench, Gemma 4 31B QAT Q4_0 (llama.cpp b11056 Vulkan, -fa 1). Server benchmark, 2k-token prompts, greedy, single stream:
Full test sets, same settings, 31B:
Run it: curl -fsSL https://embedl.com/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" embedl login embedl serve google/gemma-4-31B-it/ryzen-ai-max-395 -d embedl infer google/gemma-4-31B-it/ryzen-ai-max-395 --prompt "Explain mixture-of-experts in one sentence." embedl bench google/gemma-4-31B-it/ryzen-ai-max-395
12B, E4B and E2B are up too (2.80x / 1.58x / 1.39x end to end, in the chart). Swap the name, e.g. google/gemma-4-12B-it/ryzen-ai-max-395.
Model, methodology, and more: https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395 submitted by /u/No-Dragonfly6246
[link] [comments] |
#Embedl#Gemma 4 31B#llama.cpp#ROCm#AMD Ryzen AI Max+ 395#AMD Strix Halo