Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Embedl int4 GEMM reaches 2.43× llama.cpp on Gemma 4 31B

AI summary

Embedl’s developer reports that a custom int4-by-int4 WMMA GEMM ran Gemma 4 31B at 2.43× llama.cpp’s speed on Ryzen AI Max+ 395 hardware, with a measured instruction rate of 104 TOPS. Under identical benchmark settings, its prompt-processing results were 679 and 724 versus llama.cpp’s 287 and 276, while GSM8K, IFEval, and MMLU-Pro scores remained close. The implementation uses int4 weights and activations, rotation, GPTQ, and selective mixed precision.

In-depth insight

The reported results show that exploiting AMD Strix Halo int4 WMMA instructions could substantially accelerate local Gemma 4 inference without a clear loss in the evaluated quality benchmarks.

The 8060S has an int4 x int4 WMMA instruction at 2x the FP16 rate (we measured 104 TOPS). Nothing in ROCm or llama.cpp uses it. We wrote a GEMM for it and run Gemma 4 with int4 weights and int4 activations. To maintain accuracy we use a combination of rotation and GPTQ, along with selective mixed precision. llama-bench, Gemma 4 31B QAT Q4_0 (llama.cpp b11056 Vulkan, -fa 1). Server benchmark, 2k-token prompts, greedy, single stream: Full test sets, same settings, 31B: Run it: curl -fsSL https://embedl.com/install.sh | sh && export PATH="$HOME/.local/bin:$PATH" embedl login embedl serve google/gemma-4-31B-it/ryzen-ai-max-395 -d embedl infer google/gemma-4-31B-it/ryzen-ai-max-395 --prompt "Explain mixture-of-experts in one sentence." embedl bench google/gemma-4-31B-it/ryzen-ai-max-395 12B, E4B and E2B are up too (2.80x / 1.58x / 1.39x end to end, in the chart). Swap the name, e.g. google/gemma-4-12B-it/ryzen-ai-max-395. Model, methodology, and more: https://www.embedl.com/google/gemma-4-31B-it/ryzen-ai-max-395 submitted by /u/No-Dragonfly6246 [link] [comments]

#Embedl#Gemma 4 31B#llama.cpp#ROCm#AMD Ryzen AI Max+ 395#AMD Strix Halo

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.