Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Developer Runs Qwen3.5-0.8B Inference in Pure Assembly

AI summary

Developer suayptalha published a pure-Assembly implementation of Qwen3.5-0.8B covering tokenization, model inference and sampling, with no libc or Python required at runtime. On one AMD EPYC 7B12 thread, it reported 6.51 tokens per second versus 4.34 for Transformers, and all 32 generated tokens matched in the two-run greedy-decoding test. The Assembly version read BF16 weights, while the Transformers reference loaded FP32 weights and used PyTorch fallback kernels.

In-depth insight

The published runtime offers a reproducible, dependency-light way to run a small Qwen model locally, although its reported CPU advantage remains a developer benchmark rather than an independently validated result.

On an AMD EPYC 7B12, it generated 6.51 tokens/sec on one CPU thread. The Transformers reference reached 4.34. All 32 generated tokens matched in both runs. Code: https://github.com/suayptalha/qwen3\_5-assembly X Post: https://x.com/suayptalha/status/2119254536283906205 Tokenization, model inference and sampling all run in Assembly. No libc or Python at runtime; Python prepares the tokenizer tables at build time. Benchmark: 19 input tokens, greedy decoding, 2 runs. Assembly reads BF16 weights; Transformers loaded them as FP32 and used PyTorch fallback kernels. submitted by /u/suayptalha [link] [comments]

#Qwen3.5-0.8B#qwen3_5-assembly#Transformers#PyTorch#AMD EPYC 7B12

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.