Developer Runs Qwen3.5-0.8B Inference in Pure Assembly
Developer suayptalha published a pure-Assembly implementation of Qwen3.5-0.8B covering tokenization, model inference and sampling, with no libc or Python required at runtime. On one AMD EPYC 7B12 thread, it reported 6.51 tokens per second versus 4.34 for Transformers, and all 32 generated tokens matched in the two-run greedy-decoding test. The Assembly version read BF16 weights, while the Transformers reference loaded FP32 weights and used PyTorch fallback kernels.
The published runtime offers a reproducible, dependency-light way to run a small Qwen model locally, although its reported CPU advantage remains a developer benchmark rather than an independently validated result.
| On an AMD EPYC 7B12, it generated 6.51 tokens/sec on one CPU thread. The Transformers reference reached 4.34. All 32 generated tokens matched in both runs. Code: https://github.com/suayptalha/qwen3\_5-assembly X Post: https://x.com/suayptalha/status/2119254536283906205 Tokenization, model inference and sampling all run in Assembly. No libc or Python at runtime; Python prepares the tokenizer tables at build time. Benchmark: 19 input tokens, greedy decoding, 2 runs. Assembly reads BF16 weights; Transformers loaded them as FP32 and used PyTorch fallback kernels. submitted by /u/suayptalha [link] [comments] |
#Qwen3.5-0.8B#qwen3_5-assembly#Transformers#PyTorch#AMD EPYC 7B12