Qwen3.5-0.8B纯汇编推理实现
AI 总结
开发者 suayptalha 发布了 Qwen3.5-0.8B 的纯汇编实现,覆盖分词、模型推理和采样,运行时不需要 libc 或 Python。在单个 AMD EPYC 7B12 线程上,该实现报告每秒生成6.51个 token,而 Transformers 为4.34个;两次贪心解码测试生成的32个 token 完全一致。汇编版本读取 BF16 权重,Transformers 参考实现则加载 FP32 权重并使用 PyTorch 回退内核。
深度见解
这项已公开代码的轻依赖运行时提供了一种在本地运行小型 Qwen 模型的可复现方案,但其 CPU 性能优势仍属于开发者自测结果,尚未获得独立验证。
| On an AMD EPYC 7B12, it generated 6.51 tokens/sec on one CPU thread. The Transformers reference reached 4.34. All 32 generated tokens matched in both runs. Code: https://github.com/suayptalha/qwen3\_5-assembly X Post: https://x.com/suayptalha/status/2119254536283906205 Tokenization, model inference and sampling all run in Assembly. No libc or Python at runtime; Python prepares the tokenizer tables at build time. Benchmark: 19 input tokens, greedy decoding, 2 runs. Assembly reads BF16 weights; Transformers loaded them as FP32 and used PyTorch fallback kernels. submitted by /u/suayptalha [link] [comments] |
#Qwen3.5-0.8B#qwen3_5-assembly#Transformers#PyTorch#AMD EPYC 7B12