Embedl称Qwen3.8-27B在Jetson Thor达每秒40词元
AI 总结
Embedl 报告了 Qwen3.8-27B 在 NVIDIA Jetson Thor 上的部署方案，采用 INT4 权重、用于推测解码的 MTP 头、FlashHead，以及 TensorRT Edge-LLM 0.10.1。在其公布的批大小为 1 测试设置下，公司称解码速度为每秒 39.6 个词元，端到端速度为每秒 23.6 个词元，提示处理速度为每秒 926 个词元，峰值内存为 24.8 GiB。该 r/LocalLLaMA 帖子还列出了接近 FP16 的基准成绩，但部署效果和性能数据均为厂商自报，尚无独立验证。
深度见解
报告给出了一套面向 Jetson Thor 的 27B 参数模型边缘推理配置和具体测试参数，但其宣称的速度与准确率仍待独立验证。
| I'm on the team at Embedl that built this, and wanted to share the numbers. What's in it Speed (batch 1, 2,048-token prompts from real OpenAssistant conversations, 128 out, greedy, average of 20 requests) Accuracy vs the FP16 model (full test sets, the same engine as served) For comparison: Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M + MTP. Prompt length isn't stated there, and they don't publish accuracy. To run it The container download is about 1 GB, and the model weights are a separate download (~20 GiB). The full card has power, energy per token and methodology: https://embedl.com/Qwen/Qwen3.8-27B, and the ability to test the model on a Thor directly in the browser Let us know what model (and hardware) you want to see next! submitted by /u/No-Dragonfly6246 [link] [comments] |
#Embedl#Qwen3.8-27B#NVIDIA Jetson Thor#TensorRT Edge-LLM#FlashHead#MTP#AWQ#OpenAssistant