Embedl reports Qwen3.8-27B deployment at 40 tok/s on Jetson Thor
Embedl reported a Qwen3.8-27B deployment on NVIDIA Jetson Thor using INT4 weights, an MTP head for speculative decoding, FlashHead, and TensorRT Edge-LLM 0.10.1. Under its stated batch-one test setup, the company reported 39.6 tokens per second for decoding, 23.6 end to end, 926 tokens per second for prompt processing, and peak memory of 24.8 GiB. The r/LocalLLaMA post also reports near-FP16 benchmark results, but the deployment and performance figures are vendor claims without independent confirmation.
The report provides a detailed edge-inference configuration and reproducible test parameters for a 27B-parameter model on Jetson Thor, while its claimed speed and accuracy remain unverified.
| I'm on the team at Embedl that built this, and wanted to share the numbers. What's in it Speed (batch 1, 2,048-token prompts from real OpenAssistant conversations, 128 out, greedy, average of 20 requests) Accuracy vs the FP16 model (full test sets, the same engine as served) For comparison: Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4_K_M + MTP. Prompt length isn't stated there, and they don't publish accuracy. To run it The container download is about 1 GB, and the model weights are a separate download (~20 GiB). The full card has power, energy per token and methodology: https://embedl.com/Qwen/Qwen3.8-27B, and the ability to test the model on a Thor directly in the browser Let us know what model (and hardware) you want to see next! submitted by /u/No-Dragonfly6246 [link] [comments] |
#Embedl#Qwen3.8-27B#NVIDIA Jetson Thor#TensorRT Edge-LLM#FlashHead#MTP#AWQ#OpenAssistant