TensorRT Edge-LLM

Headlines · 2

1. 1d
   Run Nemotron-3.5 Lightning on DGX Spark with TensorRT Edge-LLM

   An NVIDIA Technical Blog tutorial shows how to run the text-only Nemotron-3.5 Lightning model on a 64GB DGX Spark using TensorRT Edge-LLM. The trtedge command-line interface downloads the model and builds inference engines, while the workflow covers terminal chat testing and serving through a local OpenAI-compatible API. The tutorial also demonstrates optional DSpark speculative decoding, which uses a draft model to propose tokens for the main model to verify. NVIDIA reports higher mean generation speeds with DSpark on its SpeedBench 1K, SpeedBench 2K and HumanEval prompt workloads. The measurements used DGX OS 7.6.0, NVFP4, batch size one, one concurrent streaming request and a 128-token output limit.

   NVIDIA Technical Blog · 🔥 5
2. 5d
   Embedl reports Qwen3.8-27B deployment at 40 tok/s on Jetson Thor

   Embedl reported a Qwen3.8-27B deployment on NVIDIA Jetson Thor using INT4 weights, an MTP head for speculative decoding, FlashHead, and TensorRT Edge-LLM 0.10.1. Under its stated batch-one test setup, the company reported 39.6 tokens per second for decoding, 23.6 end to end, 926 tokens per second for prompt processing, and peak memory of 24.8 GiB. The r/LocalLLaMA post also reports near-FP16 benchmark results, but the deployment and performance figures are vendor claims without independent confirmation.

   r/LocalLLaMA · 🔥 0

Experience and discussion from the community

Share my TensorRT Edge-LLM experienceAsk about TensorRT Edge-LLM