IGX Thor

Headlines · 1

1. 1d
   Run Nemotron-3.5 Lightning on DGX Spark with TensorRT Edge-LLM

   An NVIDIA Technical Blog tutorial shows how to run the text-only Nemotron-3.5 Lightning model on a 64GB DGX Spark using TensorRT Edge-LLM. The trtedge command-line interface downloads the model and builds inference engines, while the workflow covers terminal chat testing and serving through a local OpenAI-compatible API. The tutorial also demonstrates optional DSpark speculative decoding, which uses a draft model to propose tokens for the main model to verify. NVIDIA reports higher mean generation speeds with DSpark on its SpeedBench 1K, SpeedBench 2K and HumanEval prompt workloads. The measurements used DGX OS 7.6.0, NVFP4, batch size one, one concurrent streaming request and a 128-token output limit.

   NVIDIA Technical Blog · 🔥 5

Experience and discussion from the community

Share my IGX Thor experienceAsk about IGX Thor