Run Nemotron-3.5 Lightning on DGX Spark with TensorRT Edge-LLM
An NVIDIA Technical Blog tutorial shows how to run the text-only Nemotron-3.5 Lightning model on a 64GB DGX Spark using TensorRT Edge-LLM. The trtedge command-line interface downloads the model and builds inference engines, while the workflow covers terminal chat testing and serving through a local OpenAI-compatible API. The tutorial also demonstrates optional DSpark speculative decoding, which uses a draft model to propose tokens for the main model to verify. NVIDIA reports higher mean generation speeds with DSpark on its SpeedBench 1K, SpeedBench 2K and HumanEval prompt workloads. The measurements used DGX OS 7.6.0, NVFP4, batch size one, one concurrent streaming request and a 128-token output limit.
The tutorial provides embedded developers with a concrete workflow for deploying, testing and serving a text model on DGX Spark, including a configurable speculative-decoding option. It also provides NVIDIA’s reported generation-speed and memory-use measurements for evaluating that workflow under specified test conditions.
AI is expanding beyond the data center as developers build a new generation of agents to act at the edge: monitoring driver attention and cabin conditions, helping robots understand their surroundings and plan tasks, and coordinating fleets of machines on production lines. These physical AI applications rely on models that perceive, reason, and act within tight memory, power, and latency budgets. But getting a model ready to run can involve substantial setup.
NVIDIA DGX Spark, now available in a 64GB configuration, gives embedded developers a platform to prototype, fine-tune, and test models and agents. NVIDIA TensorRT Edge-LLM provides a lightweight C++ inference runtime designed for embedded workloads, bringing the same inference framework to development on Spark and deployment on platforms such as NVIDIA Jetson Thor, NVIDIA DRIVE AGX Thor, and NVIDIA IGX Thor.
You can get started in just a few commands with the trtedge command-line interface (CLI). It downloads your selected model and builds TensorRT engines optimized for inference on Spark.
In this tutorial, you’ll run NVIDIA Nemotron-3.5 Lightning, a text-only model, on Spark using TensorRT Edge-LLM. You’ll test prompts in your terminal and serve the model through an API for application prototyping. You’ll also enable DSpark speculative decoding to accelerate generation.
#NVIDIA#TensorRT Edge-LLM#DSpark#DGX Spark 64GB#Nemotron-3.5 Lightning#trtedge#TensorRT#Jetson Thor#DRIVE AGX Thor#IGX Thor