Reddit user reports Nemotron 3.5 Bolt E4B at 27.7 tok/s on Jetson Orin Nano Super
A Reddit user reported running NVIDIA’s Nemotron 3.5 Bolt E4B on a Jetson Orin Nano Super at 27.7 tokens per second, compared with a stated 15 tok/s result from the official llama.cpp release. The poster also offered hosted access to the model and a local TensorRT engine with int4 quantization. The performance figures, comparison and claim that the model can program games come from a single community report without independent corroboration.
The report presents a claimed edge-inference speed advantage for Nemotron 3.5 Bolt E4B on Jetson Orin Nano Super, along with access to an int4 TensorRT engine. The measurement and comparison have not been independently verified, and the supplied report does not establish whether the engine can reproduce the exact benchmark setup.
| Hi! I wanted to share that we just got NVIDIA's latest Nemotron Bolt model running at 27.7 tok/s (compared to 15 tok/s on the official release with Llama CPP) on Jetson Orin Nano Super (1.8x faster)! It can program games on a model running completely on a cheap edge device! We have our own Jetson devices serving that model on our website, if you want to get a feel for how the model runs on the device here or run it locally as a TensorRT engine with our own variant of TensorRT edge LLM supporting int4 quantization: https://embedl.com/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16/orin-nano-super?tab=run submitted by /u/Any_Frame9721 [link] [comments] |
#NVIDIA#Nemotron 3.5 Bolt E4B#Jetson Orin Nano Super#TensorRT#TensorRT Edge-LLM#llama.cpp