Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Reddit user reports Nemotron 3.5 Bolt E4B at 27.7 tok/s on Jetson Orin Nano Super

AI summary

A Reddit user reported running NVIDIA’s Nemotron 3.5 Bolt E4B on a Jetson Orin Nano Super at 27.7 tokens per second, compared with a stated 15 tok/s result from the official llama.cpp release. The poster also offered hosted access to the model and a local TensorRT engine with int4 quantization. The performance figures, comparison and claim that the model can program games come from a single community report without independent corroboration.

In-depth insight

The report presents a claimed edge-inference speed advantage for Nemotron 3.5 Bolt E4B on Jetson Orin Nano Super, along with access to an int4 TensorRT engine. The measurement and comparison have not been independently verified, and the supplied report does not establish whether the engine can reproduce the exact benchmark setup.

Hi! I wanted to share that we just got NVIDIA's latest Nemotron Bolt model running at 27.7 tok/s (compared to 15 tok/s on the official release with Llama CPP) on Jetson Orin Nano Super (1.8x faster)! It can program games on a model running completely on a cheap edge device! We have our own Jetson devices serving that model on our website, if you want to get a feel for how the model runs on the device here or run it locally as a TensorRT engine with our own variant of TensorRT edge LLM supporting int4 quantization: https://embedl.com/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16/orin-nano-super?tab=run submitted by /u/Any_Frame9721 [link] [comments]

#NVIDIA#Nemotron 3.5 Bolt E4B#Jetson Orin Nano Super#TensorRT#TensorRT Edge-LLM#llama.cpp

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.