Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Nvidia Dynamo

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 20

  1. AWS adds managed Dynamo and global routing to SageMaker Inference

    AWS Machine Learning Blog says Amazon SageMaker Inference added managed NVIDIA Dynamo, expanded HyperPod routing and multi-Region networking, and introduced Inference Gateway and Global Inference Gateway services. The latter provides one endpoint across clusters and Regions with failover in under two minutes, while managed Dynamo supports vLLM, SGLang, and TensorRT-LLM where HyperPod is available. AWS also said inference recommendations are available through SageMaker AI Studio, the SageMaker Python SDK, and coding agents.

    AWS Machine Learning Blog · 🔥 3
  2. Thinking Machines Lab releases Inkling 2 Mini open-weight model

    Thinking Machines Lab released Inkling 2 Mini, a 26.4B-parameter open-weight multimodal MoE model designed for local agentic workloads. Its 24 decoder layers include 12 middle layers that run three times per token, while the checkpoint has 3.4B active parameters and a reported 1M-token context window. Artificial Analysis independently placed it at 20 on its Intelligence Index, while Thinking Machines Lab reported leading results among open-weight models below 100B total parameters.

    Hacker News 首页(100+ 分) · 🔥 49
  3. NVIDIA Dynamo targets reinforcement-learning rollout bottlenecks

    NVIDIA Technical Blog describes Dynamo as a serving layer between reinforcement-learning orchestration and inference workers, with KV-aware routing, ModelExpress weight updates and worker-failure recovery. It says Microsoft AI, CoreWeave, Cognition, Prime Intellect and H Company use components of the stack, while CoreWeave reports up to 15× faster weight updates and H Company reports up to 20% lower training-step time and 5% lower GPU-hour use; these performance figures are vendor claims without independent benchmarks in the supplied report.

    NVIDIA Technical Blog · 🔥 0
  4. Microsoft and NVIDIA Expand AI Infrastructure Across Azure and PCs

    Microsoft and NVIDIA announced an expansion of their accelerated-computing collaboration spanning Azure, Microsoft Foundry, Azure Local, and Windows PCs. Vera Rubin NVL72 systems are running in Azure data centers, while Azure virtual machines powered by them are being made more widely available. Microsoft Foundry is expanding support for NVIDIA open models and customization, including Nemotron 3.5 models and custom model weights. Azure Local will support RTX PRO 6000 Blackwell GPUs and Jetson Thor and Jetson Orin platforms for on-premises and edge AI. Direct Stream for GPU-accelerated Windows 365 Cloud PCs supports up to two 4K streams at 60 frames per second; a limited preview is open for sign-ups, with public preview planned for early 2027. The companies also cited MLPerf results, including Signal65’s assessment of Azure’s GB300 NVL72 submissions, and an EY evaluation of Nemotron customization for corporate purchasing-record classification.

    NVIDIA Blog · 🔥 10
  5. Build Rust Tool-Calling Parsers with AI Agents

    NVIDIA Technical Blog published a tutorial on using an AI agent to build a Rust tool-calling parser for a new model, with a human-reviewed GOLDEN test set and conformance checks. The workflow uses frontend-crates to separate reasoning, visible text, and tool calls from token IDs or decoded text, then adapts parser events into OpenAI-compatible batch JSON and streaming responses.

    NVIDIA Technical Blog · 🔥 0
  6. SageMaker HyperPod Inference Adds Managed NVIDIA Dynamo Support

    AWS Machine Learning Blog says Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for multi-node model serving, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs Dynamo components and provides endpoint configuration, GPU scheduling, node recovery, and autoscaling, with deployment examples using GLM-5.3 on vLLM.

    AWS Machine Learning Blog · 🔥 0
  7. Opus 5.5 matches human baselines in CMU AI infrastructure benchmark

    新智元 reports that a Carnegie Mellon University benchmark evaluated 11 agents on six end-to-end AI infrastructure tasks spanning pretraining, intermediate and post-training, inference, agent runtimes, and cluster orchestration. Opus 5.5 paired with Claude Code reached or exceeded the human baseline on all six tasks, with the reported systems matching or outperforming production tools such as SGLang and NVIDIA Dynamo on the same GPUs. The source also says Opus 5.5 scored 1.25 on the Singularity-Index, while GPT-6.1 Sol scored 0.969 and the other agents averaged 0.58.

    虎嗅 AI · 🔥 3
  8. NVIDIA AICR Adds Qualified AIBOM Support for Kubernetes AI Workloads

    NVIDIA says its open-source AI Cluster Runtime (AICR) now includes a qualified AI bill of materials component that integrates Google's k8s-aibom to inventory recognized AI workloads on Kubernetes. The integration supports version-locked deployment bundles, provenance and security-default checks, and per-namespace opt-in inventory, while NVIDIA reports low controller overhead in a GKE test using k8s-aibom v1.6.1. Runtime observations are not signed by default, and linking them with verified provenance and deployment intent remains unfinished work.

    NVIDIA Technical Blog · 🔥 0
  9. NVIDIA Dynamo Recipes Target Four Korean Open Models

    NVIDIA is releasing Dynamo-optimized deployment recipes for A.X K2 from SK Telecom, K-EXAONE 2.0 from LG AI Research, Solar Open 2 from Upstage, and Motif-3 from Motif Technologies. The recipes target NVIDIA B200 GPUs and use Dynamo, an open-source distributed inference-serving framework for multi-node deployment. NVIDIA claims they can deliver up to twice the performance out of the box, but the supplied evidence does not independently verify that figure.

    NVIDIA Blog · 🔥 0
  10. ArtificialAnlys: DeepSeek V4.1 Flash agent throughput up to 1.5x on GB300 NVL72

    ArtificialAnlys reports that DeepSeek V4.1 Flash running on NVIDIA GB300 NVL72 gained up to 1.5x tokens per second per agent over the past month on its AA-AgentPerf benchmark, which tests datacenter systems running realistic agentic coding workloads. A single agent reached 848 tok/s, up from 596 (1.42x), while throughput per GPU rose roughly 1.6x at about 80 tok/s per agent. ArtificialAnlys attributes the gains to a newer vLLM version, new vLLM kernels for DeepSeek V4.1 on GB300, FlashInfer autotuning, fuller CUDA graph coverage, tuned speculative decoding, and running prefill and decode on the same GPUs in NVIDIA’s updated Dynamo + vLLM setup.

    ArtificialAnlys (X) · 🔥 0
  11. NVIDIA Links Full-Stack Codesign to Higher AI Memory Efficiency

    NVIDIA published two technical posts on November 5 describing a “full-stack codesign” approach that designs GPUs, CPUs, networking, storage and software together to increase useful output per byte of memory during AI inference. The company says HBM is the fastest, most expensive and most contested tier, where model weights and KV cache compete, while CPU memory, local SSDs and shared storage trade speed for capacity and lower cost per bit. The approach includes NVFP4 for weights and KV cache, weight distribution across up to 72 GPUs in NVL72 systems, large-capacity LPDDR in the Vera CPU, Dynamo KV-cache offloading and reuse, CMX context memory storage, and keeping RAG indexes outside HBM. Citing its own SemiAnalysis AgentX results, NVIDIA says Vera Rubin NVL72 delivers up to 30 times the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user. A separate comparison says Vera Rubin delivers up to 30 times the throughput per dollar at 125 tokens per second per user, while HBM capacity per GPU increased about 3% from Blackwell Ultra. These figures have not been independently verified in the supplied reports.

    NVIDIA Blog · 🔥 0
  12. NVIDIA AICR Ships Preview Vera Rubin Kubernetes Recipes

    NVIDIA says the AI Cluster Runtime (AICR), part of the NVIDIA DSX AI Factory Platform, now captures tuned Kubernetes configurations as reusable recipes, renders ordered Helm chart bundles, and validates running clusters so platform teams can be ready for NVIDIA Vera Rubin NVL72 compute trays on day one. The open-source NVIDIA/aicr repository publishes v1.0.0 preview recipes for the rke2 service and the vr200 accelerator, including training recipes validated across deployment, conformance, and an NCCL performance phase, plus an inference variant for Dynamo. The blog also notes Vera Rubin requires a new driver branch (R615.23 tested, R610 targeted for early support, R620 for official platform support) and swaps ConnectX-7/BlueField-3 for ConnectX-9 SuperNICs with BlueField-4 DPUs. NVIDIA also reports that validation surfaced an unmapped Rubin architecture value in go-nvlib, fixed in go-nvlib 0.12.0 and shipped in DRA driver v0.5.0.

    NVIDIA Technical Blog · 🔥 0
  13. NVIDIA AICR validates Kubernetes runtimes for Dynamo inference

    NVIDIA Technical Blog details how NVIDIA AI Cluster Runtime (AICR) packages NVIDIA Dynamo and its supporting Kubernetes infrastructure into versioned, validated recipes. The Dynamo recipe pins the dynamo-platform chart to version 1.4.2 and defines component versions, compatibility constraints, deployment checks, and conformance checks covering GPUs, scheduling, observability, and ingress. The walkthrough also compares Dynamo Frontend and Gateway API/EPP routing and explains the planned native Rust EPP path in Dynamo 1.5.

    NVIDIA Technical Blog · 🔥 0
  14. Nvidia backs Simplismart’s $14M Series B for AI inference optimization

    Nvidia joined Accel, Shastra VC and Micromax in Simplismart’s all-primary Series B, led by Dallas Venture Capital, which values the Bengaluru startup at about $100 million including the round. Simplismart says its software automates model deployment and optimizes latency, request handling and computing consumption, and that it helped InVideo cut GPU serving costs by 56%. Nvidia and Simplismart are also collaborating on integrations involving Nvidia Dynamo, while funding will support US expansion, hiring and inference software development.

    TechCrunch AI · 🔥 0
  15. NVIDIA and Lightcone AI Discuss Enterprise AI Value Through Token Economics

    In a discussion published by Lightcone AI and authorized by TMTPost, NVIDIA's Lai Junjie and Lightcone AI founder Wang Yisu discuss how enterprises can measure AI investment beyond GPU and data-center inputs. NVIDIA frames token economics around value, demand, supply and monetization. The speakers describe three layers of measurement: business outcomes such as task completion and business returns, process efficiency such as automation and rework, and infrastructure metrics such as cost per million tokens, latency and throughput. The article also cites rising daily token-call figures attributed to China's National Data Administration and an unnamed supermarket-chain case. The supermarket example is presented by the article as a 3% cost reduction, fourfold return on investment and a 35-day extension of payment terms.

    钛媒体 · 🔥 0
  16. Nvidia releases 120B Nemotron 3.5 Super VL and two Bolt edge models

    Nvidia released Nemotron 3.5 Super VL, an open-weight multimodal mixture-of-experts model with 120 billion total and 12 billion active parameters, aimed at always-on agents and combining reasoning, instruction following and tool use with text, image, document and video understanding. Weights are available on Hugging Face in BF16 and NVFP4, organizations can post-train it with Nvidia NeMo on their own data and workflows, and both vLLM and SGLang announced day-0 serving support. Nvidia also introduced the Nemotron 3.5 Bolt line: the 74B-A9B 24GB, a 74-billion-parameter, 9-billion-active model compressed with 2-bit quantization to fit 24GB of memory, is available now, while a roughly 4GB Bolt E4B is planned for October 23. Nvidia claims up to 5x the throughput of comparable open models on agentic tasks, an 80.8% CharXiv chart-reasoning score (84% with Python code execution), and that Dynamo can serve up to 8x as many concurrent users with 85% lower time to first token under specified conditions; these figures are vendor-reported and not independently verified.

    vLLM Blog · 🔥 0
  17. SGLang trends on GitHub's Python list

    SGLang is an open-source inference framework hosted by LMSYS for large language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts; the repository also includes SGLang Diffusion, its built-in image and video generation engine. It supports a wide range of GPUs, TPUs, NPUs, CPUs and Apple Silicon, including NVIDIA B200/B300 and H100, AMD Instinct, Google TPU, Huawei Ascend, and ships via Docker images or a uv install. The project offers a Cookbook, docs and learning resources such as Mini-SGLang, and long-term contributors can apply for coding-agent sponsorship covering Cursor, Claude Code or OpenAI Codex.

    GitHub Trending · Python · 🔥 0
  18. NVIDIA Dynamo adds session-aware routing and KV cache for agentic inference

    A PyTorch blog post details how NVIDIA Dynamo turns request-level serving into program-aware serving using one session-level identifier, with child sessions carrying a parent_session_id. Dynamo recognizes the session headers that Claude Code, Codex and OpenCode already emit, while other harnesses opt in via X-Dynamo-Session-ID, and plugins exist for Hermes and OpenClaw. The post also covers session-aware routing and admission control (porting a scheduler from the ThunderAgent paper), a shared KV cache indexer, the Session-Prefix Indexer and proposed KvHint interface, plus DYN_REQUEST_TRACE=1 trace capture with AISimulate offline replay and AIPerf live benchmarking.

    PyTorch Blog · 🔥 0
  19. Nebius, NVIDIA and Uber on squeezing more value from every unit of compute

    At a panel hosted by The Information's AI Agenda Live, executives from Nebius, NVIDIA and Uber discussed how they are optimizing compute as agentic workflows threaten to multiply token consumption. Nebius CRO Marc Boroditsky said the company tailors optimizations in its Token Factory inference platform to each customer's goals—speed, reliability, quality or price—using techniques such as speculative decoding, where smaller models predict likely next tokens that a larger model verifies in bulk, tuned with the customer's own traffic. NVIDIA's Dion Harris pointed to a Dynamo feature that locates already-computed data in a cluster's KV cache so teams avoid recalculating it, while Uber infrastructure VP Mattie Toia said its token costs have stabilized despite growing agentic use, helped by caching pushed down to individual sub-agents and by giving engineers visibility into their own usage; Uber also built an internal "context graph" of roughly 24 million nodes that cut query times from 20 minutes to 30 seconds.

    The Information · 🔥 0
  20. Vast pitches tiered storage to ease AI agent memory pressure

    In an interview with theCUBE at CoreWeave's Fully Connected 2026 event, Vast Data co-founder and CTO Alon Horev said agent memory differs from ordinary inference: it includes both in-session context and long-term memory that lets an agent review past interactions. Vast's approach tiers storage — GPU memory first, then CPU memory on the same machine, then persistent media holding petabytes of KV cache — with Nvidia's Dynamo software orchestrating the process. Horev said a 500,000-token session can occupy one-tenth to one-twentieth of a GPU's memory, and offloading such sessions to storage avoids repeat recalculation; enterprises also need to record and retain everything their agents do, data that can feed fine-tuning or purpose-built models.

    SiliconANGLE AI · 🔥 0

Experience and discussion from the community

Share my Nvidia Dynamo experienceAsk about Nvidia Dynamo

TensorRT-LLM: An open-source library for optimizing LLM and Visual Gen inference on NVIDIA GPUs.

What problem it solves LLM and Visual Gen inference requires specialized kernels, efficient runtime orchestration, and flexible deployment across NVIDIA GPU configurations. Developers also need a modular framework that supports customization while integrating with established inference tooling. Key features Optimized inference kernels — Custom kernels accelerate common inference operations, includ…

github.comhttps://github.com/NVIDIA/TensorRT-LLM