AWS Trainium

Headlines · 5

1. 2d
   Maia 300 goes inference-only as hyperscalers split training, inference chips

   An analysis piece republished by Huxiu (originally from the outlet 偏科声) says Microsoft's Maia 300 drops the redundant training-oriented hardware of earlier designs to focus on high-concurrency inference: long-context chat, low-latency interaction and multi-step agent workloads. It traces the arc from the November 2023 Maia 100, a combined training/inference architecture, to this January's Maia 200, which already tilted its architecture and scheduling toward inference. The article groups the move with Google's eighth-generation TPU lineup, where TPU 8i targets inference, and Amazon's separate Trainium (training) and Inferentia (inference) lines, framing it as hyperscalers sidestepping Nvidia's CUDA training ecosystem for lower-barrier, faster-payback commercial inference. It cites Microsoft's technical team as estimating a 30%–50% cut in per-token inference cost at scale, and IDC and Inspur projections that inference could exceed 70% of AI compute by 2026 — figures the article relays from vendors and research firms without independent verification.

   虎嗅 AI · 🔥 3
2. 5d
   Reactor raises $74M to run real-time world models under 40ms

   A report republished by 36Kr says startup Reactor has added an extension to its Series A from NVIDIA and Sapphire Ventures, bringing total funding to $74 million after a $59 million seed-plus-Series A led by Lightspeed Venture Partners. Reactor builds real-time multimodal inference infrastructure, claiming developers can run interactive video and live world models at 60+ FPS with under 40ms latency through a unified SDK and API, and it worked with the AWS Neuron Science team to run a real-time autoregressive diffusion video model on a single Trainium2 chip. The report says the platform is already used by Hollywood studios, ad platforms and streaming services, and is also pitched at physical AI and robotics for cloud-hosted policies and world models as closed-loop simulation.

   36氪 人工智能 · 🔥 0
3. 5d
   vLLM adds push-based KV transfer for disaggregated inference

   A vLLM blog post introduces a push-based NIXL connector for prefill-decode disaggregated inference that has the decode instance register its destination blocks in advance and starts the transfer from a background thread, moving post-prefill coordination and setup off the time-to-first-token critical path. In the authors' benchmark using Qwen3-32B on a 1P1D deployment of Amazon EC2 P5en instances connected by AWS EFA, push mode delivered up to 3x better TTFT and up to 30% better time-per-output-token than the default pull-based mode, with results depending on workload and configuration. At 8 QPS and 16k input, TTFT was about 540 ms in push mode versus about 1,744 ms in pull mode. The feature was built and validated on AWS Trainium clusters with EFA and contributed to vLLM as an accelerator-agnostic capability.

   vLLM Blog · 🔥 0
4. Oct 12
   vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads

   The vLLM blog describes bi-directional KV cache transfer for prefill-decode (P-D) disaggregated serving: on later turns of a conversation, a decode node can return previously computed KV to the prefill node so shared context is not recomputed. In multi-turn benchmarks on a 1P1D deployment using Amazon EC2 P5en instances with AWS Elastic Fabric Adapter, the feature reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8, with results depending on workload and configuration. The capability was originally built and validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic feature.

   vLLM Blog · 🔥 0
5. Oct 7
   Report: Anthropic taps Broadcom, Samsung for 2nm custom AI chip

   Anthropic is reportedly pursuing a custom AI accelerator designed by Broadcom and manufactured by Samsung's foundry on its SF2 2nm process, with an initial run said to be around 200,000 units. Rumored specs include roughly 4,000 TOPS INT8, about 1,300W TDP and 192GB of HBM3E (possibly 288GB), but Anthropic, Samsung and Broadcom have not publicly confirmed the arrangement or any of those figures. Anthropic has previously confirmed an internal chip team designing custom silicon for Claude, while saying its infrastructure will keep using AWS Trainium, Google TPU and NVIDIA GPUs, and has announced expanded Google and Broadcom partnerships for multi-gigawatt TPU capacity starting in 2027.

   cnBeta · 🔥 0

Experience and discussion from the community

Share my AWS Trainium experienceAsk about AWS Trainium