RDMA

Headlines · 3

1. Nov 3
   Modeling collective communication for inference

   A report on inference infrastructure models collective communication latency as synchronization steps, startup latency and bytes transferred per card, then relates the results to NVLink, RDMA, NUMA and PCIe topology. It compares ReduceScatter, AllGather, AllReduce, AllToAll and other primitives, and presents model estimates plus reported H20 measurements for intra- and inter-machine communication. The H20 topology references a public SGLang machine matrix; the supplied report does not describe independent replication of its measurements.

   掘金 · 人工智能 · 🔥 0
2. Oct 25
   Cerebras Details Multi-Wafer Inference for Trillion-Parameter Models

   Cerebras has described how it maps the open-weight Kimi K3 model across 59 WSE-3T wafers, using 4-bit weights, an 8-bit KV cache, and a pipelined runtime. The company says activations move between wafers over RDMA Ethernet while multiple requests and micro-batches occupy different pipeline stages. Cerebras claims the multi-wafer system can run multi-trillion-parameter models at 1,000–2,000+ tokens per second, but no independent measurement was supplied.

   Hacker News 首页（100+ 分） · 🔥 0
3. Oct 23
   Lenovo touts pooled memory plus near-memory computing to ease KV cache strain

   TMTpost reports that a Lenovo team led by chief scientist Xiao Limin has built a compute-storage fusion pooled system combining memory pooling with near-memory computing to address two inference pain points: insufficient GPU and CPU-attached memory for KV cache, and high-cost GPU access to external storage. The team places the system at a "G2.5" stage and describes vendor self-tests. Versus NVMe-tier pooling, memory-pool offloading cut first-token latency by 3.5x at fixed input length and 2.4x in multi-turn dialogue. Cross-instance KV reuse accelerated first-token latency by up to 12–28x versus full recomputation and 2–3x versus an RDMA distributed memory pool. Offloading attention computation to the memory-pool side cut decode latency by 2.5x, and end-to-end inference latency was 2–3x faster than the RDMA pool, with system throughput 6–20x higher as context lengthened. Lenovo plans to move from G2.5 toward G1.5 using scale-up interconnects in supernodes. Xiao said he has not seen a product that brings near-memory computing into a large-capacity memory pool to address both KV cache capacity and access overhead; no independent third-party verification was reported.

   钛媒体 · 🔥 0

Experience and discussion from the community

Share my RDMA experienceAsk about RDMA