Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

DeepGEMM

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 9

  1. MoE Inference Optimization: Batching, DeepEP, DeepGEMM and Load Balancing

    The sixth installment in an inference-infrastructure methodology series combines MoE batch-size theory with end-to-end measurements on eight H20-96 GPUs. It examines expert coverage, all-to-all communication, tile padding and expert-load imbalance, then analyzes how DeepEP, DeepGEMM and EPLB address distinct bottlenecks. The report cautions that its bench_one_batch_server sampling is not equivalent to online-service P99 latency and says an older throughput model should no longer be used for capacity planning. In the sampled runs, batch size 64 met the 30 ms TPOT limit at 2,828 tokens per second, while batch size 128 recorded 30.9 ms; the largest compliant batch size was not identified.

    掘金 · 人工智能 · 🔥 0
  2. Trade-offs in Distributed Inference for Attention and MoE

    The report analyzes distributed-inference strategies across tensor, data, expert, context, and pipeline parallelism using weight, KV-cache, activation, communication, capacity, and latency budgets. Its author argues that GQA can shard KV heads up to the KV-head count, while MLA's shared latent cache must be replicated under tensor parallelism, and compares MoE tensor parallelism with expert parallelism in terms of communication, memory use, load balancing, and implementation complexity. The report title labels it installment 2, while the body calls it the third of eight installments in an inference-infrastructure methodology series. Its comparative figures and conclusions are the author's analysis and are not independently corroborated by the supplied evidence.

    掘金 · 人工智能 · 🔥 0
  3. SGLang Reports Matching or Beating DeepInfer

    DeepSeek open-sourced DeepInfer, the inference engine behind its own service, along with related kernels. In project-run tests using DeepInfer’s benchmarks and the same hardware, SGLang reached up to 1.04x prefill throughput on four GB300 or four B300 GPUs and cut decode step time by as much as 7.4% on 64 GB300 GPUs, according to the SGLang report. A 36氪 report said vLLM also approached or partly exceeded DeepInfer under its disclosed configurations. The comparisons used project-reported results and differing workloads and measurement methods.

    LMSYS Blog · 🔥 0
  4. Prime Intellect extends Mega MoE for distributed training

    Prime Intellect says it extended DeepSeek's forward-only Mega MoE kernel with an overlapping backward pass, making the fused MoE implementation usable for distributed training. The vendor reports that its Prime Mega MoE kernels run up to 3.11 times faster than the default PyTorch implementation, while end-to-end training reaches 1.34 times the Prime-RL baseline throughput. Benchmarks used H200 GPUs with Qwen3-30B-A3B and Qwen3-235B-A22B in FP8 configurations.

    Prime Intellect Blog · 🔥 0
  5. Ant Group and vLLM open-source Humming, a quantized-inference kernel library

    Ant Group and the vLLM community have open-sourced Humming, a general-purpose kernel library for quantized inference, hosted at github.com/vllm-project/humming. The project says it supports NVIDIA GPU architectures from Turing onward plus CUDA-compatible devices such as T-Head's Zhenwu 810E, and uses a unified quantization representation covering integer and floating-point weight/activation formats, bit widths and scaling schemes for dense and MoE workloads. The authors report that Humming matches or beats vLLM Marlin, FlashInfer and DeepGEMM on many workloads in their own GEMM microbenchmarks, and say it exploits two undocumented Blackwell Tensor Core formats, float4e0m3 and float8e3m4, that are not exposed in the PTX type system.

    vLLM Blog · 🔥 0
  6. DeepSeek open-sources DeepInfer inference framework

    DeepSeek has open-sourced DeepInfer, the inference framework used for large-scale DeepSeek V4.1 Flash services, along with the deepseek-recipe token-conversion component. The supplied reports describe support for distributed inference, speculative decoding, structured output, NVIDIA SM100 Blackwell GPUs and Huawei Ascend 950 NPUs. The public release does not include all infrastructure required for online services, including KVCache management and load balancing. DeepSeek says one cluster serves 155 trillion tokens per day and generates $38 per GPU-hour at off-peak pricing; the supplied reports provide no independent measurement of those figures.

    InfoQ 中文 AI&大模型 · 🔥 0
  7. DeepSeek and Huawei Reportedly Co-Designing Future AI Chips

    DeepSeek and Huawei are reportedly working together on future AI chip designs, with DeepSeek influencing decisions about computation and data movement between processors. Huawei engineers have reportedly spent extended periods at DeepSeek's offices to incorporate the model maker's requirements. Separate reports also say DeepSeek plans to pretrain V5 on Huawei Ascend 950DT systems, but the schedule and technical details remain unconfirmed.

    The Information · 🔥 0
  8. DeepSeek open-sources Ascend ports of DeepGEMM and DeepEP kernels

    DeepSeek has published two new repositories, DeepGEMM-Ascend for matrix multiplication and DeepEP-Ascend for the all-to-all communication used by mixture-of-experts models, alongside Ascend-targeted updates to TileLang, TileKernels, DeepSelect and FlashMLA. The DeepGEMM changelog dates the Ascend release to 2026.09.30; DeepGEMM itself is a tensor-core kernel library covering FP8, FP4 and BF16 GEMMs, fused MoE and MQA scoring, with all kernels compiled at runtime through DeepJIT. SemiAnalysis reported that at the time of writing only TileLang was usable, because DeepSeek requires CANN 9.2.0 and a C++20 compiler that the beta toolkit did not provide, while TechRadar said DeepGEMM-Ascend needs Ascend 950 series silicon and CANN 9.20. The README's claim that performance matches or exceeds expert-tuned libraries across matrix shapes is the maintainers' own and has not been independently measured.

    GitHub Trending(每日) · 🔥 0
  9. vLLM v0.31.0 expands serving, speculation, and model support

    vLLM v0.31.0 adds broad serving and model-support updates, including DeepSeek-V4.1-Flash optimizations, Model Runner V2 speculative decoding, larger-scale expert parallelism, and improved multimodal support. It also introduces the `vllm preload` weight-cache daemon for faster engine restarts and experimental CRIU-based initialized-engine snapshots, alongside several security and breaking changes.

    vLLM Releases · 🔥 0

Experience and discussion from the community

Share my DeepGEMM experienceAsk about DeepGEMM

Nobody has shared their experience with DeepGEMM yet.