DeepInfer
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 8
- DeepSeek open-sources DeepInfer inference framework
DeepSeek announced on October 20 that it was open-sourcing DeepInfer, a framework for high-throughput, low-latency online inference of its large models. Its accompanying technical report said DeepSeek V4.1 Flash served 155 trillion tokens per day from a single cluster and generated US$38 per card-hour at the valley rate. These throughput and revenue figures are vendor claims that the supplied report does not independently verify.
钛媒体 · 🔥 0 - SGLang Reports Matching or Beating DeepInfer
DeepSeek open-sourced DeepInfer, the inference engine behind its own service, along with related kernels. In project-run tests using DeepInfer’s benchmarks and the same hardware, SGLang reached up to 1.04x prefill throughput on four GB300 or four B300 GPUs and cut decode step time by as much as 7.4% on 64 GB300 GPUs, according to the SGLang report. A 36氪 report said vLLM also approached or partly exceeded DeepInfer under its disclosed configurations. The comparisons used project-reported results and differing workloads and measurement methods.
LMSYS Blog · 🔥 0 - vLLM Reports 954K Tokens per Second per GPU for DeepSeek V4.1 Flash
In a vLLM-published benchmark, DeepSeek V4.1 Flash reached 954,000 tokens per second per GPU on a single NVIDIA GB300 NVL72 rack using P/D disaggregation and Wide-EP, with P90 throughput of 196 tokens per second per user. On GB200, vLLM’s decode throughput was within 6% of DeepInfer’s published results across batch sizes 8–256. A specialized decode path achieved 826 P90 tokens per second for one user on four B300 GPUs. In an AgentX prefill test at concurrency 256, DEP2 delivered 64,400 tokens per second per GPU, compared with 53,300 for DEP4. The report said the configurations differed and did not characterize the difference as an engine-to-engine speedup; replay tests attributed a 13.6% DEP4 throughput loss to data-parallel imbalance.
vLLM Blog · 🔥 0 - DeepSeek open-sources its internal DeepInfer inference framework
DeepSeek announced on October 20 that it is fully open-sourcing DeepInfer, its internal inference framework, following earlier open releases of models, the TileLang compilation tool and inference operator libraries, according to a Geek Park report. DeepSeek says DeepInfer already powers high-throughput, low-latency large-scale serving for DeepSeek V4.1 Flash, and that inference code and future optimizations for new DeepSeek models will continue to be updated there. The report cites DeepSeek's own statements and gives no technical details, benchmarks or license terms for the release.
极客公园 · 🔥 0 - DeepSeek Announces Plan to Fully Open-Source DeepInfer
DeepSeek announced on October 20 that it plans to fully open-source its internal DeepInfer inference framework. The announcement followed the open-sourcing of several models, the TileLang compiler and an inference-operator library. The supplied report does not provide a repository URL, release-package details or confirmation that the code is already available.
IT之家 AI · 🔥 0 - DeepSeek fully open-sources its DeepInfer inference framework
DeepSeek announced that it will fully open-source DeepInfer, an internal inference framework it has used to deliver high-throughput, low-latency large-scale inference for DeepSeek V4.1 Flash. The company said inference code and subsequent optimizations for new models would continue to be updated through the framework.
钛媒体 · 🔥 0 - DeepSeek open-sources inference framework DeepInfer
A Chinese-language tech report says DeepSeek has open-sourced its inference framework DeepInfer. The report states this is not a model but the middle layer that determines compute costs and electricity bills, describing it as the final piece of the serving chain. It says that when large numbers of users call the service, the inference framework coordinates and schedules large compute clusters, and its efficiency directly determines AI service compute cost and electricity cost. The report notes that ordinary users do not feel this layer, though it operates every day, framing training as making models \"learn\" and inference as making them \"usable.\" The excerpt provides no version number, license, repository link, release date, or performance benchmarks, and no publisher is identified in the supplied text.
开源中国 · 🔥 0 - DeepSeek Open-Sources the DeepInfer Production Inference Framework
DeepSeek has open-sourced DeepInfer, the inference framework used for high-throughput, low-latency production serving of DeepSeek-V4.1-Flash. The release includes distributed inference, speculative decoding, SWA Bounded Replay, structured output and the deepseek-recipe component for converting between API protocols and model tokens. DeepSeek said the system supports NVIDIA SM100 Blackwell GPUs and Huawei Ascend 950 NPUs; older NVIDIA architectures were not listed. According to DeepSeek, one cluster processes 155 trillion tokens per day and generates $38 in revenue per GPU-hour at off-peak pricing. The release does not include all of DeepSeek’s online infrastructure, as KVCache management and cross-replica load-balancing code were not fully open because of their coupling with internal systems.
InfoQ 中文 AI&大模型 · 🔥 0
Experience and discussion from the community
Share my DeepInfer experienceAsk about DeepInfer
Nobody has shared their experience with DeepInfer yet.