创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

1Cat-vLLM在多数V100测试中领先Strata

AI 总结

一项社区基准测试在四张16 GB Tesla V100上显示,1Cat-vLLM在所有受测提示长度下的单流解码速度均高于Strata;在2,654 token测试中,两者分别为124.9和74.8 token/s。1Cat-vLLM在多数预填充和并发请求测试中也领先,但Strata在约100K token处的预填充速度更好,且多token预测接受率通常更高。作者称1Cat-vLLM是专注Tesla V100的vLLM分支,其专家缓存工作为原创。

深度见解

该结果为两个推理引擎在多GPU Tesla V100上运行量化Qwen3.8-Flash-Next模型的表现提供了实用参考。

Edit: 1Cat-vLLM is a Tesla V100 focused fork of vLLM, and the expert caching work is 1Cat's original work, which may or may not one day make it to vLLM.

Everyday we see posts about speeds on Strata and its forks, but just recently I came to know that 1Cat has added support for expert caching recently, like a month ago. So I did (asked claude) a comparison of speeds on a Qwen3.8-Flash-Next UD-IQ3_XXS quant. My rig is 4x 16gb Tesla V100 on PCIe 3 x16 links, with 8 channel 3200 mt/s ddr4 ram, Epyc 7543 CPU. Now, for numbers:

Single stream

Median of 3 runs, except the 97k tokens one which I asked to run only once to save time.

Prompt (tokens)Prefill tok/s, StrataPrefill tok/s, 1CatDecode tok/s, StrataDecode tok/s, 1CatMTP acceptance, Strata / 1Cat
2,6548611,85074.8124.90.61 / 0.45
8,6741,1462,48379.3140.80.67 / 0.66
30,2992,4802,77676.4139.30.63 / 0.58
96,5883,3842,64977.2142.60.72 / 0.62

#1Cat-vLLM#Strata#vLLM#Qwen3.8-Flash-Next#Tesla V100#NVIDIA#EPYC 7543

原文约 3 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。