Reddit summary points to AMD MI455X topping a batch-inference comparison
AI summary
A Reddit user summarized a linked article comparing inference on large data-center GPUs with Kimi K3 and three other models. The summary says larger batches raise aggregate tokens per second while reducing tokens per second per request, and that AMD's MI455X placed first, though the underlying article and benchmark details are not available here.
In-depth insight
The reported result highlights the throughput-versus-responsetime tradeoff in batched LLM inference and suggests that AMD's MI455X may perform strongly in that setting.
| Article compares performance of large data center cards. Tells us for larger batches , inference speed of individual requests goes down, but overall performance of the entire batch goes up. In other words more tokens per second for the batch, but lower TPS for each request. Article uses Kimi k3 and 3 other models to compare performance. Surprisingly says the AMD MI455x comes up on top. submitted by /u/Terminator857 [link] [comments] |