开发者提出CUDA与ROCm混合推理方案
一名Reddit开发者称，其开发了Project MAYA分支和拉取请求，让GLM-5.3-Flash Maya-M GGUF在NVIDIA RTX 5090与AMD Strix Halo iGPU上协同完成预填充和解码。CUDA后端在RTX 5090上处理注意力、密集或共享层及专家缓存，ROCm后端则在Strix Halo上于预填充和解码期间执行互补专家。开发者称该方案最高可提速48%，并公布了实验结果。Project MAYA创建者对这一设计表示欢迎，并计划先行测试，再决定是否合并。
该提案提供了一种在同一推理流程中结合NVIDIA与AMD GPU的具体方案。由于Project MAYA创建者计划先测试设计方案，合并与否仍未决定。
I have seen potential in combining Strix Halo's iGPU with RTX 5090 for LLM inferencing. Initially I found no inference engine that can run LLM inference on both these GPUs at the same time, inference engine such as vLLM support inference on all CUDA GPUs, or all AMD GPUs. llama.cpp Vulkan backend was the only inference engine I found that support both, but it has a huge limitation on the prefill performance on the RTX 5090.
Finally, based on the amazing inference engine Project MAYA, I developed my fork to run GLM-5.3-Flash Maya-M GGUF prefill and decode cooperatively on both GPUs. The CUDA backend (runs on a RTX 5090) working on attention, dense/shared layers and expert cache, the ROCm backend (runs on the Strix Halo iGPU) executes complementary experts in the same process, contributing during prefill and decode.
Check out the experiment results and try yourself at my GitHub PR: https://github.com/mw00/project-maya/pull/879
the creator of project-maya has welcomed the design and plans to test it before deciding on a merge. I believe we can improve it further with the community!
submitted by /u/Ren-WuJun [link] [comments]
#Project MAYA#GLM-5.3-Flash Maya-M GGUF#NVIDIA RTX 5090#AMD Strix Halo#CUDA#ROCm#llama.cpp#vLLM