Developer proposes mixed CUDA and ROCm inference for Project MAYA
A Reddit developer said they developed a Project MAYA fork and pull request to run GLM-5.3-Flash Maya-M GGUF prefill and decode cooperatively across an NVIDIA RTX 5090 and an AMD Strix Halo iGPU. The CUDA backend handles attention, dense or shared layers, and the expert cache on the RTX 5090, while the ROCm backend executes complementary experts on the Strix Halo during prefill and decode. The developer reported the approach could be up to 48% faster and published experiment results. Project MAYA’s creator welcomed the design and planned to test it before deciding whether to merge it.
The proposal tests a concrete way to use NVIDIA and AMD GPUs together within one inference process. Its merge remains undecided because Project MAYA’s creator plans to test the design first.
I have seen potential in combining Strix Halo's iGPU with RTX 5090 for LLM inferencing. Initially I found no inference engine that can run LLM inference on both these GPUs at the same time, inference engine such as vLLM support inference on all CUDA GPUs, or all AMD GPUs. llama.cpp Vulkan backend was the only inference engine I found that support both, but it has a huge limitation on the prefill performance on the RTX 5090.
Finally, based on the amazing inference engine Project MAYA, I developed my fork to run GLM-5.3-Flash Maya-M GGUF prefill and decode cooperatively on both GPUs. The CUDA backend (runs on a RTX 5090) working on attention, dense/shared layers and expert cache, the ROCm backend (runs on the Strix Halo iGPU) executes complementary experts in the same process, contributing during prefill and decode.
Check out the experiment results and try yourself at my GitHub PR: https://github.com/mw00/project-maya/pull/879
the creator of project-maya has welcomed the design and plans to test it before deciding on a merge. I believe we can improve it further with the community!
submitted by /u/Ren-WuJun [link] [comments]
#Project MAYA#GLM-5.3-Flash Maya-M GGUF#NVIDIA RTX 5090#AMD Strix Halo#CUDA#ROCm#llama.cpp#vLLM