Ray Serve

Headlines · 2

1. Oct 12
   AWS shows multi-node inference with Ray LLM Deep Learning Containers

   An AWS Machine Learning Blog post walks through deploying one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Container, with separate prefill and decode nodes behind a single OpenAI-compatible endpoint. The container ships Ray Serve LLM, NVIDIA's NIXL library and the Elastic Fabric Adapter stack, and the example uses Qwen3.5-9B to show prefill/decode disaggregation configured entirely through YAML with no application code. The post also warns that NIXL silently falls back to a TCP-speed transport when it cannot use libfabric, so it recommends checking worker logs to confirm the backend.

   AWS Machine Learning Blog · 🔥 0
2. Oct 11
   SGLang trends on GitHub's Python list

   SGLang is an open-source inference framework hosted by LMSYS for large language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts; the repository also includes SGLang Diffusion, its built-in image and video generation engine. It supports a wide range of GPUs, TPUs, NPUs, CPUs and Apple Silicon, including NVIDIA B200/B300 and H100, AMD Instinct, Google TPU, Huawei Ascend, and ships via Docker images or a uv install. The project offers a Cookbook, docs and learning resources such as Mini-SGLang, and long-term contributors can apply for coding-agent sponsorship covering Cursor, Claude Code or OpenAI Codex.

   GitHub Trending · Python · 🔥 0

Experience and discussion from the community

Share my Ray Serve experienceAsk about Ray Serve