Intel Inference Microservices
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Intel launches inference microservices for Xeon processors
Intel announced Intel Inference Microservices, a containerized microservice offering for large language model inference aimed at letting enterprises get tuned inference on Xeon processors they already own without days of manual hardware optimization. Each microservice is a standalone container serving one model over an OpenAI-compatible API; at startup it detects the processor generation, assigned cores, NUMA layout and built-in accelerators such as AMX, AVX-512, AVX2 and VNNI, then applies a validated configuration profile. The first versions are based on the vLLM engine with BF16 precision, run in Docker or Kubernetes 1.24+ on 4th, 5th or 6th Gen Intel Xeon with 32GB RAM suggested. Intel says later versions will add Helm charts, a Kubernetes Operator, more engines and precisions, and that the same model will extend to Intel GPUs including the Crescent Island data-center inference GPU.
IT之家 AI · 🔥 0
Experience and discussion from the community
Share my Intel Inference Microservices experienceAsk about Intel Inference Microservices
Nobody has shared their experience with Intel Inference Microservices yet.