SemIf

Headlines · 1

1. 4d
   AWS details llama.cpp decision-model serving on SageMaker AI with SemIf

   AWS published a walkthrough for serving decision models with llama.cpp on Amazon SageMaker AI using specialized Deep Learning Containers. The standard path deploys a GGUF model from Amazon S3 and maps InvokeEndpoint to llama-server's OpenAI-compatible chat completions; an optional SemIf child image (SemIf was formerly OpenJev) returns probabilities over caller-declared options by reading final-position logits instead of decoding an answer token. On 231 public JevBench tasks with Qwen3.5-4B Q4_K_M, a 32-thread CPU endpoint (ml.c6i.8xlarge) was correct on 179/231 (77.5%, p50 1.715s, p95 19.291s), while full GPU offload on ml.g6e.xlarge scored 180/231 (77.9%, p50 0.352s, p95 0.599s) — 4.9x faster at p50 and 32.2x at p95. AWS says the two images used different llama.cpp builds and disagreed on six hard tasks, so it calls the accuracy gap runtime variation rather than proof that GPU offload improves quality, and cites Jev, Kev and Strands Decider 2B as related decision-model work.

   AWS Machine Learning Blog · 🔥 0

Experience and discussion from the community

Share my SemIf experienceAsk about SemIf