Deploy decision models with llama.cpp and SemIf on Amazon SageMaker AI
Amazon SageMaker AI now supports deploying quantized generative models using the llama.cpp Deep Learning Container, with OpenAI-compatible chat completions. A new pathway using the SemIf extension enables structured decision APIs, exposing probabilities over user-declared options. This supports both CPU and GPU deployments for routing, prioritization, and policy workflows.
What changed?
Amazon Web Services added an official Deep Learning Container (DLC) for llama.cpp on SageMaker AI. This DLC serves OpenAI-compatible chat completions using quantized models through the SageMaker InvokeEndpoint API and supports deployment on both CPU and GPU instances. Additionally, a new extension called SemIf allows SageMaker endpoints to serve structured decision APIs. Instead of returning generated text, this extension exposes probabilities and logits for user-declared options using a custom adapter built via AWS CodeBuild.
Why does it matter to an everyday developer?
With the llama.cpp DLC, developers can now deploy quantized language models on managed SageMaker infrastructure, enabling cost- and latency-efficient inference on both CPUs and GPUs. Standard text generation supports the OpenAI Chat Completions schema, which makes integration with existing LLM-based applications straightforward. The new SemIf extension targets workflows where the model must select from a fixed set of options and exposes calibrated probabilities for use in routing, prioritization, or policy enforcement. This is useful for backend automation, support ticket triage, or any context where the set of possible decisions is known at request time. The solution leverages standard AWS tools and infrastructure, with documented accuracy and latency characteristics. However, using the SemIf option requires a custom DLC build.
What can the developer do now?
Developers can deploy the official llama.cpp DLC on SageMaker AI for OpenAI-compatible chat completions by packaging their model in a .gguf format and configuring their endpoint as described in the AWS blog. For structured decision workflows, developers can use the provided CodeBuild setup to build a SemIf-compatible container, deploy it to SageMaker, and use the new API to get option probabilities instead of text output. This supports both CPU and GPU instances, letting developers balance cost, speed, and model size for their application. Developers should review their use case requirements (text generation or structured decision), choose the appropriate deployment path, and validate latency and accuracy with representative workloads. Practical code samples and deployment scripts are available in the referenced AWS sample repository.
