Deploying a JumpStart Foundation Model on SageMaker

JumpStart deploys a model in three lines, which is the problem: the three lines hide an image URI, an instance type, a serializer, a payload schema and a licence acceptance, and every one of those is something you will need to know the moment the endpoint does not do what you expected. This page deploys one and then opens the box.

What JumpStart is actually doing

A JumpStart model ID is a lookup key into a catalogue of pre-baked deployment configurations. Given the ID, the SDK resolves a deep learning container image, the S3 location of the artifacts, a default instance type, environment variables the container expects, and the serialisers matching the container’s input contract. Then it calls the same CreateModel, CreateEndpointConfig and CreateEndpoint you would have called yourself.

Nothing about the resulting endpoint is special. It is billed as an ordinary real-time endpoint, it emits the ordinary CloudWatch metrics, it is invoked with InvokeEndpoint, and it can be autoscaled with an ordinary Application Auto Scaling policy. That is worth stating because JumpStart is often described as a product, which encourages people to look for JumpStart-specific answers to problems that have ordinary SageMaker answers.

Model IDs live in the pre-trained model table published with the SageMaker Python SDK, and AWS keeps a separate list of available foundation models in its developer guide. Proprietary models are not in this flow at all: AWS documents those as requiring an AWS Marketplace subscription and deployment through the ModelPackage class instead.

Which import to use

Be careful here, because the SDK moved and the documentation is half-migrated. Version 2 of the SageMaker Python SDK exposes JumpStartModel directly:

from sagemaker.jumpstart.model import JumpStartModel

my_model = JumpStartModel(model_id="meta-textgeneration-llama-2-13b")
predictor = my_model.deploy(accept_eula=True)

Version 3 routes JumpStart through ModelBuilder, and AWS’s deploy publicly available foundation models page now shows that form:

from sagemaker.serve import ModelBuilder
from sagemaker.core.jumpstart.configs import JumpStartConfig

jumpstart_config = JumpStartConfig(model_id="huggingface-text2text-flan-t5-xl")
model_builder = ModelBuilder.from_jumpstart_config(jumpstart_config=jumpstart_config)

model = model_builder.build()
endpoint = model_builder.deploy()

Accepting the EULA, and the version split

Several JumpStart models — the Llama family being the obvious one — require explicit acceptance of an end-user licence agreement, and there are two different ways to give it depending on how old your SDK is. AWS documents both on its model sources and license agreements page.

• SDK 2.198.0 and later: pass accept_eula=True to deploy(). AWS notes the value defaults to None and must be explicitly set to True.
• Earlier than 2.198.0: acceptance goes on the inference call instead, as a custom attribute: predictor.predict(payload, custom_attributes="accept_eula=true"). The predictor returns an error if you invoke with it false.
• Fine-tuning: acceptance goes on estimator.fit(accept_eula=True, ...). AWS notes that once you have fine-tuned, the resulting weights are yours and deploying them later needs no further acceptance.

The custom_attributes form has a detail worth knowing if you are building a wrapper around it. AWS documents the parameter as accepting "key1=value1;key2=value2" pairs and, where a key repeats, the inference server takes the last value. So a library that appends a default of accept_eula=false after a caller’s accept_eula=true silently reverses the caller’s decision.

None of this replaces reading the licence. AWS is explicit that models arrive under whatever terms their source assigned — Apache 2.0, BigScience RAIL, CreativeML Open RAIL++-M, vendor-specific terms — and that reviewing them is your responsibility. A flag in a deploy call is a mechanical acknowledgement, not a legal review.

Reading back what was chosen for you

This is the part worth the page. The retrieval helpers are stable across both entry points, and they answer the questions that otherwise become guesswork.

1. Find the instance type JumpStart would pick, before it picks it. A large model card can default to an instance type you do not have quota for, and discovering that during CreateEndpoint costs you the whole deploy cycle.

   from sagemaker import instance_types
   
   print(instance_types.retrieve_default(
       model_id=model_id,
       model_version=model_version,
       scope="inference",
   ))

   instance_types.retrieve() returns every supported type for that model, which is what you want when the default is unavailable in your region.
2. Find out what the container will accept and return. This is the answer to “what shape is the payload”, which is otherwise reverse-engineered from an example notebook.

   print(sagemaker.serializers.retrieve_options(
       model_id=model_id, model_version=model_version))
   print(sagemaker.deserializers.retrieve_options(
       model_id=model_id, model_version=model_version))
   print(sagemaker.content_types.retrieve_options(
       model_id=model_id, model_version=model_version))
   print(sagemaker.accept_types.retrieve_options(
       model_id=model_id, model_version=model_version))

3. Deploy, then invoke, then look at the endpoint in DescribeEndpointConfig and DescribeModel. The image URI and the environment variables JumpStart set are visible there, and they are the thing to copy if you later want to reproduce the deployment without JumpStart — which is a reasonable destination, since a JumpStart deployment is a configuration you can own.

   aws sagemaker describe-model --model-name <name> \
     --query 'PrimaryContainer.{Image:Image,Env:Environment}'

One more option worth knowing about at deploy time: endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED puts the model behind an inference component rather than owning the endpoint outright, which is how you get several models onto shared hardware with explicit resource reservations.

Network isolation and what it implies

AWS states that all JumpStart models run in network isolation mode: once the model container is created, it makes no outbound calls. This is a security property and mostly a good one, but it has two practical consequences people meet by accident.

The first is that the container cannot fetch anything at runtime. Code that expects to download a tokenizer, pull a config from a hub or call out to a licence server will fail inside the container rather than at deploy time, and the failure surfaces as a container error rather than a network one. The second is that if you deploy into a VPC, the VPC does not need internet access but does need S3 access — AWS documents the required S3 permissions as covering both your own bucket and the regional jumpstart-cache-prod-<region> bucket, and an S3 gateway endpoint that omits the second is a deployment that hangs pulling artifacts.

Once it is running, it is an ordinary endpoint with ordinary economics, which for an intermittently used model is the expensive kind — see why a SageMaker endpoint costs more than expected, and when to use SageMaker over a hosted API for the prior question of whether to host at all.