Amazon SageMaker is a managed Machine Learning platform that enables developers and data scientists to build, train, and deploy models at scale. It provides a variety of tools and capabilities, such as pre-built ML algorithms and libraries, Jupyter notebook-based authoring and collaboration, and easy model deployment to the cloud. SageMaker also integrates with other AWS services, such as S3, to provide a seamless workflow for storing and accessing data, and with AWS IAM for security and access control.
In this section we will learn how to integrate Sagemaker's training and prediction pipelines with whylogs and uploading profiles to WhyLabs.
#Monitor data used for model training
One of the things that will be useful to integrate your existing models with WhyLabs is to capture and monitor reference profiles. Those will be useful in case you wish to make a static comparison, and understand whether the distribution you saw during training is the same as in production. For that, we will profile the same data used for training and write it to WhyLabs, using whylogs.
Make sure you have whylogs with the WhyLabs extension installed by running:
Then, run a batch profiling job to create a
whylogs reference profile from the training data and upload it to WhyLabs. This will be the profile used to compare your model's predictions and trigger alerts in case drift happens. You can run this code in any environment that has access to S3 and can run Python scripts, such as ECS, Lambda, EKS, etc.
Note that you will need to set your credentials to WhyLabs as environment variables. You can use the following code snippet to do so:
#Monitoring model predictions
On this section we present a way to log your model predictions to WhyLabs along with it's input features. Those will be useful to run monitors and see if your expectations are met when compared to a reference profile.
To log your model prediction data, you will need to modify the predict function on your entry point. You will also enable the rolling logger, which is set to write profiles to WhyLabs on a fixed cadence. On this example we will persist profiles every 15 minutes.
To do so, define
And deploy your model with WhyLabs' credentials set as environment variables, making sure that
predict.py is written on the same path as the model call.
NOTE: You must have
whylogs[whylabs]available on the inference environment. To do that, you can persist a
requirements.txtfile with the desired
whylogspackage version on the same path as your source directory.
#Online inferences: Decoupling whylogs
To monitor your production online models, another option is to decouple whylogs profiling from inferencing. Some options are listed below:
- Persist data to a messaging queue service (like AWS' SNS) and have a lambda function responsible for profiling and uploading to WhyLabs on a schedule
- Use our whylogs container application to host a dedicated profiling endpoint
- For cases where messaging queue services aren't supported, persist data to an HDFS (like S3) and have a batch pySpark profiling job running on a managed Spark service, such as AWS EMR.
#Get in touch
In this documentation page, we brought some insights on how to integrate WhyLabs with your Sagemaker models, both at training and inference time, using whylogs profiles and its built-in WhyLabs writer. If you have questions or wish to understand more on how you can use WhyLabs with your models, contact us at anytime!