Ad-Hoc Monitoring

Overview

As users upload profiles to WhyLabs models over time, monitors are run against them on a regular cadence. When these monitors run, alerts are generated and notifications are sent out to users. In some cases, users may want to preview these alerts for recently uploaded profiles before the monitor run occurs. Ad-hoc Monitoring gives real-time visibility into potential data quality and model health issues as soon as the profiles have been ingested.

The Ad-hoc Monitoring feature is currently in Beta for all Self-Service Starter plans. If you would like to enable the feature for your Enterprise account, simply reach out to us via Slack or email.

Once enabled, users will see a “Preview now” button when viewing monitored metrics for a particular feature. Note that there are no alerts available for the most recent date since the monitor run is not scheduled to occur for another 2 hours.

When clicking “Preview now”, an ad-hoc monitor run will be executed on each of the features in the first page of features listed on the left. Users can filter inputs as desired to run the ad-hoc monitor on a more targeted set of features.

In the image above, we see alerts in orange which are generated by the ad-hoc monitor run for the 12 features that we filtered on. Users can explore these alerts across these 12 features as desired. The preview can be closed by clicking “Close preview” in the top banner.

With Ad-hoc monitoring, users can be even more proactive when it comes to staying on top of data and model health.

Limitations

As this is a beta feature, some limitations apply. Firstly, the ad-hoc monitor run generates alert previews based on the default monitor configuration. If users set any monitor setting overrides at the feature level or altered the monitor settings for a model, these won’t be reflected in the previewed alerts.

The ad-hoc monitor run is also limited to the features visible in the list of the left with a maximum of 30 features.

Looking forward

In the future, ad-hoc monitoring will be compatible with customized monitor settings. This will enable ad-hoc monitoring to become a powerful tool when fine-tuning your model monitor settings for an optimal signal to noise ratio.

Users will be able to iteratively tune their monitor settings and preview alerts to be generated from these within a selected date range to arrive at these optimal settings in one sitting rather than fine tuning them one step at a time as new profiles are uploaded.