Skip to main content

Model monitoring with WhyLabs

It's painless to start monitoring whylogs profiles after they have been uploaded to the platform. Unlike other monitoring solutions, WhyLabs' monitor requires zero configuration out-of-the box, and can be enabled or disabled with a single click. This approach removes the friction of manually configuring monitors on an individual basis where each monitor requires a specific metric, threshold, or range to be defined.

It's possible to manually adjust monitor settings for each input feature, model output, or performance metric. This combination of model level defaults with manual overrides enables a fine grain of control over the monitor's behavior.

Monitor concepts#

Batches and profiles#

Dataset profiles are generated by the whylogs library. Profiles contain statistical summaries of some subset of a dataset, typically with an associated time.

Batches refer to profiles of data collected over some period of time. As profiles are uploaded to WhyLabs for 1:00pm for an hourly dataset, for example, they'll get merged into a single profile. That merged profile is a batch.

Discrete vs. continuous columns#

It's important to keep in mind that some algorithm features only work for discrete probability distributions. That is, distributions of values that can often be matched for equality (==). This works well for low cardinality numbers (e.g. a small set of integers) or strings. This doesn’t work well for continuous numbers, such as 0.95 and 0.96, because comparing with strict equality won't capture practical similarities. Instead, we do comparisons within some interval, such as 0.9 <= value < 1.0.

WhyLabs automatically determines if a dataset column is continuous or discrete based on a statistical test of the cardinality of the data relative to the overall count. That classification slightly changes the way some of the monitors below behave.

Reference profile#

When a monitor runs, it compares a given batch of data against a reference data profile. That reference profile can represent a single batch of data, configured by its timestamp or reference ID, or a whole time window of data profiles defined by the number of days or hours into the past you want to look.

See MonitorRequestReference in our api docs for more detail.

We have a few different ways you can reference profiles.

  • Trailing Window: A number of previous batches describes how many hours/days/weeks of data prior to the current time to use as a baseline for comparison. The granularity of the batch is controlled by the model's granularity.
  • Reference Profile Batch: Any single batch of data logged previously can be used as the baseline for comparison against the current batch. For instance, this could be the very first day of data logged for the dataset.
  • Single Reference Profile: An individual profile uploaded via the LogReference API can be used as the baseline for comparison. This profile might represent the training dataset, for example.

Config defaults and learned thresholds#

There are a few strategies that we use to determine defaults for various configurations.

  • For configuration fields that represent thresholds, we default to dynamically learning them based on analyzing windows of time relative to today. We're constantly tuning the exact approach, so reach out to us on Slack to ask about the details.
  • For some configuration parameters that aren't specified explicitly, we have defaults that will be used. When you GetMonitorConfig via our API, any defaults that we applied will be returned as part of the config.

Monitor types#

WhyLabs has several types of monitoring algorithms that can be configured to run, enumerated below.

Data type monitor#

  • Output: 0 or 1 if a column in a profile has a type that is different from the reference profile
  • Config: See DatatypeMonitorRequestConfig in our api docs.

This monitor alerts based on the inferred data type of a column compared against the inferred data type of the same column in the reference profile. If the inferred data type is different, this triggers an alarm.

For example, for some column name, if you enable this monitor on a dataset that is configured to use the previous three days, and the previous three days have inferred name to be of type TEXT, but the current batch of data inferred name to be of type NULL, then this will trigger an alarm.

The inferred data type for a column for a batch of data is determined to be the most frequent data type observed. For example, if 45% of values in a column are TEXT, 20% are FRACTIONAL, and 25% are NULL then the inferred type will be TEXT. It works the same for a window of profiles. For example, if the last 7 days of profiles have 45% of values in a column are TEXT, 20% are FRACTIONAL, and 25% are NULL then the inferred type will be TEXT.

Distribution monitor#

  • Output: Value within interval [0, 1] that represents the distance between current and previous batches of data.
  • Config: See DistributionMonitorRequestConfig in our api docs.

This monitor alerts based on the distribution distance compared with previous batches of data, as computed by the Hellinger distance. If this monitor is in alert, then it means that the distribution of the batch of data being analyzed is different enough from the reference batch that it exceeded the threshold set in the monitor configuration.

For example, if you enable this monitor on a dataset that is configured to use the previous 7 days with a threshold of 2, then we will compute the Hellinger distance for every new profile that you upload relative to the last 7 days and trigger an alarm if that number is greater than 2.

The calculation for distribution distance can differ slightly for discrete and non-discrete features. Discrete features utilize the most frequent items collected while continuous numerical values utilize the histogram.

Missing value monitor#

  • Output: Value within interval [0, 1].
  • Config: See MissingValuesMonitorRequestConfig in our api docs.

This monitor alerts based on the ratio of missing values calculated as num_null_type_values/num_total_values. If the output of the monitor is higher than the configured threshold then this monitor will trigger an alarm. For example, if the monitor is configured with a threshold of 0.01 and a batch is generated that has 11 null values out of 1000 total values in some column, then an alert will be triggered for that column.

If a dataset is configured to use a time range as a baseline reference then we use the median missing value ratio for the time range. For example, if we're using this monitor on a model that has been configured to use the past seven days as a baseline and missing value ratios across the last seven days were [0, 0, 0.1, 0.2, 0.3, 0.4, 0.6] then we would choose 0.2 as our baseline missing value ratio.

Unique value monitor#

  • Output: Value within interval [0, 1] for non-discrete features, integer value for discrete features.
  • Config: See UniqueValuesMonitorRequestConfig in our api docs.

This monitor tracks the unique values ratio and works similarly to the missing value monitor for continuous features. For discrete features, we monitor based on the total count of unique values.

For example, if a dataset is configured to use this monitor with a minimum threshold of .2 and a maximum threshold of .8 and a column in a new batch of data contains values A, B, and C with 100 total rows, then the unique value ratio is 0.03, or 3/100.

Missing profiles monitor#

  • Output: 0 or 1

When a dataset ceases to log new profiles, WhyLabs will generate an alert to indicate a potential issue with the integration. By default WhyLabs will alert after two subsequent missing batches. For example, if you have configured an hourly dataset and something breaks in your system at 8:00 such that profiles are no longer sent to WhyLabs for 9:00 and 10:00, then this will alert when our monitor runs.

Missing recent data monitor#

  • Output: 0 or 1

Every WhyLogs profile should be logged with a dataset timestamp. It's the only way that WhyLabs knows what time period the data in the profile belongs to. This monitor alerts if too many dataset timestamps are missing from profiles, even if profiles themselves are being uploaded. For example, if you have configured this monitor to look at 2 batches for an hourly dataset and something breaks in your system at 8:00 such that all dataset timestamps are showing up as 1970, then this monitor will trigger an alert shortly after at 10:00.

Seasonal timeseries monitor#

  • Output: upper and lower 95% Confidence Intervals for expected value

We estimate 95% confidence intervals for each point in a timeseries using Seasonal ARIMA (SARIMA) forecasting. Observed values that fall outside the forecast window will trigger alerts. This forecasting method requires much more baseline data than other monitors, but works well with seasonal data we have seen so far.

Configuring the monitor#

The monitor can be configured either through the WhyLabs UI or programmatically via our REST API if you want to override any of our defaults. If you're updating settings through the UI, then you'll want to head to Monitor Settings page when scoped into a model. It's accessed via the last tab in the model page, which can be seen below.

Monitor Settings in WhyLabs

Alternatively you can access the page via its URL, specifying the appropriate model ID: hub.whylabsapp.com/models/model-ID/monitor-settings.

Requests can be made via http REST calls using the Swagger api as a guide. We also have a Python and Java client that supply types and call logic.

You can supply a configuration for the monitor using our PutMonitorConfig API. It takes a JSON config as the body with the following schema.

interface ProfileReference {
type: "reference_profile";
// Pick one of the following to supply
profileTimestamp?: number; // Millisecond timestamp of the reference profile to use (if it was uploaded with the dataset)
profileId?: string; // ID of the profile to use (if it was uploaded separately)
}
interface TimeWindowReference {
type: "reference_window";
num_batches?: number; // global number of batches to use when monitoring the model/dataset
}
interface SourceMonitorConfig {
schema_version: "0.1.0";
/**
* If the monitored value falls more than the specified number of
* std dev away from mean, the value is considered an anomaly.
*
* This value is used in each of the thresholds in each of the monitor
* configs that are unset, which means they're implicitly set to be
* dynamically learned using this standard deviation.
*/
num_std_dev?: number;
/**
* A profile to reference as the baseline.
*/
reference?: TimeWindowReference | ProfileReference;
distribution?: {
enable?: boolean;
threshold?: number | null;
};
missing_values?: {
enable?: boolean;
threshold_lower_bound?: number;
threshold_upper_bound?: number;
};
unique_values?: {
enable?: boolean;
/**
* The minimum number of rows of data required before we'll consider
* alerting. This is to cut down on noise.
*/
min_record_count?: number;
min_threshold?: number | null;
max_threshold?: number | null;
};
datatype?: {
enable?: boolean;
};
missingRecentData?: {
enable?: boolean;
}
missingRecentProfiles?: {
enable?: boolean;
}
seasonalARIMA?: {
enable?: boolean;
/**
* The number of batches that comprise the seasonality.
* For a daily dataset, a seasonality of 7 would indicate that the
* data is expected to have a weekly cycle.
*/
seasonalityBatches?: number;
/**
* The list of metrics that you want to enable ARIMA for.
* See WhyLogsMetric below
*/
metrics?: WhyLogsMetric[];
};
}
/**
* Metrics that are present inside of whylogs profiles
*/
enum WhyLogsMetric {
TotalCount,
Median,
Min,
Max,
StdDev,
Mean,
}
interface SourceDatasetMonitorConfig {
config: SourceMonitorConfig;
per_feature_config?: Record<string, SourceMonitorConfig>;
}

Here is an example of what a config looks like. The sample has a global configuration that sets the baseline to the profile with an id of ref-vLlXzaNvysnuqacL and configures the distribution monitor to have a threshold of 2. It also overrides the monitor settings for the feature_a and feature_b features in this particular model. feature_a's distribution threshold is nudged a bit higher to 3 and feature_b's datatype monitor is disabled.

const myConfig: SourceDatasetMonitorConfig = {
config: {
schema_version: "0.1.0",
reference: {
type: "reference_profile",
profileId: "ref-vLlXzaNvysnuqacL",
},
distribution: {
enable: true,
threshold: 2,
},
},
per_feature_config: {
feature_a: {
schema_version: "0.1.0",
distribution: {
enable: true,
threshold: 3,
},
},
feature_b: {
schema_version: "0.1.0",
datatype: {
enable: false,
},
},
},
};

For quick prototyping, you can open up this playground to customize the myConfig object at the bottom. If you want to use the API to upload a reference profile, then take note of the profile ID that gets returned to you. You can use that as the profileId in the config reference. The playground uses TypeScript as an easy way to validate the types, but the payload will have to be converted into JSON before it can be sent to our API.

Adjusting monitor sensitivity#

Different models and use cases will require different monitoring settings for the best signal-to-noise ratio. For example, in some instances low levels of missing values within an input feature may be tolerable, while in others any reading above 0% may indicate a critical issue. While our monitors come with preconfigured defaults that should work for most teams, some monitor configuration tweaks may be necessary once patterns begin to emerge in the underlying data.

You can manually adjust how sensitive a particular monitor is by changing its thresholds. This can be accomplished both via the UI and our API. For more information, see the section about the particular monitor you are interested in modifying.

It is also possible to mark a given monitor alert as "unhelpful" through the UI:

Unhelpful Alerts

This feedback is used to evaluate our default monitor configuration settings, and, in the case of non-deterministic monitoring algorithms such as Seasonal ARIMA, is also utilized as part of the monitor retraining process to minimize the number of false positives in the monitor's output.

Prefooter Illustration Mobile
Run AI With Certainty
Get started for free
Prefooter Illustration