Alerts and notifications are a fundamental aspect of observability. Within the WhyLabs Platform, alerts and notifications can be divided into the following groups:
- Data health alerts
- Model health alerts
- Ingestion alerts
All alerts and notifications can be configured to integrate with the customer's preferred workflow. Check out Notification Workflows documentation for more information.
Once the user begins sending whylogs profiles, the WhyLabs Platform starts generating data health alerts. Data health alerts identify:
- Data distribution drift for numeric and categorical data
- Data quality problems, such as missing values and changes in cardinality
- Changes in single point metrics, such as count, median, etc
- Schema changes
- Training-serving skew
When the user configures profiling of model output and/or model performance, the WhyLabs Platform begins generating output alerts and model performance alerts. Model health alerts identify:
- Model output health, such as output distribution, cardinality (uniqueness), and schema
- Model metrics such as accuracy, precision, and recall and other KPIs such as conversion and churn (this is use case specific and is fully defined by the user)
WhyLabs supports custom metrics for both performance and KPIs.
The WhyLabs platform monitors data ingestion by default, ensuring that profiles and metrics are ingested continuously and are not empty. The following alerts are configured by default:
- Empty profiles: if a profile that was ingested is empty or is corrupted (unreadable), the user will receive a notification. An alert will be generated for each empty profile, until the user disables monitoring on the model.
- No profiles: if profiles have not been uploaded on a regular cadence, the user will receive a notification after 2 batches of data are missing. Cadence depends on whether the model is configured for hourly or daily ingestion. An alert will be generated for each missing profile, until the user disables monitoring on the model.
In addition to the dedicated ingestion alerts, when an empty or corrupted profile has been ingested, it may trigger other monitors. For example, distribution and count monitors will begin producing alerts due to corrupted profiles causing a change in these metrics.