Notifications and Actions

When a user sets up a monitor, they have the option to select specific actions to be taken upon anomaly detection. These actions may include notifications which can be via Slack, Pager, or email. Users can also submit a support request for additional notification channels or custom actions such as custom webhooks or pipeline triggers.

Actions

In the case of notifications, users can enable one or more of these at the monitor level. Users can also include a custom message and assign a severity level to the notification to help with prioritization.

Before these notification channels are available for selection in the monitor manager, users must first configure the notifications in the Notifications and Digest Settings page in the settings section.

WhyLabs Notifications

Email Notifications#

To set up email notifications, we recommend creating a mailing list dedicated to WhyLabs Notifications, so that multiple team members can subscribe to it. Please work with your email administrator to configure permissions for the mailing list to receive external messages.

Email notifications will include models with alerts along with alert types and additional info configured within the monitor.

email notification

Slack Notifications#

To set up Slack notifications, we recommend creating a dedicated Slack channel for WhyLabs Notifications. Make sure all relevant team members are subscribed to the channel.

To obtain the Slack webhook, please follow Slack documentation: https://api.slack.com/messaging/webhooks.

Slack based notifications will look like the following.

slack notification

PagerDuty Notifications#

To set up PagerDuty Notifications, please follow the steps below:

In PagerDuty, create a Service to receive WhyLabs Alerts
Once created, go to Integrations tab for this Service
Click Add New Integration
Search for "WhyLabs" and select the WhyLabs AI Observability integration.

WhyLabs AI Observability integration

Copy the integration key, you’ll need it for the next step

WhyLabs integration key

Go to notification settings in WhyLabs
Enable PagerDuty Notifications
Paste your integration key and set a schedule for your Notifications

WhyLabs PagerDuty input field

Notification Content#

Notifications will include a breakdown of Alerts detected since the last Notification was sent. For example, the PagerDuty Notification may look like this:

{
  "contexts": [],
  "description": "Digest: Your monitor \"frequent_items_drift-monitor\" on dataset \"lending_club_credit_model\" detected 2 low severity issues",
  "event_type": "trigger",
  "incident_key": "DIGEST-model-0-frequent_items_drift-monitor-3cd59859-8607-469f-9f2a-62e3a3ad8c6b",
  "service_key": "xxxxxxxx",
  "details": {
    "datasetId": "model-0",
    "datasetName": "lending_club_credit_model",
    "id": "25466dad-8801-47bd-ba10-38e56056ea2a",
    "mode": "DIGEST",
    "most_recent_anomalies": [
      {
        "algorithm": "hellinger",
        "algorithm_mode": "frequent_items",
        "analysis_id": "35a95372-48c2-3ccf-bbd3-b4bfe0e89b4a",
        "analyzer_id": "frequent_items_drift-monitor-analyzer",
        "creation_timestamp": 1660350637716,
        "dataset_id": "model-0",
        "dataset_timestamp": 1660262400000,
        "feature": "addr_state",
        "granularity": "DAYS",
        "id": "4af75ba3-b13b-3107-9edf-308ec6194b3a",
        "metric": "frequent_items",
        "run_id": "3cd59859-8607-469f-9f2a-62e3a3ad8c6b",
        "segment": "purpose=car&verification_status=Not Verified",
        "target_level": "column",
        "type": "drift",
        "url": "https://hub.whylabsapp.com/models/model-0/segments/key=purpose&value=car&key=verification_status&value=Not Verified/features/addr_state?dateRange=2022-07-29-to-2022-08-12&targetOrgId=org-0",
        "weight": 1
      },
      {
        "algorithm": "hellinger",
        "algorithm_mode": "frequent_items",
        "analysis_id": "768aaacb-60c3-3e92-9207-de4b672f0ce0",
        "analyzer_id": "frequent_items_drift-monitor-analyzer",
        "creation_timestamp": 1660350637475,
        "dataset_id": "model-0",
        "dataset_timestamp": 1660262400000,
        "feature": "earliest_cr_line",
        "granularity": "DAYS",
        "id": "81b7eb3b-8bd7-3edc-91a9-0391b74a3f82",
        "metric": "frequent_items",
        "run_id": "3cd59859-8607-469f-9f2a-62e3a3ad8c6b",
        "segment": "purpose=car&verification_status=Verified",
        "target_level": "column",
        "type": "drift",
        "url": "https://hub.whylabsapp.com/models/model-0/segments/key=purpose&value=car&key=verification_status&value=Verified/features/earliest_cr_line?dateRange=2022-07-29-to-2022-08-12&targetOrgId=org-0",
        "weight": 1
      }
    ],
    "org_id": "org-0",
    "schema_version": "3.0",
    "severity": 3,
    "time_range": "2022-08-12T00:00:00.000Z/2022-08-12T00:00:00.000Z",
    "total_alerts": 2
  }
}

If the base Notification template does not work for your use case, and your team would like to receive customized Notifications instead, please reach out to us.

Configuring Multiple Delivery Methods#

In some cases, users may wish to set up multiple delivery methods. For example, users may want specific projects to deliver notifications to different locations than other projects. This level of customization is possible through the API by following steps outlined in this page.