Notifications Overview

When setting up a monitor, you have the option to select specific actions to be taken upon anomaly detection. Notifications can be sent via various channels such as Slack, PageDuty, email, or custom webhooks. Users can also submit a support request to request custom actions such pipeline triggers.

To make notification channels are available for selection in the Monitor Manager, they must be configured from the Notification settings page:

Email Notifications

To set up email notifications, we recommend creating a mailing list dedicated to WhyLabs Notifications, so that multiple team members can subscribe to it. Please work with your email administrator to configure permissions for the mailing list to receive external messages.

Email notifications are formatted based on the configured monitor type. When an alert is triggered, the notification will include information on what is being alerted, the corresponding segments affected, and any trace or debug information configured for investigation.

Slack Notifications

To set up Slack notifications, we recommend creating a dedicated Slack channel for WhyLabs Notifications. Make sure all relevant team members are subscribed to the channel.

To obtain the Slack webhook, please follow Slack documentation: https://api.slack.com/messaging/webhooks.

Slack based notifications will look like the following.

PagerDuty Notifications

To set up PagerDuty Notifications, please follow the steps below:

1. In PagerDuty, create a Service to receive WhyLabs Alerts
2. Once created, go to Integrations tab for this Service
3. Click Add New Integration
4. Search for "WhyLabs" and select the WhyLabs AI Observability integration.

5. Copy the integration key, you’ll need it for the next step

6. Go to notification settings in WhyLabs
7. Enable PagerDuty Notifications
8. Paste your integration key and set a schedule for your Notifications

Notification Content

Notifications will include a breakdown of Alerts detected since the last Notification was sent. For example, the PagerDuty Notification may look like this:

{
  "contexts": [],
  "description": "Digest: Your monitor \"frequent_items_drift-monitor\" on dataset \"lending_club_credit_model\" detected 2 low severity issues",
  "event_type": "trigger",
  "incident_key": "DIGEST-model-0-frequent_items_drift-monitor-3cd59859-8607-469f-9f2a-62e3a3ad8c6b",
  "service_key": "xxxxxxxx",
  "details": {
    "datasetId": "model-0",
    "datasetName": "lending_club_credit_model",
    "id": "25466dad-8801-47bd-ba10-38e56056ea2a",
    "mode": "DIGEST",
    "most_recent_anomalies": [
      {
        "algorithm": "hellinger",
        "algorithm_mode": "frequent_items",
        "analysis_id": "35a95372-48c2-3ccf-bbd3-b4bfe0e89b4a",
        "analyzer_id": "frequent_items_drift-monitor-analyzer",
        "creation_timestamp": 1660350637716,
        "dataset_id": "model-0",
        "dataset_timestamp": 1660262400000,
        "feature": "addr_state",
        "granularity": "DAYS",
        "id": "4af75ba3-b13b-3107-9edf-308ec6194b3a",
        "metric": "frequent_items",
        "run_id": "3cd59859-8607-469f-9f2a-62e3a3ad8c6b",
        "segment": "purpose=car&verification_status=Not Verified",
        "target_level": "column",
        "type": "drift",
        "url": "https://hub.whylabsapp.com/models/model-0/segments/key=purpose&value=car&key=verification_status&value=Not Verified/features/addr_state?dateRange=2022-07-29-to-2022-08-12&targetOrgId=org-0",
        "weight": 1
      },
      {
        "algorithm": "hellinger",
        "algorithm_mode": "frequent_items",
        "analysis_id": "768aaacb-60c3-3e92-9207-de4b672f0ce0",
        "analyzer_id": "frequent_items_drift-monitor-analyzer",
        "creation_timestamp": 1660350637475,
        "dataset_id": "model-0",
        "dataset_timestamp": 1660262400000,
        "feature": "earliest_cr_line",
        "granularity": "DAYS",
        "id": "81b7eb3b-8bd7-3edc-91a9-0391b74a3f82",
        "metric": "frequent_items",
        "run_id": "3cd59859-8607-469f-9f2a-62e3a3ad8c6b",
        "segment": "purpose=car&verification_status=Verified",
        "target_level": "column",
        "type": "drift",
        "url": "https://hub.whylabsapp.com/models/model-0/segments/key=purpose&value=car&key=verification_status&value=Verified/features/earliest_cr_line?dateRange=2022-07-29-to-2022-08-12&targetOrgId=org-0",
        "weight": 1
      }
    ],
    "org_id": "org-0",
    "schema_version": "3.0",
    "severity": 3,
    "time_range": "2022-08-12T00:00:00.000Z/2022-08-12T00:00:00.000Z",
    "total_alerts": 2
  }
}

If the base Notification template does not work for your use case, and your team would like to receive customized Notifications instead, please reach out to us.

Configuring Multiple Delivery Methods

In some cases, users may wish to set up multiple delivery methods. For example, users may want specific projects to deliver notifications to different locations than other projects. This level of customization is possible through the API by following steps outlined in this page.