Feature Walkthrough

WhyLabs Observe provides full AI lifecycle observability for insights into your data and model health, alerting you to drift events, performance degradations, potential attacks, and model behavior changes. All model types including large language models (LLMs) are supported, as are all data types including structured, unstructured, and streaming data.

This page provides a walkthrough of the platform features available in WhyLabs Observe.

Project Dashboard

The Project Dashboard is the jumping-off point for many of the features available in the WhyLabs AI Control Center. The Project Dashboard serves as a centralized location offering observability into all of your models and datasets within a custom date range.

Navigation, date ranges, and page-level controls

On the Project Dashboard—and on all other pages—the dark header contains controls that set the overall scope of the current dashboard. Beneath the dark header is the page-level control area. These controls are specific to the current page, although some controls are shared across all pages. See below for a breakdown of the header and page-level controls:

WhyLabs header and page level controls

1. The global date range selector, located in the top right corner of the header
2. Tabs for switching between the "All Resources" and "Dashboards" views, positioned under the date range selector
3. For teams with more than one org, the org selector is located on the left of the header, under the WhyLabs logo

The page-level controls for the Project Dashboard are:

4. Filters to refine the list of resources displayed on the page based on their model type (LLM, classification, regression, ranking etc.), or dataset type (source, stream, transform, etc.)
5. A button shortcut to set up a new resource
6. An anomaly summary widget that shows the total number of anomalies detected across all resources in the organization
7. A Layout toggle button to switch between the dashboard and between card (default) view and table view

The Project Dashboard is separated into two views by tabs as follows:

1. The default "All Resources" view, and
2. The "Dashboards" tab where org-level summary dashboards are located, such as model summaries and custom dashboards

All Resources View

The Project Dashboard showing the "All Resources" view

The All Resources view contains a list of all resources in the organization, with shortcuts to the individual resource dashboards. The following information is displayed for each resource:

• Resource name: the name of the model or dataset
• Freshness: the date of the last uploaded profile to the model or dataset
• Global and resource-specific anomaly summary by day
• Resource type (model or dataset)
• Resource subtype (For example: regression, classification or large language models; source, stream or transform datasets)
• Anomalies in range: the global distribution of anomalies by type (data quality, drift, performance, etc.)
• Profile lineage for each resource

It's possible to sort the order of the resources on the Project Dashboard. To do this, use the layout toggle to access the table view, then use the column headers to apply sorting.

Overall Summary Dashboards

From the Project Dashboard click on the "Dashboards" tab located on the right side of the WhyLabs header. This tab lets you access two types of dashboard views that have org-wide scope:

1. Overall summary dashboards for models and datasets

These dashboards display aggregate metrics to summarize total resources, monitoring coverage, total inferences, data volume, total anomaly counts, and more. They provide a snapshot of operational heath for all resources in the organization, and can be useful for ops health reports at business review meetings.

There are two org-level summary dashboards in WhyLabs Observe:

• Datasets Summary
• Models Summary

Additional information on these dashboards can be found on the Dashboard Overview page.

2. Custom dashboards

Custom dashboards are managed in the "My Dashboards" tab, which is the default view when clicking on the "Dashboards" tab. These user-defined dashboards can be created and saved for specific use cases not covered by the default dashboards. They can be created by any user with the correct permissions, and are viewable by all users that have access to the organization.

Additional information and documentation on these dashboards can be found on the Custom Dashboards page.

Dataset Summary

The Datasets Summary tab contains:

• Total dataset count with a breakdown by subtype
• Monitoring coverage across all datasets
• Total anomaly count across all datasets within the time range, with a breakdown by category
• Total record count across all datasets within the time range
• A table summary of all datasets, including their anomaly distribution, volume, lineage and freshness

Model Summary

The Models Summary tab contains:

• Total model count with a breakdown by subtype
• Monitoring coverage across all models
• Total anomaly count across all models within the time range, with a breakdown by category including performance
• Total inference count across all models within the time range
• A table summary of all models, including their anomaly distribution, inference volume, and lineage

Custom Dashboards

Please refer to the detailed overview of custom dashboards on this page.

The "My dashboards" page showing a list of custom dashboards in an organization

Single Resource Dashboards

Resource Summary

When clicking into a resource from the Project Dashboard, the "Summary" tab shows users various metrics specific to that resource (model or dataset) for profiles within the selected date range.

For a Dataset, the summary cards include:

• Profile count and date range
• Monitoring coverage - which categories of monitoring are covered or not
• Integration health - whether profiles have been uploaded recently
• Columns health - summary of changes in data volume and anomaly volume
• Segments - count of segments and changes in their anomaly volume

Individual dataset summary view

For a Model, the summary cards include:

• Profile count and date range
• Monitoring coverage - which categories of monitoring are covered or not
• Integration health - whether profiles have been uploaded recently
• Input and Output health - summary of changes in data volume and anomaly volume
• Model performance - summary of model performance metrics
• Segments - count of segments and changes in their anomaly volume
• Explainability information

Individual model summary view

Profiles

The "Profiles" tab allows users to compare up to three batch profiles which belong to a specific resource. You can learn more about profiles here. From this page users can:

• Compare multiple uploaded profiles directly
• Compare distributions from multiple profiles directly for any column
• Compare statistics for specific profiles
• Compare most frequent items from two profiles

By clicking on the "Insights" button, users can see a list of observations about the selected profiles which may help uncover unexpected conditions in the data.

See Profiles for more guidance on working with profiles.

Inputs, Outputs and Columns

The "Inputs" and "Outputs" tabs for Models and the "Columns" tab for Datasets provide a view of the anomalies by column through the selected time range for various monitored metrics. These metrics include:

• Inferred data type
• Total count
• Null fraction
• Drift distance
• Estimated unique values
• Discreteness
• Data type count

From this view, users can click on an individual feature or column for a fine grained view of monitored metrics for that feature. From here, users can view:

• Drift distance (if the feature is monitored for drift)
• Distribution showing estimated quantiles for non-discrete features
• Distribution of most frequent values for discrete features
• Individual statistics for continuous features (mean, median, min, max)
• Missing value count and ratio
• Estimated unique values count and ratio
• Inferred data type

The "Inputs" and "Outputs" tabs are displayed for all models. Inputs represent the data provided to the model or other columns useful in profiling (e.g., sensitive attributes for monitoring fairness, derived statistics about the input data such as toxicity or sentiment in large language models). Outputs represent inferences or other columns generated by the model, as well as statistics derived from the outputs.

Outputs are initially set to be any column with 'output' in the column name - this can be changed using the WhyLabs entity schema API.

A single "Columns" tab is displayed for datasets other than data transforms. "Inputs" and "Outputs" tabs are shown for data transforms, with inputs representing the data to be transformed and outputs representing the transformed data.

Drift Comparison

Drift is an important early indicator of possible model performance problems. On the "Inputs" tab, users can visualize the drift of the model inputs and compare it with the specific baseline or baselines against which it is being monitored.

Segments

What is a segment?

A segment is a subgroup of your data or a prediction slice. You can segment your data based on any categorical column or feature in that data (be it for an ML model, data pipeline, data stream, or dataset). For example, you can choose to segment your data by geographies such as “zip code” or “country”, or human attributes like “gender” or “age range”.

Segments are powerful because they allow you to monitor changes in specific categories, classes, or other subsets of your data. They can detect bias or changes that may not be visible when only monitoring the dataset as a whole.

When uploading profiles with whylogs, users can define segments they wish to slice their data on. This is reflected in the segments section.

The "Segments" tab contains all of the individual segments defined by users when uploading profiles.

Users can click on one of these segments to view the details tabs (e.g. "Inputs", "Performance") filtered to data within the segment.

See Segmenting Data for more details about segments.

Monitor Manager

The Monitor Manager tab allows users to customize their monitors for a particular resource. This includes:

• Choosing a monitor type
• Targeting specific features/segments
• Setting analysis type & thresholds
• Setting a baseline
• Configuring actions

Users can also choose a monitor from a variety of presets.

See the Monitor Manager Overview for more details.

Performance

The "Performance" tab contains a summary of performance metrics. The performance metrics that are available for the current model are automatically rendered in the dashboard. Different metrics will be displayed depending on the model type (classification, regression, ranking, etc. ).

Note that this view is only available for model resources. To visualize the performance data, users must select the appropriate model subtype from the model settings page, and upload performance metrics via whylogs.

Classification

• Total output and input count
• Accuracy
• ROC
• Precision-Recall chart
• Confusion Matrix
• Recall
• FPR (false positive rate)
• Precision
• F1

Regression

• Total output and input count
• Mean Squared Error
• Mean Absolute Error
• Root Mean Squared Error

Ranking

• Average Precision @ K
• Reciprocal Rank
• NDCG @ K
• Precision @ K
• Recall @ K
• Top rank
• Gain sum @ K

See the Performance section for more information about working with performance metrics.

Segment Analysis

The "Segment Analysis" tab is enabled for resources with segmented data, and lets users analyze a wide list of column and dataset metrics across segments. The dashboard supports various debugging and root-cause workflows including:

• Discovering which segments within the data contribute negatively or positively towards model performance,
• Investigating if there's bias in either the dataset or a specific segment,
• Identifying potential data quality problems in segments

See the Segment Analysis section for more details.

Explainability

The "Explainability" tab is available for Model resources. It lets users view feature importance for a model's inputs, and compare them with other models.

See Explainability section for more details.

Explainability data can be uploaded using the Feature Weights API.

Anomalies Feed

The Anomalies Feed allows users to see a centralized feed of all anomalies for a given resource. It is located on the Monitor Manager tab. This view includes the anomaly timestamp, anomaly type, column, and anomaly description.

For more details, see the Anomalies section.

Organizations

An Organization is the highest level entity within the WhyLabs platform. An organization houses any number of WhyLogs models and contains any number of users. A model can only belong to one organization, but users can potentially be added to multiple organizations.

Upon creating a free account in WhyLabs, an organization will be created and your user will be added to that organization. Users belonging to multiple organizations can switch between organizations using the organization dropdown in the Model Dashboard

Settings

The settings section can be accessed from the hamburger button in the top left corner. From here, you can manage API tokens, models, notifications, and users. The settings section also contains a tool to assist with the process of setting up a new integration.

Access Token Management

Access to the WhyLabs API is controlled via Access Tokens. Uploading data and interacting with our platform via direct API calls requires a valid token. These tokens are managed by each organization's administrator.

Admins can create tokens and optionally set an expiration date for these tokens. Admins also have the ability to revoke existing tokens.

Resource Management

Whether you have one dataset in need of monitoring or a few hundred, WhyLabs makes it easy to add and begin monitoring new resources with just a few clicks.

Users are also able to rename resources from here or (in the case of Models) change the model type (regression, classification, unknown) by clicking "Edit Settings".

Model vs Dataset Types

In WhyLabs you can choose either a model or dataset type, there are a few primary differences between the two:

1. Models refer to columns as input or output features, datasets refer to them all as columns.
2. Models include performance metrics, performance tracing, and explainability. Datasets do not include these tabs.
3. Model monitoring includes presets for performance monitoring. Datasets do not have these.

Notifications

WhyLabs Platform allows receiving regular updates about the state of your data via one of the supported messaging integrations (e.g. Slack, email, etc). These notifications include a summary of the data quality anomalies, and allow you to keep tabs on your data health metrics without having to manually check in on them in the Platform.

See Notifications and actions for more detail on managing notifications.

User Management

You define who gets access to your organization's data on WhyLabs. The platform makes it easy to add and remove users, enabling you to have full control over which team members can observe and monitor your data and ML model health metrics.

Role-Based Access Controls (RBAC)

From the User Management page, Enterprise customers can attached permission based roles to users added to their organization. The platform supports the following user roles:

• Admin: can manage all aspect of the platform's functionality including creation of API tokens and user management
• Member: read-only access with the ability to create and manage monitors
• Viewer: read-only access

See Role-based Access Control (RBAC) for more details.

Integration examples

From the main menu, users can access Integration Examples. This page contains tools to instantly generate code for several example integrations specific to models and datasets in your organization.

Other

Send feedback / Support Center

Users can submit support requests from directly within the WhyLabs Platform.

Privacy policy

Users can access the WhyLabs Privacy Policy from directly within the WhyLabs Platform.

Documentation

Users can access the documentation you’re reading now directly from the WhyLabs Platform 🙂