Custom Metrics

Every ML and data pipeline use case is different! It is not unusual for AI Observatory users to supply custom metrics into the platform. Overall, there are three use cases for custom metrics:

• Data-type specific metrics: for models that are powered by complex, unstructured data
• Model outputs: for model outputs that track custom KPIs
• Model performance: for performance metrics that is not typical in the industry for a specific model type

Please reach out for help with your specific use case.

Data Type Specific Metrics

Any metrics collected by whylogs can be configured. Out of the box whylogs supports structured data, images, and text. We are currently working with design partners for audio and embedding data.

A quick tutorial on how to configure custom data type specific metrics is available here.