Customer Experience Ruleset

Customer Experience is designed to prevent various issues that impact the application experience by detecting sentiment, toxicity, refusals, and any PII that a user attempts to share.

The following yaml code can be added to your policy to enable the Truthfulness ruleset.

  - ruleset: score.customer_experience
    options:
      behavior: observe
      sensitivity: medium

This ruleset adds the equivalent of the following metric section to your yaml policy and uses those metrics to compute the overall guardrail scores prompt.score.customer_experience and response.score.customer_experience.

metrics:
  - metric: prompt.sentiment.sentiment_score
  - metric: prompt.pii
  - metric: response.sentiment.sentiment_score
  - metric: response.toxicity.toxicity_score
  - metric: response.regex.refusal
  - metric: prompt.pca.coordinates
  - metric: response.pca.coordinates

The *.pca.coordinates fields are included to allow visualization of the traces. They do not contribute to the guardrail score.

The prompt.pii metric comprises a set of individual PII metrics as described in the section on Secure Container Metrics

The overall guardrail score for the prompt is calculated by first normalizing the constituent metrics to be within the range from 0 to 100 and then taking the maximum value of the normalized metrics.

The following metric scores are calculated when this ruleset is enabled, in addition to the raw metrics listed above:

customer_experience_metrics = [
    "prompt.score.customer_experience",
    "prompt.score.customer_experience.prompt.pii.credit_card",
    "prompt.score.customer_experience.prompt.pii.email_address",
    "prompt.score.customer_experience.prompt.pii.phone_number",
    "prompt.score.customer_experience.prompt.pii.redacted",
    "prompt.score.customer_experience.prompt.pii.us_bank_number",
    "prompt.score.customer_experience.prompt.pii.us_ssn",
    "prompt.score.customer_experience.prompt.sentiment.sentiment_score",
    "response.score.customer_experience",
    "response.score.customer_experience.response.regex.refusal",
    "response.score.customer_experience.response.sentiment.sentiment_score",
    "response.score.customer_experience.response.toxicity.toxicity_score",
]

The general pattern with rulesets is that they include:

• the overall guardrail metric(s) for the ruleset
• the set of individual normalized metrics that contributed to the overall metric score
• the set of raw metrics that were used to compute the normalized metrics

The individual normalized metric names consist of the raw metric name that they were calculated from, prefixed with the name of the overall metric that they contributed to.