WhyLabs Secure Policy
Overview
Whylabs Secure Policy is the central place where you define the rules and actions that apply to your LLM applications. These constraints are stored in a policy document in YAML or JSON format. The policy is centrally managed and versioned in WhyLabs, and is used in the WhyLabs Guardrails deployment to enforce the rules and actions.
Guardrail Score
A Guardrail score is a score from 1 to 100 that indicates how likely that an LLM interaction (typically prompt/response pair) might be a risky request for a certain type of behavior.
For example, a score of 1 means WhyLabs is quite certain the interaction is safe , while a score of 99 means WhyLabs is quite certain the interaction is a high risk one.
- 0: this score is applied when such metrics weren't computed
- 1-100: this is a range of score that customers can control the sensitivity of a given behavior.
Scores are available to be use in rule expressions in the policy document to customize the actions.
Rule
Rule consists of a score and a threshold range, and maps to one or multiple actions (callbacks).
Customers can create rules based on LangKit metrics and use them in the policy document to customize the actions and callbacks.
Actions
WhyLabs provies the following actions:
- Observe: WhyLabs will capture the metrics and the traces even if the interaction is not risky.
- Flag: WhyLabs will capture the metrics and the traces and flag the interaction as risky.
- Block: WhyLabs will block the interaction and also capture the metrics and the traces.
Callbacks
Callbacks are custom actions that can be triggered when a rule is met. Callbacks can be used to trigger custom actions in your application, such as sending an alert or calling a webhook.
We have the following built-in callbacks:
- Webhook: a JSON message is sent to a webhook URL.
- Amazon SQS: a message is sent to an Amazon SQS queue.
Rulesets
Rulesets are built-in sets of rules managed by WhyLabs. Each of a ruleset can have their own scores, ranging from 0-100 similar to a rule. More information on each ruleset can be found on their respective pages of this documentation:
