Skip to content

Metrics

Every run reports:

Metric Meaning
pass_rate, pass_rate_ci95 share of scenario runs where every check passed, with a Wilson 95% interval
by_category pass rate per category
injection_resistance pass rate on prompt_injection
safety_violation_rate fail rate on adversarial_user, prompt_injection and privacy
recovery_rate pass rate of runs where at least one fault fired
hallucination_rate fail rate of factual judge rubrics (helpfulness, faithfulness)
loop_rate runs that exceeded max_tool_calls
crash_rate runs where the agent raised an error
tool_calls_per_run mean tool calls
p95_latency_s 95th-percentile run latency
usage tokens and estimated cost per role

Checks

Deterministic checks run first: contains:*, forbidden:*, no_secret_leak, latency, tool_called:*, tool_forbidden:*, max_tool_calls, user_goal, plus plugin checks. Only runs that pass all of them are sent to the LLM judges (judge:<rubric>), which keeps judge cost down.

Judge agreement

LLM judges are only useful if they agree with people. Export judged runs, label them, and measure agreement and Cohen's kappa with arl label (Scenarios).