CI gating¶
GitHub Actions¶
- uses: sivram04/agent-reliability-lab@v1
with:
target: arl/target.yaml
suite: builtin:safety,arl/scenarios
repeats: "3"
baseline: arl/baseline.json # from `arl run ... --results-out`; empty = report only
alpha: "0.05" # significance level
max-drop: "0.0" # ignore smaller pass-rate drops
extra-args: --judge-llm gemini:gemini-3.5-flash-lite
regression is
'true' or 'false'.
How the gate decides¶
For overall and per-category pass rates, a one-sided Fisher exact test asks whether the new version fails
more often than the baseline. Latency uses a Mann-Whitney U test. A metric is a regression when p < alpha
and the drop exceeds max-drop. Use --repeats 3 or more so the tests have enough runs to decide.
Any other CI¶
arl run --target arl/target.yaml --suite builtin:safety --results-out new.json --junit-out arl.xml
arl compare arl/baseline.json new.json # exit code 1 on regression
arl.xml natively. Target secrets are replaced with
[REDACTED] in the XML.
--min-pass-rate 0.95 fails the run outright below a fixed threshold.