Suites

A suite runs several flows as steps, in order, and grades them together. rk run-suite needs the licence feature suites.

YAML

Shell

Steps

A key rk doesn't know, such as repetitons:, stops the suite from loading, with the file, line and key.

Key	What it does
`name`	required, unique in the suite
`flow`	required: the flow to run
`repetitions`, `rate`, `duration`	the step's workload (one start by default)
`timeout`	the step's deadline; by default `-step-timeout`, which defaults to 30s
`load`	a stepped load run instead of `repetitions`/`rate`/`duration`; the flow's own `load:` is ignored
`params`	values given to the flow
`carry_params`	pass this step's params on to the next step (one repetition, no load)
`background_flows`	server flows that answer during the step; each must be `type: server`
`stop_on_failure`	stop the suite when this step fails; `true` by default
`always_run`	run this step even after the suite stopped (cleanup)
`success_when`	grade the step by its UEs' final states: exactly one of `at_least_one_final_state`, `exactly_n_final_state {state, count}`, `all_final_state`
`checks`	checks on the step's metrics
`gen_subscriber`, `subscriber_seed`	subscribers for the step; see Subscribers
`trace`	wire tracing for the step

At the suite level, final_checks run against every step's metrics combined: counters are summed, latency histograms are merged.

How a suite runs

Steps run in order. A step fails the suite and stops it when it fails and stop_on_failure is true, or -fail-fast is given. After that, only always_run steps run. An unsupported step doesn't stop the suite.
Cycles. One run of every step is a cycle. -repetitions sets the number of cycles; -duration repeats cycles back to back until the time is up. Cycles never overlap, and -rate isn't accepted.
Grading a step:
1. a run error, or a finding that fired, fails it;
2. otherwise success_when decides, when it's set;
3. otherwise the flow's verdict decides.
Grading a cycle:
- it fails if a step failed, a final check failed, or the cycle errored;
- otherwise it's unsupported if a step was unsupported;
- otherwise it passes.

rk run-suite checks every reference before it runs: each flow exists, background flows are server flows, success_when states are final states, and check metrics are valid. rk lint -suite <file> -templates <dir> runs the same check without running anything.

Flags

-suite (required), -repetitions, -duration, -timeout (the whole run; none by default), -step-timeout, -fail-fast, -profile-dir, -profile-window, -peer-metrics, plus the flags shared with run-flow (-templates, -c, -s, -output, …).