Testing

Durting the execution of a template there are two different types of testing that can be done, the first are Checks, a check can be run during a flow execution on a specific flow, the second are Tests, a test can be run at the end of the flow execution to tests if the flow itself was completed successfully or not.

Both checks and testing are configured by setting an Operations, a check operation can be set only on specific seuquence context (last response message, state transition metrics) a test operation can be set only on global metrics (total/successfull/failure/timeout messages metrics).

Operations

Operations can be run during a flow execution as a check, or once a flow execution completes as a test.

There are 5 different available operations:

1. isEquals

2. isNotEquals

3. isLess

4. isGreater

5. isEmpty

6. isNotEmpty

7. isIn

8. isNotIn

9. isPercentOf

Formats

isEquals, isNotEquals, isLess, isGreater, isEmpty, isNotEmpty - operation({{parameter}}, value)

For example, if we want to check that a {{number}} is equals to 5 =>

isEquals({{number}}, 5)

isIn, isNotIn - operation({{parameter}}, ["item1", "item2"])

For example, if we want to check that {{parameter}} is in the following list "hello", "world" => isIn({{parameter}}, ["hello", "world"])

isPercentOf - isPercentOf({{parameter}}, percent-to-test, >/<, value)

For example, if we want to check if a {{number}} is greater than 40% out of 500 =>

isPercentOf({{number}}, 500, >, 40)

Checks

Checks validate conditions in your flow during its execution. Testers often use checks to validate that the system is responsding with the expected content. For example, a check could validate that the NGSetupResponse outcome is successful isEquals({{res.NGAPOutcome}},success}

Checks are similar to what many testing frameworks call an assert, but failed checks do not cause the test to abort or finish with a failed status by default, but can be configured to do so. Failed check are tracked and kept as part of the full report with exact context details.

Check are configured as follows:

checks:
    - name: NGSetupSuccess
      operations:
        - isEquals({{res.NGAPOutcome}},success)

• name - name of the check, which can be inserted in the flow templates as part of the Actions during a state transition
• operations - a List of Operations to be execute as part of the check

Available fields to check

{{res.Field}} - Check last response message parameter
{{state.duration}} - Check last state transition duration

Tests

A tests validate the end of execution of a flow. Testers often use tests to validate that the entire flow was exectued as expected. For example, a test to make sure that all flows within an execution were executed successfully without errors/timeouts.

Tests are configured as follows:

tests:
    - name: FlowSuccessfull
      operations:
        - isEquals({{total_flows}}, 10) 
        - isEquals({{total_flows_success}}, 10)
        - isEquals({{total_flows_timeout}}, 0)

• name - name of the test, which than is outputed as part of the report

• operations - a List of Operations to be executed as part of the test

Available fields to tests

{{total_messages_tx}} - total number of sent messages 
{{total_messages_rx}} - total number of received messages 
{{total_flows - total}} flows that were started 
{{total_flows_success}} - total flows that were completed successfully
{{total_flows_failed}} - total flows that completed with a failure 
{{total_flows_timeout}} - total flows that were timed out 
{{total_flows_error}} - total flows with errors
{{total_checks}} - total checks that were done
{{total_checks_fail}} - total checks that have failed 
{{total_checks_success}} - total checks that were completed successfully
{{average_flow_latency}} - average latency of a flow from start to finish