Metrics

rk serves Prometheus metrics while it runs, when -metrics-port is set:

Commands: run-flow, run-suite, daemon, server uspace and server dpdk.
Licence: needs the licence feature obs.prometheus.
Address: /metrics on -metrics-addr, by default 127.0.0.1 (this machine only). Use 0.0.0.0 for a remote scraper.
Errors: a port that can't be bound is exit code 4.
Profiling: with RK_PPROF=1, run-flow and run-suite also serve /debug/pprof/ on the same listener.

Go runtime and process metrics are included.

Flows and messages

Metric	Type	Labels	Meaning
`runik_flow_latency_seconds`	histogram	flow, result	End-to-end flow latency, from the flow's start to its final state.
`runik_flows_active`	gauge	flow	UEs whose flow is in progress: started and not yet in a final state.
`runik_flows_total`	counter	flow, result	Completed flows, labelled by flow and result (success/failed/timeout).
`runik_fsm_state_timeouts_total`	counter	flow, state	State timeouts that fired (on_timeout), by flow and state.
`runik_message_bytes_total`	counter	direction, msg_type	Bytes of NGAP messages sent (tx) and received (rx), by direction and msg_type.
`runik_messages_total`	counter	direction, msg_type	NGAP messages sent (tx) and received (rx), by direction and msg_type.
`runik_ngap_decode_errors_total`	counter	msg_type	Received NGAP messages that failed to decode. msg_type is unknown.
`runik_ngap_encode_errors_total`	counter	msg_type	NGAP messages a flow sends that failed to encode, by the flow's message name.
`runik_pipeline_stage_seconds`	histogram	direction, stage, msg_type	Time each message spends in each processing stage.
`runik_procedure_failures_total`	counter	procedure	Procedure failures observed on the wire, labelled by procedure name.
`runik_step_latency_seconds`	histogram	flow, from_state, to_state	Time spent in each state, by flow and transition (from_state, to_state).
`runik_ue_held`	gauge	flow	UEs held in their flow's hold_state by a population-hold run (-hold).

Network state

Metric	Type	Labels	Meaning
`runik_gnb_overload_shed_total`	counter	gnb, action	Originating accesses (INITIAL UE MESSAGE) a simulated gNB refused to send because the AMF signalled NGAP OVERLOAD START, labelled by gNB and by the OverloadAction in force.
`runik_gnb_registered`	gauge		Simulated gNBs that completed NG Setup with the AMF.
`runik_pdu_sessions_active`	gauge		PDU sessions currently active.
`runik_ue_deregistrations_total`	counter		UE deregistrations completed (Deregistration Accept received).
`runik_ue_registered`	gauge		UEs currently registered with the core.

SCTP (N2)

The labels kind, name and peer identify the association; path is one peer address.

Metric	Type	Labels	Meaning
`runik_sctp_association_up`	gauge	kind, name, peer	1 while the SCTP association is up, 0 once closed or lost.
`runik_sctp_data_chunks_total`	counter	kind, name, peer, direction	DATA chunks (ordered and unordered) sent (direction=out) and received (direction=in) on Runik's SCTP associations.
`runik_sctp_events_total`	counter	kind, name, peer, event	SCTP notifications on Runik's N2 associations, by type:state (assoc_change:COMM_LOST, peer_addr_change:unreachable, shutdown).
`runik_sctp_packets_total`	counter	kind, name, peer, direction	SCTP packets sent (direction=out) and received (direction=in) on Runik's SCTP associations.
`runik_sctp_path_active`	gauge	kind, name, peer, path	1 when the peer address is in state active, 0 otherwise.
`runik_sctp_path_events_total`	counter	kind, name, peer, path, event	SCTP peer address notifications per path of Runik's N2 associations (event: unreachable, available, potentially-failed, made-primary, ...). A path failure the association survives counts here, not as an association loss.
`runik_sctp_path_primary`	gauge	kind, name, peer, path	1 for the peer address that is the SCTP association's primary path, 0 for its other addresses.
`runik_sctp_path_rto_seconds`	gauge	kind, name, peer, path	Current retransmission timeout of each peer address of an SCTP association.
`runik_sctp_path_srtt_seconds`	gauge	kind, name, peer, path	Smoothed round-trip time of each peer address of an SCTP association.
`runik_sctp_retransmitted_chunks_total`	counter	kind, name, peer	DATA chunks retransmitted on Runik's SCTP associations.

The internal message bus

Metric	Type	Labels	Meaning
`runik_bus_events_dropped_total`	counter	reason	Events dropped, by reason: no handler, a stopped bus, or overload_shed (an inbound frame the bus had no room for within its admission wait: Runik itself is overloaded).
`runik_bus_handler_duration_seconds`	histogram	address	Wall-clock time spent in each event bus handler.
`runik_bus_handler_errors_total`	counter	address	Errors returned by event bus handlers, per bus address.
`runik_bus_handler_total`	counter	address	Events handled, per bus address.
`runik_bus_in_flight`	gauge		Events the bus has accepted and not yet finished.
`runik_bus_ingress_blocked_seconds_total`	counter		Total time transport readers waited for bus room.
`runik_bus_ingress_deferred_total`	counter		Inbound frames whose transport reader waited for bus room (backpressure).
`runik_bus_queue_depth`	gauge		Current number of events waiting in the bus queue.

rk daemon

Metric	Type	Labels	Meaning
`runik_worker_busy`	gauge		1 while the worker is running a job, else 0.
`runik_worker_connected`	gauge		1 while the worker is connected to Runik Control, else 0.
`runik_worker_held_budget_total`	counter	action	Unacknowledged reports and suite cycles the held-memory budget trimmed or dropped, by action (trimmed_report / dropped_cycle / dropped_report).
`runik_worker_job_duration_seconds`	histogram		Wall-clock time of a job, from its start to its report.
`runik_worker_jobs_total`	counter	result	Jobs the worker finished, labelled by result (pass / fail / unsupported / error / cancelled / rejected).
`runik_worker_live_events_dropped_total`	counter	reason	Live job events the worker did not send to Runik Control, by reason (queue_full / no_stream). The job's report still carries its full event log.
`runik_worker_reconnects_total`	counter		Times the worker reconnected to Runik Control after losing the connection.

rk server uspace

Metric	Type	Meaning
`runik_server_uspace_bytes_rx_total`	counter	Inbound bytes received by rk server uspace.
`runik_server_uspace_bytes_tx_total`	counter	Echoed bytes sent by rk server uspace.
`runik_server_uspace_errors_total`	counter	Errors observed by rk server uspace.
`runik_server_uspace_pkt_rx_total`	counter	Inbound packets received by rk server uspace.
`runik_server_uspace_pkt_tx_total`	counter	Echoed packets sent by rk server uspace.

rk server dpdk

The same counters as the run's report (transport.gtpu_reflector[]), read when Prometheus scrapes. Every drop reason is always present, at 0 until it happens.

Metric	Type	Labels	Meaning
`runik_server_dpdk_active_seconds`	gauge		Seconds that carried G-PDUs; reflected packets per second is gpdu_reflected_total / active_seconds.
`runik_server_dpdk_bytes_rx_total`	counter		Bytes received by rk server dpdk.
`runik_server_dpdk_bytes_tx_total`	counter		Bytes sent by rk server dpdk.
`runik_server_dpdk_drops_total`	counter	reason	Packets dropped, by reason: the report's drops_by_reason (gtp_dropped, not_served, tx_queue, nic_rx_missed, ...).
`runik_server_dpdk_gpdu_reflected_total`	counter		G-PDUs reflected back as downlink (with -gtp).
`runik_server_dpdk_gpdu_rx_total`	counter		G-PDUs received (with -gtp).
`runik_server_dpdk_pkt_rx_total`	counter		Packets received by rk server dpdk.
`runik_server_dpdk_pkt_tx_total`	counter		Packets sent by rk server dpdk.