How Grafana Cloud Asserts determines causality between events

The system logs assertions according to statistical anomalies (such as seasonality), 5m, 30m, 1hr, 6hr multi-window error rates, system saturation, failures, and updates (such as new deployments, configuration changes, and so on). Asserts uses the entity graph and time and space correlation to determine and display the potential cause-and-effect relationships between assertions.

The following chain of events occurs when an alert fires:

Asserts uses the labels in the alert to reverse lookup to find the target entity in the entity graph.
Asserts traverses the graph to infer the first, second, third, and fourth degree connections and corresponding assertions.
- First-degree connection examples: Service - Service, Service - Pod
- Second-degree connection examples: Service - Pod - Node, Producer - Topic - Consumer
- Third and fourth-degree connection example: Service - (Container) - Node - (Container) - Service

The Asserts Knowledge Engine is familiar with failure patterns for many popular and some less popular components and run times. For more information about assertions and the SAAFE model, refer to About base assertions.

You can use code or the user interface to add rules for new entities, relations, assertions, and inferences.