Stack architecture recommendations

Grafana recommends that customers use a single stack as their production observability environment.

Single stack for larger companies

For larger companies comprised of multiple departments or teams, a single production stack would look as follows:

Each group or team within the company should be represented as a Grafana Team. Each Grafana Team is composed of Grafana users.
Each Grafana Team should be allocated a folder in the stack’s Managed Grafana instance where members of that team can create their own resources, such as dashboards, and organize them using sub-folders.
All metrics, logs, traces, and profiles data should be sent to the endpoints associated with the single stack. Grafana recommends creating one write access policy per-team or per-collector such as Grafana Alloy. This allows you to revoke write access to the stack for a single team or collector without disrupting other teams or collectors.
To restrict query access to the data sources connected to your Grafana, use Data Source permissions. Access to each data source can be set on a per-user or per-team level.
The Prometheus and Loki data sources connected to the metrics and logs stored in Grafana Cloud additionally support finer grained access controls via the label based access controls features (LBAC), choose between LBAC for access policies and LBAC for data sources.
To understand how each team is contributing to the total volume of telemetry being ingested into the stack, segment ingestion volume based on a user-defined label key. Telemetry data sent to Grafana Cloud must be properly labeled to be segmented.
- To segment metric ingestion volume into the stack by team, use usage attribution reports.
- To segment log ingestion volume into the stack by team, use log volume explorer.
- To segment traces ingestion volume into the stack by team, enable the metrics-generator.

Use staging and development stacks

Consider creating a staging and development stack to use as a sandbox environment to prototype new dashboards and alerts. After validating changes in staging and development, you can promote the changes to your production observability stack, reducing the risk of unintended breakages. This can be automated with a continuous integration and continuous deployment (CI/CD) pipeline. To get started, refer to Provisioning Grafana Cloud with infrastructure as code.

Use multiple production stacks for isolation

While a single production stack is the default recommendation, multiple production stacks are recommended when the goal is complete isolation among departments or teams and there is no expectation that data or resources in different stacks need to be combined. This is typically used by resellers and managed service providers that have customers who shouldn’t have access to one another’s data.