Documentation Grafana Enterprise Traces Set up Set up GET tenants

Set up a tenant in your Grafana Enterprise Traces cluster

Tenants provide a mechanism for trace isolation. Access policies may be set on a per-tenant basis. Authorization of requests is based on specified access policies.

These instructions assume that you have the Grafana Enterprise Traces (GET) plugin installed. Use this plugin to create tenants, access policies, and tokens for your GET cluster. The steps include:

1. Create a GET tenant.
2. Create an access policy.
3. Configure remote-write to your tenant.
4. Set up your tenant as a Grafana data source.
5. Visualize your data.

Create a GET tenant

Once a cluster is running, you can create new tenants. To create a GET tenant:

1. Within Grafana Enterprise, navigate to Apps > Grafana Enterprise Traces > Tenants.

2. Click on Create tenant.

3. Choose a name for this tenant. For demonstration purposes, use the name dev-tenant.

4. Select the cluster.

5. Click Create tenant.

Create an access policy

Access policies are used to authorize actions and operations by specified tenants. Access policies have a realm, which defines the set of tenants they apply to, and a scope, which defines the set of actions upon which they confer permissions to use.

Grafana Enterprise requires a data source access policy and token to access the traces in the tenant named dev-tenant.

1. Navigate to Apps > Grafana Enterprise Traces > Access policies.
2. Click Create access policy.
3. Choose a name for the policy. For demonstration purposes, use dev-read-write-policy.
4. Under Scopes, select the toggle next to the question Are you planning on creating a data source that uses this access policy?. This enables the traces:read policy.
5. Enable the scope traces:write to create an access policy that grants access for both reading and writing trace data (as the previous step toggles the traces:read policy automatically).
6. Select the tenant dev-tenant.
7. Click on Create.
8. From the newly created access policy, click Add token.
9. Name the token dev-token and click on Create.
10. In the next window, copy the token by selecting Copy to clipboard.

At this point, you can add a data source to your Grafana Enterprise instance by using the Create a data source option on this dialog or by manually adding one using the procedure below.

1. Select Create a data source. The data source fields are automatically configured based upon your instance.
2. Verify the created data source by opening the Configuration option in the Grafana menu and then selecting Data sources.

Set up remote-write to your tenant

To enable writes to your cluster, add a remote-write configuration snippet to the configuration file of an existing Grafana Alloy instance. If you do not have an existing traces collector, refer to Set up with Grafana Alloy. For Kubernetes, refer to the Grafana Alloy Kubernetes quick start guide.

The sample below is an example Alloy configuration below for the Kubernetes Helm chart (the values.yaml manifest):

yaml Copy

alloy:
  extraPorts:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
      protocol: TCP
  configMap:
    create: true
    content: |-
      // Creates a receiver for OTLP gRPC.
      // You can easily add receivers for other protocols by using the correct component
      // from the reference list at: https://grafana.com/docs/alloy/latest/reference/components/
      otelcol.receiver.otlp "otlp_receiver" {
        // Listen on all available bindable addresses on port 4317 (which is the
        // default OTLP gRPC port) for the OTLP protocol.
        grpc {
          endpoint = "0.0.0.0:4317"
        }

        // Output straight to the OTLP gRPC exporter. We would usually do some processing
        // first, most likely batch processing, but for this example we pass it straight
        // through.
        output {
          traces = [
            otelcol.exporter.otlp.get.input,
          ]
        }
      }

      // Define an OTLP gRPC exporter to send all received traces to GET.
      // The unique label 'get' is added to uniquely identify this exporter.
      otelcol.exporter.otlp "get" {
          // Define the client for exporting.
          client {
              // Use Header authentication, defined below.
              auth = otelcol.auth.headers.credentials.handler
              // Send to the locally running Tempo instance, on port 4317 (OTLP gRPC).
              endpoint = "http://get-cluster-tempo-enterprise-gateway.get.svc.cluster.local:3100"
              // Configure TLS settings for communicating with the endpoint.
              tls {
                  // The connection is insecure.
                  insecure = true
                  // Do not verify TLS certificates when connecting.
                  insecure_skip_verify = true
              }
          }
      }

      // A default GET install doesn't expose a TLS-encrypted gateway.
      // Grafana Alloy expects gRPC (HTTP/2) traffic to be TLS-encrypted, and
      // therefore fails if the "basic_auth" authorization block is used. Because of
      // this, use the generic "headers" block to authenticate a GET
      // tenant and token.
      // The authorization must be exposed as a '<tenant>:<writeToken>' pair that
      // has been base64 encrypted.
      otelcol.auth.headers "credentials" {
        header {
          key = "Authorization"
          value = "Basic ZGV2LXRlbmFudDpaR1YyTFhKbFlXUXRkM0pwZEdWeUxYQnZiR2xqZVMxa1pXMXZMWFJ2YTJWdU9qWS9lemR1TVRWaEpEUXZQR012THpRMVN6Z3NKakZiTVE9PQo="
        }
      }

In the above Flow configuration, the token for authentication was generated using the dev-tenant user and writeable token via:

bash Copy

echo 'dev-tenant:ZGV2LXJlYWQtd3JpdGVyLXBvbGljeS1kZW1vLXRva2VuOjY/ezduMTVhJDQvPGMvLzQ1SzgsJjFbMQ==' | base64

Create a Grafana data source

To allow Grafana to read traces from GET, you must create a Tempo data source with the proper credentials.

1. Navigate to Connections > Data Sources.

2. Click on Add a data source.

3. Select Tempo.

4. Set the URL to http://<get-gateway-host>:<http_listen_port>/<http_api_prefix>, filling in the path to your gateway and the configured HTTP API prefix.

5. Enable Basic Auth. Use User dev-tenant and the token from your clipboard as the Password.

6. Click Save & Test.

You should see a message that says Data source is working.

Visualize your data

Once you have created a data source, you can visualize your traces in the Grafana Explore page.

Test your configuration using the Intro to MLTP application

The Intro to MLTP application provides an example five-service application that generates data for Tempo, Mimir, Loki, and Pyroscope. This procedure installs the application on your cluster so you can generate meaningful test data.

1. Navigate to https://github.com/grafana/intro-to-mltp to get the Kubernetes manifests for the Intro to MLTP application.
2. Clone the repository using commands similar to the ones below:
   bash Copy

     git clone git+ssh://github.com/grafana/intro-to-mltp
     cp intro-to-mltp/k8s/mythical/* ~/tmp/intro-to-mltp-k8s

3. Change to the cloned repository: cd intro-to-mltp/k8s/mythical
4. In the mythical-beasts-deployment.yaml manifest, alter each TRACING_COLLECTOR_HOST environment variable instance value to point to the Alloy location. For example, based on an Alloy installation in the default namespace called and with a Helm installation called test:
   yaml Copy

    	- env:
        ...
        - name: TRACING_COLLECTOR_HOST
          value: test-grafana-alloy.default.svc.cluster.local

5. Deploy the Intro to MLTP application. It deploys into the default namespace.
   bash Copy

       kubectl apply -f mythical-beasts-service.yaml,mythical-beasts-persistentvolumeclaim.yaml,mythical-beasts-deployment.yaml

6. Once the application is deployed, go to Grafana Enterprise and select the Explore menu item.
7. Select the GET data source from the list of data sources.
8. Select the Search Query type for the data source.
9. Select Run query.
10. Traces from the application are displayed in the traces Explore panel.