Documentation Grafana Cloud Testing and synthetics Synthetic Monitoring Set up Private probes

Set up private probes

Probes are blackbox agents responsible for executing the configured checks on specified targets, which are URLs or endpoints of web applications or services that you want to monitor. The probes simulate user interactions with these targets by sending requests and recording the responses.

Each probe sends metrics and logs to the Synthetic Monitoring backend, including such information as target availability and health and response latencies. The Synthetic Monitoring dashboard then displays that information.

In addition to the public probes run by Grafana Labs, you can also install private probes. These are only accessible to you and only write data to your Grafana Cloud account. Private probes are instances of the open source Grafana Synthetic Monitoring Agent.

Before you begin

• You must have the Admin role in Grafana to access the Synthetic Monitoring config page.
• Install and configure Synthetic Monitoring before adding a private probe.
• Installing from the Grafana APT repository on amd64-based Debian and Ubuntu systems is the only packaging option provided. On other systems, you need to build the agent from source.
• You can install multiple private probes. Each private probe has its own unique Probe Authentication Token.

Add a new probe in your Grafana instance

1. Navigate to Testing & synthetics > Synthetics > Probes.
2. Click Add Private Probe.
3. Enter a Probe Name.
4. Enter a Latitude and Longitude to see an accurate location on the map panel.
5. Enter a Region. Defaults are AMER, EMEA, and APAC, but you may define your own.
6. Optionally enter up to three custom labels to identify your probe.
7. Click Save.
8. Copy the Probe Authentication Token and save it for the next step.

Probe API Server URL

Based on the region of your stack, you need to use a different API server URL when setting up a private probe. If you are unsure about the region of your stack, go to your Grafana instance and the Config page under Synthetic Monitoring and search for backend address.

Firewall configuration

To ensure successful communication between your private Synthetic Monitoring probes and the necessary services, adjust your firewall rules as follows:

Synthetic Monitoring server

• Allow outbound connections from the probe to the “API server URL” specified in the preceding table.

Prometheus and Loki instances

1. Identify the data sources: navigate to the Config section of the Synthetic Monitoring plugin in your Grafana instance.
2. Locate the Linked Data Sources section and click on each data source.
3. Obtain the URL: under the Connection section, note the URL associated with each data source.
4. Allow outbound connections from the probe to the hosts corresponding to these URLs.

Monitored targets

Consider the check type: the firewall configuration varies based on the specific checks you’ve configured.

• ICMP: for ping and traceroute checks.
• UDP or TCP: for DNS checks.
• TCP: for GRPC, HTTP, and MultiHTTP checks.

Allow inbound connections to the targets you want to monitor, ensuring compatibility with the required protocols.

Install the agent on Ubuntu/Debian

You can install the agent using the Grafana official APT repository.

1. Add the required packages and the repository GPG key.

   bash Copy

   sudo apt install -y wget gnupg
   sudo mkdir -p /etc/apt/keyrings/
   wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg

2. Add the repository for stable releases.

   bash Copy

   echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

3. Run the following commands.

   bash Copy

   sudo apt update
   sudo apt install synthetic-monitoring-agent

4. Configure the probe.

   1. Open /etc/synthetic-monitoring/synthetic-monitoring-agent.conf.
   2. Replace <YOUR TOKEN HERE> with the probe authentication token.
   3. Repleace API_SERVER with Probe API Server URL based on your stack region. (defaults to US region)
   4. Save the file.

5. Restart the agent.

   bash Copy

   sudo service synthetic-monitoring-agent restart

The new probe appears on your Synthetics > Probes page.

Install the agent on distributions using yum

You can install the agent using the official YUM repository.

1. Add the repository and the GPG key to the configuration.

   bash Copy

   sudo yum-config-manager --add-repo https://rpm.grafana.com
   wget https://rpm.grafana.com/gpg.key
   sudo rpm --import gpg.key

2. Install the agent.

   bash Copy

   sudo yum install synthetic-monitoring-agent
   sudo systemctl status synthetic-monitoring-agent.service

   This last command will tell you that the service is loaded but disabled. Before enabling, it you need to add the probe to your Synthetic Monitoring configuration.

3. Configure the agent.

   Edit the file /etc/synthetic-monitoring/synthetic-monitoring-agent.conf and make the following changes:

   • Replace <YOUR TOKEN HERE> with the probe authentication token.
   • Replace API_SERVER with the API Server URL corresponding to your stack’s region. Defaults to the US region.

4. Enable the service and start it.

   bash Copy

   sudo systemctl enable synthetic-monitoring-agent.service
   sudo systemctl start synthetic-monitoring-agent.service

Deploy the agent using Docker

You can find the Synthetic Monitoring Agent Docker image on Docker Hub

1. Pull image from Docker Hub.

   bash Copy

   docker pull grafana/synthetic-monitoring-agent:latest

2. Export required configuration as environment variables.

   Replace <YOUR TOKEN> with the probe authentication token. Replace <PROBE_API_SERVER_URL> with Probe API Server URL based on your stack region.

   bash Copy

   export API_TOKEN=<YOUR_TOKEN>
   export API_SERVER="<PROBE_API_SERVER_URL>"

3. Start the agent.

   bash Copy

   docker run grafana/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --verbose=true

The new probe appears on your Synthetics > Probes page.

Deployment with Kubernetes

You can deploy the Synthetic Monitoring Agent to Kubernetes using the Synthetic Monitoring Agent Docker image.

In the agent GitHub repository, you can also find a Kubernetes Deployment example. The following section uses the example files to deploy a Synthetic Monitoring agent in a Kubernetes namespace.

1. Get the deployment files.

   bash Copy

   wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/namespace.yaml
   wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/secret.yaml
   wget https://raw.githubusercontent.com/grafana/synthetic-monitoring-agent/main/examples/kubernetes/deployment.yaml

2. Create the synthetic-monitoring namespace.

   bash Copy

   kubectl apply -f namespace.yaml

3. Create a secret with the Probe Authentication Token.

   Replace YOUR_TOKEN_HERE in secret.yaml with your Probe Authentication Token.

   bash Copy

   kubectl apply -f secret.yaml
   
   # verify secret
   kubectl get secrets -n synthetic-monitoring

4. Deploy the Synthetic Monitoring Agent.

   Replace PROBE_API_SERVER_URL in deployment.yaml with Probe API Server URL based on your stack region.

   bash Copy

   kubectl apply -f deployment.yaml

The new probe appears on your Synthetics > Probes page.

Configure feature flags

You can use feature flags to enable features in the private probe.

There are two ways to enable features:

• Edit the file /etc/synthetic-monitoring/synthetic-monitoring-agent.conf and make the following changes:

  ini Copy

  FEATURES = <list of features comma separated>

• Export the list of features as an environment variable.

  bash Copy

  export FEATURES=<list of features comma separated>

Available feature flags

These features are disabled by default.

Troubleshooting private probes

If you are facing problems with your private probes, check the following:

1. Check the probe version using the -version flag and make sure you are running a supported version.
2. Make sure the probe has a valid probe access token.
3. Make sure the probe can connect to the Synthetic Monitoring API server.
4. Run the Synthetic Monitoring Agent in debug mode with the -debug and -verbose flags and check for errors in the logs.
5. Check the Synthetic Monitoring Agent metrics. The default address for Synthetic Monitoring Agent Prometheus metrics is localhost:4050/metrics.

If you are still facing issues with Private Probes, ask for help in the community forum. Refer to Account types and support available for Grafana Cloud users for more information.

Production deployments

The agent process exposes Prometheus-style metrics on an HTTP server running on port 4050 with the “/metrics” endpoint.

You can scrape and monitor your private probe using Prometheus or Grafana Agent.

Next steps

You must reconfigure any checks to use your new probe even if you selected all probes when initially creating the check.

The Synthetic Monitoring Agent is open source and available on GitHub.