Documentation Grafana Cloud Cost management and billing Reduce costs Metrics costs Adaptive Metrics Adaptive Metrics API

Adaptive Metrics HTTP API

The following topic describes the Adaptive Metrics HTTP API.

Before you begin

To interact with the API, gather the following information:

• URL: In the form https://<your-grafana-cloud-prom-url>.grafana.net. To find your URL value, go to your grafana.com account and check the Details page of your hosted Prometheus endpoint.
• TENANT: The numeric instance ID where you want to use Adaptive Metrics. To find your TENANT value, go to your grafana.com account and check the Details page of your hosted Prometheus endpoint for Username / Instance ID.
• TOKEN: A token from a Grafana Cloud Access Policy that has the required permissions to interact with the Adaptive Metrics resources for your stack ID, as shown in the following examples. More details about available permissions and scopes can be found in RBAC for Adaptive Metrics.

The format of an aggregation rule

Each aggregation rule looks similar to this:

Copy

  {
    "metric": "agent_request_duration_seconds_sum",
    "drop_labels": [
      "container",
      "instance",
      "method",
      "namespace",
      "pod",
      "provider",
      "status_code",
      "ws"
    ],
     "aggregations": [
    	"sum:counter"
    ]
  }

In the preceding example:

• metric is the name of the metric to be aggregated.
• drop_labels is an array of the labels that will be aggregated and their value will be set to <aggregated>.
• aggregations is an array of the aggregation types to calculate for this metric.

You can use an aggregation rule file to define multiple rules simultaneously.

The following example rule file is an array of one or more aggregation rules:

json Copy

[
  {
    "metric": "agent_request_duration_seconds_sum",
    "drop_labels": ["namespace", "pod"],
    "aggregations": ["sum:counter"]
  },

  {
    "metric": "prometheus_request_duration_seconds_sum",
    "drop_labels": ["container", "instance", "ws"],
    "aggregations": ["sum:counter"]
  }
]

List recommendations

Download recommendations for metrics to aggregate using the command below. TOKEN and TENANT are variables defined within the requirements section.

bash Copy

curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations"

TOKEN must belong to an access policy with the adaptive-metrics-recommendations:read scope.

You can use an optional verbose flag to retrieve more information about each recommendation:

bash Copy

curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations?verbose=true"

The following is example of verbose recommendation rule:

json Copy

[
  {
    "metric": "probe_http_duration_seconds",
    "drop_labels": ["config_version", "group", "method", "name", "url"],
    "aggregations": ["count", "sum"],
    "recommended_action": "add",
    "usages_in_rules": 0,
    "usages_in_queries": 0,
    "usages_in_dashboards": 2,
    "kept_labels": ["name", "instance", "job"],
    "total_series_after_aggregation": 1040,
    "total_series_before_aggregation": 1210
  },
  {
    "metric": "probe_dns_duration_seconds",
    "drop_labels": ["config_version"],
    "aggregations": ["count", "sum"],
    "recommended_action": "add",
    "usages_in_rules": 0,
    "usages_in_queries": 0,
    "usages_in_dashboards": 2,
    "kept_labels": ["name", "instance", "job"],
    "total_series_after_aggregation": 264,
    "total_series_before_aggregation": 396
  }
]

Any existing aggregation rules that have been previously applied will be included in the downloaded list of recommendations. This simplifies the process of viewing the recommended aggregation rules and applying them. By default, you can apply recommended rules and still maintain all of your existing rules.

List current recommendations configuration

Download the current configuration of the recommendations service using the following command:

bash Copy

curl -u "$TENANT:$TOKEN" "$URL/aggregations/recommendations/config"

TOKEN must belong to an access policy with the adaptive-metrics-config:read scope.

The only tunable parameter exposed by the recommendations service is the keep_labels parameter. This parameter allows the user to define a list of labels that they never want recommended for aggregation. This can be useful at organizations where certain labels are always expected on metrics, regardless of whether or not those labels have been recently queried.

An example response from the /recommendations/config endpoint would look as follows:

json Copy

{
  "keep_labels": ["instance", "pod"]
}

The preceding response indicates that the recommendations service has been configured to never recommend aggregating the instance or pod labels.

Recommendation exemptions

The Adaptive Metrics API provides the possibility to exempt certain metrics, labels, and metric-label combinations from generated recommendations.

The recommendation exemption format looks like this:

Below are examples of exemptions and how they would be reflected in the preceding mentioned format:

1. Completely exclude metric api_http_requests_total from being recommended for aggregation (keep full cardinality).

   json Copy

   {
     "metric": "api_http_requests_total"
   }

2. Exclude labels namespace and status for metric api_http_requests_total from being recommended for aggregation.

   json Copy

   {
     "metric": "api_http_requests_total",
     "keep_labels": ["namespace", "status"]
   }

3. Exclude the pod label for all metrics from being recommended for aggregation.

   json Copy

   {
     "keep_labels": ["pod"]
   }

Create a recommendation exemption

bash Copy

curl -u "$TENANT:$TOKEN" --request POST --data @exemption.json "$URL/v1/recommendations/exemptions

TOKEN must belong to an access policy with the adaptive-metrics-exemptions:write scope.

If successful, the server returns a JSON object including the id of the newly created exemption.

List recommendation exemptions

bash Copy

curl -u "$TENANT:$TOKEN" "$URL/v1/recommendations/exemptions

TOKEN must belong to an access policy with the adaptive-metrics-exemptions:read scope.

Get a recommendation exemption

bash Copy

curl -u "$TENANT:$TOKEN" "$URL/v1/recommendations/exemptions/<exemption_id>

TOKEN must belong to an access policy with the adaptive-metrics-exemptions:read scope.

The exemption_id is the identifier of the exemption to be fetched, as returned by the server when creating the exemption.

Update recommendation exemption

bash Copy

curl -u "$TENANT:$TOKEN" --request PUT --data @exemption.json "$URL/v1/recommendations/exemptions/<exemption_id>

TOKEN must belong to an access policy with the adaptive-metrics-exemptions:write scope.

The exemption_id is the identifier of the exemption to be updated, as returned by the server when creating the exemption.

Delete a recommendation exemption

bash Copy

curl -u "$TENANT:$TOKEN" --request DELETE "$URL/v1/recommendations/exemptions/<exemption_id>

TOKEN must belong to an access policy with the adaptive-metrics-exemptions:write scope.

The exemption_id is the identifier of the exemption to delete, as returned by the server when creating the exemption.

Update recommendations configuration

Upload new recommendations configuration using the following command:

bash Copy

curl -u "$TENANT:$TOKEN" --request POST --data @config.json "$URL/aggregations/recommendations/config"

TOKEN must belong to an access policy with the adaptive-metrics-recommendations:write scope.

This command uses the same endpoint described in List current recommendations configuration and expects the same JSON format.

List currently applied aggregation rules

Download your existing aggregation rules:

bash Copy

curl -u "$TENANT:$TOKEN" "$URL/aggregations/rules"

TOKEN must belong to an access policy with the adaptive-metrics-rules:read scope.

Upload new aggregation rules

Uploading new aggregation rules is a multi-step process:

1. Fetch the currently applied rules.
2. Modify rules locally.
3. Upload rules back.

There is a delay between uploading new aggregation rules and those metrics aggregations taking effect in your environment. In most cases, the delay is approximately 5-10 minutes, but we currently have no mechanism to let you know precisely when new aggregations take effect.

Grafana currently limits how often new aggregation rules are applied. Although you can upload as many new versions of your aggregation rules as you like, those updates are only applied once every 10 minutes. If you make multiple updates in quick succession, the system applies your first received (oldest) update. Then, 10 minutes later, the most recently received update is applied. The intermediate updates never get applied.

Fetch the currently applied rules

Use this command:

bash Copy

curl -u "$TENANT:$TOKEN" -D headers.txt "$URL/aggregations/rules" > rules.json

TOKEN must belong to an access policy with the adaptive-metrics-rules:read scope.

The preceding command uses an additional -D headers.txt argument, which stores the headers in a file called headers.txt.

This step is required if you want to then upload a new rule file, for example if you want to update the existing aggregation rules you have in place. The information in these headers ensures there are no update collisions. An update collision is the scenario where multiple users try to edit the rules file at the same time and overwrite one another’s changes.

Modify the rules locally

Use your editor of choice to modify the rules.json file downloaded in the prior step.

Upload rules back

The API supports uploading an entire rules file.

To upload your modified rules.json file from the previous step, use the following shell script:

bash Copy

TMPFILE=$(mktemp)
trap 'rm "$TMPFILE"' EXIT

cat headers.txt | grep -i '^etag:' | sed 's/^ETag:/If-Match:/i' > "$TMPFILE"

curl --request POST --header @"$TMPFILE" --data-binary @$1 -u "$TENANT:$TOKEN" "$URL/aggregations/rules"

TOKEN must belong to an access policy with the adaptive-metrics-rules:write scope.

The cat headers.txt command modifies the headers.txt file created in the previous curl call that pulled down the existing aggregation rules.

The curl --request POST command uploads your new rules file, as well as the updated headers.

Save the shell script as rules_upload.sh.

To run that script, use the following command:

bash Copy

./rules_upload.sh <your_new_rules_file.json>

Replace <your_new_rules_file.json> with the name of the rules file you wish to upload.