Graphite HTTP API
The HTTP API is the same as that of Graphite, with the addition of ingestion, authentication, and storage-aggregation/storage-schemas.conf
endpoints.
First of all, there are two endpoints you will be talking to. They are provided in the Cloud Portal by first going to the Details for your stack, then selecting the Details of the Graphite card within your stack.
They will look something like:
<ingest_endpoint>
:https://something.grafana.net/graphite/metrics
<query_endpoint>
:https://something.grafana.net/graphite
You also need to create an access policy with the metrics:read
and/or metrics:write
scopes.
You can then create a token for that access policy, which will be used to talk to the API (as shown below).
See Cloud Access Policies for more information.
Authentication
All endpoints require authentication, which can done using any of these HTTP headers:
Authorization: Basic base64(<user>:<token>)
Authorization: Bearer <user>:<token>
Your user can be found in the details of your Graphite instance details page in your Grafana.com stack.
Common Request Parameters
Many of the API methods involve using Graphite patterns (queries), tag queries and the standard Graphite from/to syntax.
Graphite Patterns
Graphite patterns are queries that involve glob patterns (*
, {}
, []
, ?
).
Tag Expressions
Tags expressions are strings, and may have the following formats:
tag=spec tag value exactly matches spec
tag!=spec tag value does not exactly match spec
tag=~value tag value matches the regular expression spec
tag!=~spec tag value does not match the regular expression spec
Any tag spec that matches an empty value is considered to match series that don’t have that tag, and at least one tag spec must require a non-empty value. Regular expression conditions are treated as being anchored at the start of the value.
From/To (Until)
Endpoints
Adding New Data: Posting To /metrics
The main entry point for any publisher to publish data to, be it carbon-relay-ng, or any other script or application such as the hosted-metrics-sender-example
- Method: POST
- Access policy scope:
metrics:write
Headers
Content-Type
: supports 3 values:application/json
: the simplest one, and the one used herert-metric-binary
: same datastructure, but messagepack encodedrt-metric-binary-snappy
: same as above, but snappy compressed
Data Format
Each metricpoint message can have the following properties:
name // Graphite style name (required)
interval // the resolution of the metric in seconds (required)
value // float64 value (required)
time // unix timestamp in seconds (required)
tags // list of key=value pairs of tags (optional)
Example
timestamp_now_rounded=$(($(date +%s) / 10 * 10))
timestamp_prev_rounded=$((timestamp_now_rounded - 10))
curl -X POST -H "Authorization: Bearer <user>:<token>" -H "Content-Type: application/json" "<ingest_endpoint>" -d '[{
"name": "test.metric",
"interval": 10,
"value": 12.345,
"time": '$timestamp_prev_rounded'
},
{
"name": "test.metric",
"interval": 10,
"value": 12.345,
"time": '$timestamp_now_rounded'
},
{
"name": "test.metric.tagged",
"interval": 10,
"value": 1,
"tags": ["foo=bar", "baz=quux"],
"time": '$timestamp_prev_rounded'
},
{
"name": "test.metric.tagged",
"interval": 10,
"value": 2,
"tags": ["foo=bar", "baz=quux"],
"time": '$timestamp_now_rounded'
}
]'
Render /render
(return data for a given query)
Graphite-web-like api. It can return JSON, pickle or messagepack output. https://graphite.readthedocs.io/en/latest/render_api.html
- Method: GET or POST (recommended, as GET may result in too long URLs)
- Access policy scope:
metrics:read
Parameters
- maxDataPoints: unsigned int (default: 800)
- target (required): one or more metric names or patterns.
- from: see timespec format (default: 24h ago) (exclusive)
- to/until: see timespec format(default: now) (inclusive)
- format: json, msgp, pickle, or msgpack (default: json) (note: msgp and msgpack are similar, but msgpack is for use with Graphite)
- meta: This is a legacy flag used for a previous version of Cloud Graphite. For supported functions, the response will look like
{ "meta": {...}, "series" : <default json format> }
. If a function does not support themeta
parameter, the reponse will be returned in the default json format instead. The recommendation is to avoid using themeta
parameter, as many of themetadata
fields are not set by the current Cloud Graphite version and this parameter may be completely removed in later versions.
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/render?target=statsd.fakesite.counters.session_start.*.count&from=-3h&to=-2h"
Response
https://graphite.readthedocs.io/en/latest/render_api.html
Finding Metrics
Non-tagged with /metrics/expand
Returns metrics which match the query
. Results are limited to series that have received an update no longer than 32 days ago for performance reasons.
- Method: GET or POST
- Access policy scope:
metrics:read
Parameters
- query: required and may be specified multiple times, in the format described in Graphite pattern
- groupByExpr: if true, then the results get grouped by the query which yielded them, otherwise all results are in a flat list. (defaults to false)
- leavesOnly: if true, only leaf nodes get returned, if false branch nodes also get returned. (defaults to false)
- jsonp: true/false: enables jsonp
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/expand?groupByExpr=true&query=some.id.of.a.metric.[1-3]&query=some.id.of.a.metric.[67]"
{
"some.id.of.a.metric.[1-3]": [
"some.id.of.a.metric.1",
"some.id.of.a.metric.2",
"some.id.of.a.metric.3"
],
"some.id.of.a.metric.[67]": [
"some.id.of.a.metric.6",
"some.id.of.a.metric.7"
]
}
Non-tagged With /metrics/find
Returns metrics which match the query
and have received an update since from
,
de-duplicated by the last name node (used for the name auto-complete). Results
might be limited to data that is no older than 32 days for performance reasons.
- Method: GET or POST
- Access policy scope:
metrics:read
Parameters
- query: required and may be specified multiple times, in the format described in Graphite pattern
- format: json, treejson, completer, pickle, or msgpack. (defaults to json)
- jsonp: true/false: enables jsonp
- from: see timespec format (defaults to now-24hours)
Output formats
- json, treejson (default/unspecified): the standard format
- completer: used for graphite-web’s completer UI
- msgpack: optimized transfer format
- pickle: deprecated
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/find?query=foobar"
[
{
"allowChildren": 1,
"expandable": 1,
"leaf": 0,
"id": "foobar",
"text": "foobar",
"context": {}
}
]
The response indicates that there are metric names that live under the “foobar” term (it is expandable) and there is no data under the name “foobar” (it is not a leaf node).
So we update the query to see what we can expand to:
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/find?query=foobar.*"
[
{
"allowChildren": 1,
"expandable": 1,
"leaf": 0,
"id": "foobar.aggstats",
"text": "aggstats",
"context": {}
}
]
The response for the updated query shows which data lives under the “foobar” name, in this case the tree extends under “foobar.aggstats”.
As we continue to dig deeper into the tree, by updating our query based on what we get back, we eventually end up at the leaf:
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/find?query=foobar.aggstats.*.tank.metrics_active.gauge32"
[
{
"allowChildren": 0,
"expandable": 0,
"leaf": 1,
"id": "foobar.aggstats.us-east2-id-name.tank.metrics_active.gauge32",
"text": "gauge32",
"context": {}
}
]
Tagged With /tags/findSeries
Returns metrics which match tag queries and have received an update since from
.
Note: the returned results are not deduplicated and in certain cases it is possible
that duplicate entries will be returned.
- Method: GET or POST
- Access policy scope:
metrics:read
Parameters
- expr (required): a list of tag expressions
- from: see timespec format (defaults to now-24hours)
- format: series-json, lastts-json. (defaults to series-json)
- limit: max number to return. Zero means no limit. (default: 0) Note: the resultset is also subjected to the cluster configuration. if the result set is larger than the cluster configuration, an error is returned. If it breaches the provided limit, the result is truncated.
- meta: this is a legacy flag used for a previous version of Cloud Graphite. If enabled, the response includes meta information like warnings, with the format
{ "warning": {...}, "series" : <default format> }
. The recommendation is to avoid using themeta
parameter, as this parameter may be completely removed in later versions. (defaults to false)
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/findSeries?expr=datacenter=dc1&expr=server=web01"
[
"disk.used;datacenter=dc1;rack=a1;server=web01"
]
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/findSeries?expr=datacenter=dc1&expr=server=web01&format=lastts-json"
{
"series": [
{
"lastTs": 1576683990,
"val": "disk.used;datacenter=dc1;rack=a1;server=web01"
}
]
}
All Series With /metrics/index.json
Returns the names of all metrics (both tagged and non-tagged) that have received an update within the last hour.
- Method: GET or POST
- Access policy scope:
metrics:read
Parameters
- jsonp: a jsonp callback to wrap the response (optional)
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/metrics/index.json
[
"collectd.prod-us-central-0.mysql.bytes.ibuf_size",
"collectd.prod-eu-west-2.mysql.bytes.ibuf_size",
"collectd.prod-us-east-2.mysql.bytes.ibuf_size"
]
Exploring Tags
Finding Tags With /tags
Returns the existing metric tags that match the given filter and time range. Results might be limited to data that is no older than 32 days for performance reasons.
- Method: GET or POST
- Access policy scope:
metrics:read
Parameters
- from: see timespec format (defaults to now-24hours)
- to/until: see timespec format(default: now) (inclusive)
- filter: a regex to use to filter the tags. (default: .+)
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags?filter=foo"
[
{
"tag": "foobar"
},
{
"tag": "foobaz"
}
]
Finding Tag Values With /tags/{tag}
Returns the existing values for the given metric tag. Results might be limited to data that is no older than 32 days for performance reasons.
- Method: GET or POST
- Access policy scope:
metrics:read
Parameters
- from: see timespec format (defaults to now-24hours)
- to/until: see timespec format (default: now) (inclusive)
- tag (required): the tag for which to find values.
- filter: a regex to use to filter the values. (default: .+)
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/foobar?filter=gr"
{
"tag": "foobar",
"values": [
{
"count": 1,
"value": "graphite"
},
{
"count": 2,
"value": "grafana"
}
]
}
Autocompletion Of Tags With /tags/autoComplete/tags
Returns the existing tags of metrics that match the given expression. Results might be limited to data that is no older than 32 days for performance reasons.
- Method: GET or POST
- Access policy scope:
metrics:read
Parameters
- from: see timespec format (defaults to now-24hours)
- to/until: see timespec format(default: now) (inclusive)
- expr: an expression to filter the tagged metrics. Multiple are supported.
- tagPrefix: a prefix to filter tags further. (default: “”)
- limit: a number to limit to number of results. (default: 100)
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/autoComplete/tags?expr=foobar=graphite&tagPrefix=qu"
[
"qux",
"quy"
]
Autocompletion Of Tag Values With /tags/autoComplete/values
Returns the existing values for tags of metrics that match the given expression. Results might be limited to data that is no older than 32 days for performance reasons.
- Method: GET or POST
- Access policy scope:
metrics:read
Parameters
- from: see timespec format (defaults to now-24hours)
- to/until: see timespec format(default: now) (inclusive)
- tag (required): the tag for which to find values.
- expr: an expression to filter the tagged metrics. Multiple are supported.
- valuePrefix: a prefix to filter the tag’s values further. (default: “”)
- limit: a number to limit to number of results. (default: 100)
Example
curl -H "Authorization: Bearer <user>:<token>" "<query_endpoint>/tags/autoComplete/values?tag=qux&expr=foobar=graphite&valuePrefix=lo"
[
"loremipsum"
]
Read or Adjust storage-schemas.conf and storage-aggregation.conf
Grafana Cloud Graphite allows for dynamic changing of both the Graphite storage schemas and aggregation configurations. When interval data or rollups are improperly configured (for example, defaults not matching the real data or their intended use) this can result in suboptimal visualizations. Correcting these configurations can fix these problems instantly, unlike with standard Graphite installations, because changes are applied retroactively without any loss of information. You can post as many changes as needed. If you do not update these configurations, the default Grafana Cloud settings are applied instead.
Carbon-relay-NG version 1.2 or later automatically pushes both of these files for you. You should not need to manually push new configurations, and carbon-relay-ng is the recommended way of publishing these files.
You can manually POST to this endpoint when:
- you haven’t upgraded your relay yet.
- you want to experiment iteratively without having to repeatedly restart your relay.
There are two URL paths:
/graphite/config/storageSchemas
for storage-schemas.conf/graphite/config/storageAggregation
for storage-aggregation.confMethod: GET or POST
Parameters
None, and when posting the body is the file contents.
Example
You can download the current Schema and Aggregation configurationss using the API endpoints in GET mode:
curl -X GET -H "Authorization: Bearer <user>:<api key>" <query_endpoint>/config/storageSchemas
curl -X GET -H "Authorization: Bearer <user>:<api key>" <query_endpoint>/config/storageAggregation
You can push new configurations in POST mode:
curl -H "Authorization: Bearer <user>:<api key>" "<query_endpoint>/config/storageSchemas" --data-binary '@<path-to-storage-schemas.conf>'
curl -H "Authorization: Bearer <user>:<api key>" "<query_endpoint>/config/storageAggregation" --data-binary '@<path-to-storage-aggregation.conf>'