Grafana Cloud
Instrument and send data
Metrics
Send or visualize Graphite metrics
Graphite data ingestionGraphite data ingestion
The recommended way of sending data to Grafana Cloud Graphite is by using carbon-relay-ng. carbon-relay-ng is a Graphite carbon relay that supports aggregations and sending data to our endpoint over a secure, robust transport. You can also use our HTTP API directly; for that, see our docs and the golang, python, and shell examples.
Send data with carbon-relay-ng
This is the recommended and most popular option for sending data. In most situations, you should install a carbon-relay-ng service in each datacenter or region that will send metrics.
carbon-relay-ng will accept plain-text carbon (Graphite) input, and stream your encrypted metrics to the Grafana Cloud Graphite. Since carbon-relay-ng can buffer metric streams in memory, this also provides increased resiliency to connectivity issues.
carbon-relay-ng is available for most platforms and is easy to install. Visit the repository’s releases page to download the latest version.
Once installed, you can configure the relay in two steps:
- Edit
carbon-relay-ng.conf - Provide
storage-schemas.confandstorage-aggregation.conf
1. Edit carbon-relay-ng.conf
Edit the carbon-relay-ng.conf configuration file (normally located at /etc/carbon-relay-ng/carbon-relay-ng.conf), and replace it with the settings. Adjust the options to suit your needs:
- The “apikey” setting must be a Cloud Access Policy token with the
metrics:writescope.- Make sure the “schemasFile” and “aggregationFile” fields are set to the path to your storage-schemas.conf and storage-aggregation.conf files (see section below)
## Global settings ##
# Instance IDs distinguish stats of multiple relays.
# Do not run multiple relays with the same instance ID.
# Supported variables:
# ${HOST} : hostname
instance = "${HOST}"
## System ##
# This setting can be used to override the default GOMAXPROCS logic.
# It is ignored if the GOMAXPROCS environment variable is set.
max_procs = 2
pid_file = "carbon-relay-ng.pid"
# directory for spool files
spool_dir = "spool"
## Logging ##
# One of trace debug info warn error fatal panic.
# See docs/logging.md for level descriptions.
# Note: If you used to use "notice", you should now use "info".
log_level = "info"
## Inputs ##
### plaintext Carbon ###
listen_addr = "0.0.0.0:2003"
# Close inbound plaintext connections if they've been idle for this long ("0s" to disable).
plain_read_timeout = "2m"
### Pickle Carbon ###
pickle_addr = "0.0.0.0:2013"
# Close inbound pickle connections if they've been idle for this long ("0s" to disable).
pickle_read_timeout = "2m"
## Validation of inputs ##
# You can also validate that each series has increasing timestamps.
validate_order = false
# How long to keep track of invalid metrics seen
# Useful time units are "s", "m", "h"
bad_metrics_max_age = "24h"
[[route]]
key = 'grafanaNet'
type = 'grafanaNet'
addr = '<metrics endpoint>'
apikey = '<Your Cloud Access Policy token>'
schemasFile = '/etc/carbon-relay-ng/storage-schemas.conf'
aggregationFile = '/etc/carbon-relay-ng/storage-aggregation.conf' # optional
## Instrumentation ##
[instrumentation]
# In addition to serving internal metrics via expvar, you can send them to graphite/carbon.
# IMPORTANT: Setting this to "" will disable flushing, and metrics will pile up and lead to OOM.
# See https://github.com/grafana/carbon-relay-ng/issues/50
# So for now you MUST send them somewhere. Sorry.
# (Also, the interval here must correspond to your setting in storage-schemas.conf if you use grafana hosted metrics.)
graphite_addr = "localhost:2003"
graphite_interval = 10000 # in msWe also have Grafana Labs engineers ready to advise further on set up, if needed.
2. Provide storage-schemas.conf and storage-aggregation.conf
The relay requires a [storage-schemas.conf](https://graphite.readthedocs.io/en/latest/config-carbon.html#storage-schemas-conf) file. This file describes your metrics resolution (interval) and rollups/retention.
Optionally, you can also provide a [storage-aggregation.conf](https://graphite.readthedocs.io/en/latest/config-carbon.html#storage-aggregation-conf) file. This file describes using which function the rollups (long-term downsampled data) should be aggregated to. Some values you may want to aggregate into averages (default), others you may want the minima and maxima, etc. If this file is not provided, default Grafana Cloud aggregations are applied instead.
If you already use Graphite, you can use your existing files as starting points, although you should add rules to cover the carbon-relay-ng stats themselves.
storage-schemas.conf
Here’s a default to get started. Note the additional rules for the carbon-relay-ng stats themselves.
[crng-service]
pattern = ^service_is_carbon-relay-ng
# interval should match graphite_interval in your relay configuration (default 10.000 ms)
retentions = 10s:1y
[crng-stats]
pattern = ^carbon-relay-ng\.stats
# interval should match graphite_interval in your relay configuration (default 10.000 ms)
retentions = 10s:1y
[default]
pattern = .*
retentions = 10s:8d,1m:60d,30m:2yMake sure your rules accurately describe your metrics. The carbon-relay-ng intervals should match the ‘graphite_interval’ config setting. This default assumes you are sending metrics at a 10s resolution, ie. 6 times per minute, which you may need to change.
storage-aggregation.conf
You can simply take your Graphite configuration file or use this to get started:
[default]
pattern = .*
xFilesFactor = 0.5
aggregationMethod = avgOnce these files are configured, you can send metrics to the relay in carbon/Graphite format in port 2003, and they should show up in Grafana.
Including carbon-relay-ng to an existing Graphite stack
If your Graphite stack does not currently contain any carbon-relays, then you can simply add carbon-relay-ng and have your clients (statsd, collectd, diamond, etc.) send data to it, which in turn can send data to your existing Graphite server and to our platform.
If you already have carbon-relays, you can choose from two options:
- Replace any existing carbon-relay with carbon-relay-ng.
- Run the carbon-relay-ng as an extra component in your existing pipeline, and configure it to transparently send a copy of your traffic to Grafana Cloud Graphite.
Use carbon-relay-ng as a replacement for carbon-relay or carbon-cache
Carbon-relay-ng is a drop-in replacement for carbon-relay for the most part. The only caveat is that the consistent-hashing implementation in carbon-relay-ng is different from the one in carbon-relay, so if you’re using consistent-hashing then switching from carbon-relay to carbon-relay-ng will re-distribute the metrics among the destinations.
Refer to this section for instructions on how to configure carbon-relay-ng to send data to your cloud instance.
Use carbon-relay-ng as an extra component in your existing pipeline
This method is useful if you want to send a copy of your data to Grafana Cloud Graphite with minimal changes to your existing Graphite infrastructure. There are two ways to do this:
- Add carbon-relay-ng in front of carbon-relay or carbon-cache, and configure it to mirror the traffic.
- Add a route to carbon-relay to mirror the traffic to carbon-relay-ng.
Duplicate traffic by adding carbon-relay-ng in front of carbon-relay or carbon-cache
carbon-relay-ng allows you to specify routes of various types. To send a copy to Grafana Cloud Graphite you need to add a route of the type grafanaNet. To send a copy to your existing carbon-relay/carbon-cache you can add a carbon route.
For example if you currently have carbon-relay listening on port 2003 and all of your infrastructure is sending its carbon traffic there, you could change it to listen on port 2053 instead and then start a carbon-relay-ng on port 2003 with this config to send a copy of the traffic to localhost:2053:
[[route]]
key = 'carbon'
type = 'sendAllMatch'
destinations = [
'localhost:2053 spool=true pickle=false'
]
[[route]]
key = 'grafanaNet'
type = 'grafanaNet'
addr = 'https://<Your endpoint address>'
apikey = '<Your Grafana.com User>:<Your Cloud Access Policy token>'
schemasFile = '/etc/carbon-relay-ng/storage-schemas.conf'
aggregationFile = '/etc/carbon-relay-ng/storage-aggregation.conf'Duplicate traffic by adding a route to carbon-relay
Use relay method rules
If you’re already using carbon-relay with the relay method rules, you can make it duplicate the traffic and send a copy to carbon-relay-ng. This can be done by using the flag continue in the rules file.
For example, if your current relay-rules.conf looks like this:
[default]
default = true
destinations = <original relay destination>:2003Then, you can use the continue keyword like this to duplicate the data and send a copy to carbon-relay-ng:
[carbonRelayNg]
pattern = .*
destinations = <carbon-relay-ng host>:2003
continue = True
[default]
default = true
destinations = <original relay destination>:2003Use relay method consistent-hashing
If you’re using the consistent-hashing relay method and the relay.REPLICATION_FACTOR setting is equal to the number of hosts in relay.DESTINATIONS, then you can make carbon-relay send another copy of your data to carbon-relay-ng by adding the carbon-relay-ng destination to relay.DESTINATIONS and increasing the relay.REPLICATION_FACTOR by 1. However, if you use a replication factor which is less than the number of destinations, then one of the other described methods to duplicate the traffic would need to be used.
For example if your current relay section looks like this:
[relay]
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 2
DESTINATIONS = <original relay destination 1>:2003,<original relay destination 2>:2003After adding the carbon-relay-ng destination and increasing the replication factor, the relay section would look like this:
[relay]
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 3
DESTINATIONS = <original relay destination 1>:2003,<original relay destination 2>:2003,<carbon relay ng>:2003High availability and scaling of carbon-relay-ng
Scaling with carbon-relay-ng
When distributing traffic among multiple instances of carbon-relay-ng it is important to ensure that the same metrics always get sent to the same carbon-relay-ng instances to preserve the order of the data points. It’s ok to have many carbon-relay-ng instances send data to GrafanaCloud concurrently.
|-------------------| |-------------------| |-------------------| |-------------------| |-------------------| |-------------------|
| metric producer 1 | | metric producer 2 | | metric producer 3 | | metric producer 4 | | metric producer 5 | | metric producer 6 |
|-------------------| |-------------------| |-------------------| |-------------------| |-------------------| |-------------------|
\ \ | | / /
\ \ | | / /
\ \ | | / /
\ \ | | / /
\ \ | | / /
\ |-------------------| |-------------------| /
\---------------| carbon-relay-ng-1 | | carbon-relay-ng-2 |----------------/
|-------------------| |-------------------|
\ /
\ /
|--------------|
| GrafanaCloud |
|--------------|Failure tolerance with carbon-relay-ng
As mentioned in the previous chapter, it is important that metrics get distributed among carbon-relay-ng instances in a consistent way to preserve the order of the datapoints. However, it is ok to run carbon-relay-ng in a hot / cold-standby setup to ensure that if the primary goes down the secondary would take over its load.
This setup gives you failure tolerance and scalability:
|-------------------| |-------------------| |-------------------| |-------------------| |-------------------| |-------------------|
| metric producer 1 | | metric producer 2 | | metric producer 3 | | metric producer 4 | | metric producer 5 | | metric producer 6 |
|-------------------| |-------------------| |-------------------| |-------------------| |-------------------| |-------------------|
\ / / \ / /
\ / / \ / /
\ / /-----------------/ \ / /------------------/
\ / / \ / /
\ <hot> / / <cold-standby> \ <hot> / / <cold-standby>
|-------------------| |-------------------| |-------------------| |-------------------|
| carbon-relay-ng-1 | | carbon-relay-ng-2 | | carbon-relay-ng-3 | | carbon-relay-ng-4 |
|-------------------| |-------------------| |-------------------| |-------------------|
\ \ / /
\ \ / /
\ \ / /
\ |-------------------| /
\-----------------------| GrafanaCloud |-----------------------/
|-------------------|Was this page helpful?
Related resources from Grafana Labs
