Collect logs with Promtail

The Grafana Cloud stack includes a logging service powered by Grafana Loki, a Prometheus-inspired log aggregation system. This means that you are not required to run your own Loki environment, though you can ship logs to Grafana Cloud using Promtail or another supported client if you maintain a self-hosted Loki environment. If you want to use Grafana Agent instead of Promtail to send logs, refer to Collect logs with Grafana Agent.

Before you begin

To follow the steps in this guide, you need the following:

A Grafana Cloud account
An application or system generating logs
Docker (optional)

Install Promtail

Promtail is an agent that ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed monitoring.

Promtail primarily:

Discovers targets.
Attaches labels to log streams.
Pushes them to the Loki instance.

Currently, Promtail can tail logs from two sources: local log files and the systemd journal (on AMD64 machines only).

For the options available to install Promtail, see Install Promtail.

Configure Promtail to send logs

There are two options described in this guide for configuring Promtail:

Send logs from a standalone host
Send logs from a Kubernetes cluster

Option 1: Send logs from a standalone host

To create a Promtail configuration file for sending logs from a standalone host:

Log in to your Grafana Cloud environment and click on Connections from the Home menu located near the top left-hand side of the page.
This will bring you to the Connect Data page. From here, click Custom data and select Hosted logs.

Select the option for a standalone host and name the token that this integration will use.

Scroll down to review the configuration file that has been created for you. The example below is a Linux YAML file called config.yaml and saved in /etc/promtail/.

server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: https://$InstanceID:$GrafanaCloudToken@$logs-endpoint.grafana.net/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*.log

Option 2: Send logs from a Kubernetes cluster

To create a Promtail configuration file for sending logs from a Kubernetes cluster:

Log in to your Grafana Cloud environment and click on Connections from the Home menu located near the top left-hand side of the page.
This will bring you to the Connect Data page. From here, click Custom data and select Hosted logs.
Select the option for a Kubernetes cluster and name the token that this integration will use:

Scroll down to review the cURL command that you will use to create the configuration file. An example of the install script is shown below:

curl -fsS https://raw.githubusercontent.com/grafana/loki/main/tools/promtail.sh | sh -s <$InstanceID> <$Your Grafana Cloud Token> $Your logs-endpoint.grafana.net default | kubectl apply --namespace=default -f  -

Confirm logs are being ingested into Grafana Cloud

Within several minutes, logs should begin to be available in Grafana Cloud. To test this, use the Explore feature.

To confirm that logs are being sent to Grafana Cloud:

Click on Explore from the left-hand side menu.
At the top left-hand side of the page, use the dropdown menu to select your Loki logs data source. This should be named grafanacloud-$yourstackname-logs.
Next, set the time to “Last 5 minutes” in the time selector near the top right-hand side of the Explore page.
From here, you can either enter a Loki query or use the Label browser to review detected log labels.

If log labels do not appear in the Label browser, logs have not been received for the time range selected. This likely means that logs have not being received from your environment. If the Label browser does display results, you can click on the log labels to explore the logs further.

If logs are not displayed after several minutes, ensure Promtail is running and check your steps for typos. See the Troubleshooting Promtail guide for troubleshooting tips.

In addition to the Label browser feature, the Explore user interface also supports autocomplete options for queries. If you are new to Loki, we recommend reviewing the LogQL documentation below:

Query logs and create panels

Once you have Promtail up and running in your environment, allow some time to start collecting logs. Eventually, you will be able to query logs and create panels inside dashboards using Loki as a data source.

Querying logs is done using LogQL which can be used in both Explore and when creating dashboard panels.

For examples and feature showcases, check out play.grafana.org for ideas and inspiration.