Loki tutorial: How to set up Promtail on AWS EC2 to find and analyze your logs
Amazon’s Elastic Compute Cloud (AWS EC2) is one of the most popular ways to run applications in the cloud, but finding logs for a given instance is a common struggle. That’s where Loki can help. With Loki aggregation, you can group all your logs from all your virtual machines in one place, and with its search capabilities, you can quickly find and analyze them. It’s a great way to gain visibility in your cloud deployment.
This tutorial explains how you can set up the Promtail agent on an AWS EC2 instance and configure it to send all its logs to a Loki instance, so you can start getting the most out of your workload.
Requirements
Before we start you’ll need:
- An AWS account (with the
AWS_ACCESS_KEY
andAWS_SECRET_KEY
). - A VPC that is routable from the internet. (Follow these instructions if you need to create one.)
- A SSH public key. (Follow these instructions if you need a new one.)
- The AWS CLI configured (run
aws configure
). - A Grafana instance with a Loki data source already configured.
For the sake of simplicity we’ll use a Grafana Cloud Loki and Grafana instance, but all the steps are the same if you’re running your own Loki and Grafana instance. (You can get a 30-day trial of Grafana Cloud Loki here.)
To make it easy to learn, all of the following instructions are manual. However, in a real setup we recommend that you use provisioning tools such as Terraform, CloudFormation, Ansible, or Chef.
Creating an EC2 instance
As a first step we’re going to import our SSH key to AWS so that we can SSH to our future EC2 instance. Let’s run our first command:
aws ec2 import-key-pair --key-name "promtail-ec2" --public-key-material fileb://~/.ssh/id_rsa.pub
Next we’re going to create a security group. Be sure to note the group id for the following command:
aws ec2 create-security-group --group-name promtail-ec2 --description "promtail on ec2" --vpc-id vpc-668d120f
{
"GroupId": "sg-02c489bbdeffdca1d"
}
Now let’s authorize inbound access for SSH and Promtail server:
aws ec2 authorize-security-group-ingress --group-id sg-02c489bbdeffdca1d --protocol tcp --port 22 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id sg-02c489bbdeffdca1d --protocol tcp --port 3100 --cidr 0.0.0.0/0
You don’t need to open those ports to all IPs. As shown above, you can use your own IP range.
We’re going to create an Amazon Linux 2 instance, as this is one of the most popular, but feel free to use the AMI of your choice.
To create the instance, use the following command. Make sure you note the instance id:
aws ec2 run-instances --image-id ami-016b213e65284e9c9 --count 1 --instance-type t2.micro --key-name promtail-ec2 --security-groups promtail-ec2
To make it more interesting later, let’s tag (Name=promtail-demo
) our instance:
aws ec2 create-tags --resources i-041b0be05c2d5cfad --tags Key=Name,Value=promtail-demo
Tags enable you to categorize your AWS resources in different ways; for example, by purpose, owner, or environment. This is useful when you have many resources of the same type – you can quickly identify a specific resource based on the tags that you’ve assigned to it.
Finally let’s grab the public DNS of our instance:
aws ec2 describe-instances --filters "Name=tag:Name,Values=promtail-demo" --query "Reservations[].Instances[].NetworkInterfaces[].Association.PublicDnsName"
and start an SSH session:
ssh ec2-user@ec2-13-59-62-37.us-east-2.compute.amazonaws.com
Setting up Promtail
First let’s make sure we’re running as root by using sudo -s
.
Next we’ll download, install, and give executable rights to Promtail.
mkdir /opt/promtail && cd /opt/promtail
curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/promtail-linux-amd64.zip"
unzip "promtail-linux-amd64.zip"
chmod a+x "promtail-linux-amd64"
Now we’re going to download the promtail configuration file below and edit. Don’t worry, we will explain what it means.
curl https://raw.githubusercontent.com/grafana/loki/master/docs/sources/clients/aws/ec2/promtail-ec2.yaml > ec2-promtail.yaml
vi ec2-promtail.yaml
The server section indicates that Promtail will bind its http server to 3100. Promtail serves HTTP pages for troubleshooting service discovery and targets.
The clients section allows you to target your Loki instance. If you’re using Grafana Cloud, simply replace <user id>
and <api secret>
with your credentials. Otherwise just replace the whole URL with your custom Loki instance (e.g., http://my-loki-instance.my-org.com/loki/api/v1/push
).
Promtail uses the same Prometheus scrape_configs. This means if you already own a Prometheus instance, the config will be very similar and easy to grasp.
Since we’re running on AWS EC2, we want to use EC2 service discovery. This will allow us to scrape metadata about the current instance (and even your custom tags) and attach those to our logs. This way, managing and querying on logs will be much easier.
Make sure to replace accordingly your current region
, access_key
and secret_key
; alternatively you can use an AWS Role ARN. For more information about this, see the ec2_sd_config
documentation.
Finally, the relabeling_configs
section has three purposes:
- Selecting the discovered labels that you want to attach to your targets. In our case here we’re keeping
instance_id
as instance, the tagName
as name, and thezone
of the instance. Make sure to check out the Prometheus documentation for the full list of available labels. - Choosing where Promtail should find the log. In our example, we want to include all log files that exist in
/var/log
using the glob/var/log/**.log
. If you need to use multiple globs, you simply need to add another job. - Ensuring discovered targets are only for the machine Promtail currently runs on. This is achieved by adding the label
__host__
using the incoming metadata__meta_ec2_private_dns_name
. If it doesn’t match the current HOSTNAME environment variable, the target will be dropped.
All right, we should be ready to fire up Promtail! We’re going to run it using the flag --dry-run
. This is to ensure that everything is correct, especially when you’re still playing around with the configuration. Don’t worry when using this mode; Promtail won’t send any logs and will remember any file positions.
./promtail-linux-amd64 -config.file=./ec2-promtail.yaml --dry-run
If everything is going well, you should see a log indicating line that would have been sent to the Loki instance with its discovered labels as shown below:
2020-07-08T14:51:38 {filename="/var/log/cloud-init.log", instance="i-041b0be05c2d5cfad", name="promtail-demo", zone="us-east-2c"} Jul 07 21:37:24 cloud-init[3035]: util.py[DEBUG]: loaded blob returned None, returning default.
If you want to see the existing targets and available labels, you can reach Promtail server using the public DNS assigned to your instance:
open http://ec2-13-59-62-37.us-east-2.compute.amazonaws.com:3100/
For example, the page below is the service discovery page. It shows you all discovered targets, with their respective available labels, and if any were dropped, the reason why.
Configuring Promtail as a service
Now that we have correctly configured Promtail, we usually want to make sure it runs as a systemd service, so it can automatically restart on failure or when the instance restarts.
Let’s create a new service using vim /etc/systemd/system/promtail.service
and copy the service definition below:
[Unit]
Description=Promtail
[Service]
User=root
WorkingDirectory=/opt/promtail/
ExecStartPre=/bin/sleep 30
ExecStart=/opt/promtail/promtail-linux-amd64 --config.file=./ec2-promtail.yaml
SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Let’s reload the systemd, enable, then start the Promtail service:
systemctl daemon-reload
systemctl enable promtail.service
systemctl start promtail.service
You can verify that the service is running correctly by using the following command:
systemctl status promtail.service -l
● promtail.service - Promtail
Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2020-07-08 15:48:57 UTC; 4s ago
Main PID: 2732 (promtail-linux-)
CGroup: /system.slice/promtail.service
└─2732 /opt/promtail/promtail-linux-amd64 --config.file=./ec2-promtail.yaml
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal systemd[1]: Started Promtail.
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal systemd[1]: Starting Promtail...
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=warn ts=2020-07-08T15:48:57.559085451Z caller=filetargetmanager.go:98 msg="WARNING!!! entry_parser config is deprecated, please change to pipeline_stages"
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=info ts=2020-07-08T15:48:57.559869071Z caller=server.go:179 http=[::]:3100 grpc=[::]:35127 msg="server listening on addresses"
Jul 08 15:48:57 ip-172-31-45-69.us-east-2.compute.internal promtail-linux-amd64[2732]: level=info ts=2020-07-08T15:48:57.56029474Z caller=main.go:67 msg="Starting Promtail" version="(version=1.5.0, branch=HEAD, revision=12c7eab8)"
You can now verify in Grafana that Loki has correctly received your instance logs using for instance the LogQL query {zone="us-east-2"}
.
Sending systemd logs
Just like we did with Promtail, you’ll most likely manage your applications with systemd, which usually stores applications logs in journald. Promtail actually supports scraping logs from journald, so let’s configure it.
We will edit our previous config (vi ec2-promtail.yaml
) and add the following block in the scrape_configs
section.
- job_name: journal
journal:
json: false
max_age: 12h
path: /var/log/journal
labels:
job: systemd-journal
relabel_configs:
- source_labels: ['__journal__systemd_unit']
target_label: 'unit'
Note that you can use relabeling to convert systemd labels to match what you want. Finally, make sure that the path of journald logs is correct; it might be different on some systems.
That’s it! Save the config, and you can reboot
the machine (or simply restart the service systemctl restart promtail.service
).
Let’s head back to Grafana and verify that your Promtail logs are available in Grafana by using the LogQL query {unit="promtail.service"}
in Explore. Finally, make sure to checkout live tailing to see logs appearing as they are ingested in Loki.