Documentation Grafana Cloud Monitor infrastructure Cloud Provider Observability AWS Observability AWS logs Logs with Lambda Configure ALB access logs

Configure Application Load Balancer Logs

Send Application Load Balancer access logs to Grafana Cloud Logs using a Lambda function.

Whether you configure with CloudFormation or Terraform, the following diagram shows the components required and how the process works:

Complete the following steps to create the Lambda function.

Navigate to Application Load Balancer Logs

1. Navigate to your Grafana Cloud portal.

2. In your Grafana Cloud stack, expand Infrastructure in the main menu.

3. Click AWS, then click Add your AWS services.

4. At the Configuration page, find and click the Application Load Balancer Logs tile.

5. On the Application Load Balancer Logs page, click Configure.

6. Perform subsequent steps to configure with:

   • Cloud Formation
   • Terraform

Configure with CloudFormation

Complete the following process to configure with CloudFormation.

Before you begin

To use Application Load Balancer Logs, you must have:

• Configured ALB logs to be written to an S3 bucket in the same region where you want the lambda-promtail function to run
• Created an S3 bucket in the same region to receive the Lambda image

Choose a method for creating AWS resources

Click Use CloudFormation.

Upload the lambda-promtail zip file to S3

To upload the compressed lambda-promtail file to the S3 bucket in your region, copy the code in the GUI, and run the command.

Create a Grafana.com access token

Create a Grafana.com access token with the necessary permissions to authenticate with Grafana Cloud Logs.

1. In the API token name box, enter your API token name.
2. Click Create token to generate the token.
3. Copy the generated API key.

Enable AWS EventBridge

Perform one of these two methods to enable AWS EventBridge so that the lambda-promtail function can respond to access log files being added to the AWS S3 bucket:

• Follow the Amazon EventBridge guide.
• Copy the code available in the GUI and and paste it into your CloudFormation file.

Launch CloudFormation stack

1. Click Launch stack to navigate to AWS.
2. Run the CloudFormation setup, and specify the S3 bucket where the ALB logs are located so they can be forwarded to Grafana Cloud.

Explore logs

1. Return to the Grafana Configuration Details page, and click Go to Explore.
2. At the Explore page, use the following query to find all the load balancing access logs:

plain Copy

{__aws_log_type=~"s3_lb"}

Configure with Terraform

Complete the following process to configure with Terraform.

Choose a method for creating AWS resources

Click Use Terraform.

Create a Grafana.com API key

1. In the API token name box, enter your API token name.
2. Click Create token to generate the token.
3. Copy the generated API key.

Terraform setup

Configure the AWS CLI. Remember to set the correct AWS region where lambda-promtail should run and pull logs from.

1. Copy and paste the following snippet into a Terraform file.

   terraform Copy

   terraform {
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = "~> 4.0"
       }
     }
   }
   
   provider "aws" {
     region = "us-west-2"
   }
   
   resource "aws_s3_object_copy" "lambda_promtail_zipfile" {
     bucket = var.s3_bucket
     key    = var.s3_key
     source = "grafanalabs-cf-templates/lambda-promtail/lambda-promtail.zip"
   }
   
   resource "aws_iam_role" "lambda_promtail_role" {
     name = "GrafanaLabsALBLogsIntegration"
   
     assume_role_policy = jsonencode({
       "Version" : "2012-10-17",
       "Statement" : [
         {
           "Action" : "sts:AssumeRole",
           "Principal" : {
             "Service" : "lambda.amazonaws.com"
           },
           "Effect" : "Allow",
         }
       ]
     })
   }
   
   resource "aws_iam_role_policy" "lambda_promtail_policy_alb_logs" {
     name = "alb-logs"
     role = aws_iam_role.lambda_promtail_role.name
     policy = jsonencode({
       "Statement" : [
         {
           "Action" : [
             "logs:CreateLogGroup",
             "logs:CreateLogStream",
             "logs:PutLogEvents",
           ],
           "Effect" : "Allow",
           "Resource" : "arn:aws:logs:*:*:*",
         },
         {
           "Action" : [
             "s3:GetObject",
           ],
           "Effect" : "Allow",
           "Resource" : format("arn:aws:s3:::%s/*", var.access_logs_s3_bucket),
         }
       ]
     })
   }
   
   resource "aws_lambda_function" "lambda_promtail" {
     function_name = "GrafanaCloudLambdaPromtail"
     role          = aws_iam_role.lambda_promtail_role.arn
   
     timeout     = 60
     memory_size = 128
   
     handler   = "main"
     runtime   = "provided.al2023"
     s3_bucket = var.s3_bucket
     s3_key    = var.s3_key
   
     environment {
       variables = {
         WRITE_ADDRESS = var.write_address
         USERNAME      = var.username
         PASSWORD      = var.password
         BATCH_SIZE    = var.batch_size
         EXTRA_LABELS  = var.extra_labels
       }
     }
   
     depends_on = [
       aws_s3_object_copy.lambda_promtail_zipfile,
       aws_iam_role_policy.lambda_promtail_policy_alb_logs,
     ]
   }
   
   resource "aws_lambda_function_event_invoke_config" "lambda_promtail_invoke_config" {
     function_name          = aws_lambda_function.lambda_promtail.function_name
     maximum_retry_attempts = 2
   }
   
   // The permission below will allow S3 to trigger lambda-promtail, upon files being added
   resource "aws_lambda_permission" "lambda_promtail_allow_s3" {
     statement_id  = "lambda-promtail-allow-s3"
     action        = "lambda:InvokeFunction"
     function_name = aws_lambda_function.lambda_promtail.function_name
     principal     = "s3.amazonaws.com"
   }
   
   // Configures the access logs S3 bucket to emit a notification to lambda-promtail, when a file
   // is added.
   resource "aws_s3_bucket_notification" "bucket_notification" {
     bucket = var.access_logs_s3_bucket
   
     lambda_function {
       lambda_function_arn = aws_lambda_function.lambda_promtail.arn
       events              = ["s3:ObjectCreated:*"]
     }
   
     depends_on = [
       aws_lambda_permission.lambda_promtail_allow_s3,
     ]
   }
   
   output "role_arn" {
     value       = aws_lambda_function.lambda_promtail.arn
     description = "The ARN of the Lambda function that runs lambda-promtail."
   }

2. Copy and paste the following snippet into a variables.tf file.

   terraform Copy

   variable "write_address" {
     type        = string
     description = "This is the Grafana Cloud Loki URL that logs will be forwarded to."
     default     = ""
   }
   
   variable "username" {
     type        = string
     description = "The basic auth username for Grafana Cloud Loki."
     default     = ""
   }
   
   variable "password" {
     type        = string
     description = "The basic auth password for Grafana Cloud Loki (your Grafana.com API Key)."
     sensitive   = true
     default     = ""
   }
   
   variable "s3_bucket" {
     type        = string
     description = "The name of the bucket where to upload the 'lambda-promtail.zip' file."
     default     = ""
   }
   
   variable "s3_key" {
     type        = string
     description = "The desired path where to upload the 'lambda-promtail.zip' file (defaults to the root folder)."
     default     = "lambda-promtail.zip"
   }
   
   variable "extra_labels" {
     type        = string
     description = "Comma separated list of extra labels, in the format 'name1,value1,name2,value2,...,nameN,valueN' to add to entries forwarded by lambda-promtail."
     default     = ""
   }
   
   variable "batch_size" {
     type        = string
     description = "Determines when to flush the batch of logs (bytes)."
     default     = ""
   }
   
   variable "access_logs_s3_bucket" {
       type = string
       description = "The S3 buckets where ALB access logs are stored"
       default = ""
   }

3. Return to the Grafana Configuration Details page.

4. Copy and paste the write_address, username, and password into the appropriate places in the variables.tf file.

5. In the variables.tf file, configure variables according to their descriptions. Note that the Access Logs S3 bucket must be in the same regions where the Lambda is deployed.

6. Run the Terraform apply command:

   bash Copy

    terraform apply -var-file="variables.tf"

   After the Terraform apply command has finished creating the resources, it outputs the role_arn of the Lambda function that runs lambda-promtail.

The previous Terraform snippets should get you started with a basic configuration for lambda-promtail. For additional setup (specifically, VPC subnets and security groups), refer to this extended example Terraform file, for the comments within the Terraform snippet.

Explore logs

1. Return to the Grafana Configuration Details page, and click Go to Explore.
2. At the Explore page, use the following query to find all the load balancing access logs:

plain Copy

{__aws_log_type=~"s3_lb"}

Labels

ALB access logs sent to Grafana Cloud Logs include the following special labels assigned to them:

• __aws_s3_lb: Name of the Application Load Balancer that generated the logs
• __aws_s3_lb_owner: AWS account ID of the owner of this log

You can specify extra labels (as key-value pairs) to be added to logs streamed by lambda-promtail: __extra_<name>=<value>. Add these labels in the CloudFormation parameter ExtraLabels, with the format label1,value1,label2,value2.