Cross-Account Logging: Shipping AWS Lambda Logs to OpenSearch

In today’s distributed systems, logging and monitoring play a crucial role in detecting anomalies and ensuring system health. AWS Lambda and OpenSearch are often paired to deliver efficient, scalable logging solutions. However, complexities can arise when these resources live in separate AWS accounts. This blog post will guide you through the process of sending AWS Lambda logs from Account A to an OpenSearch cluster in Account B using Terraform as the Infrastructure as Code (IAC) tool and GitLab for CI/CD pipelines.

Prerequisites

Two AWS accounts (Account A for AWS Lambda and Account B for OpenSearch)
A GitLab account with runners provisioned on AWS for CI/CD pipelines
Terraform installed on your system
Basic understanding of AWS Lambda, OpenSearch, AWS IAM, GitLab, and Terraform

Step 1: Configuring Your AWS Accounts

Before proceeding, ensure that both AWS accounts are correctly set up, and you have the necessary access permissions. We will also need an IAM role in Account B with the necessary permissions to access the OpenSearch cluster.

In Account A, create an AWS Lambda function. Be sure to attach the necessary IAM role and policy, enabling the function to create and write logs to AWS CloudWatch.

Step 2: Set Up OpenSearch in Account B

Next, in Account B, set up an OpenSearch cluster. The cluster should be correctly configured to receive logs from an external account. In addition, you should create an IAM role that allows the Log Stream from Account A to write to this cluster.

Step 3: Setting Up Terraform

Terraform is a powerful tool for managing infrastructure as code. With Terraform, you can define and provide data center infrastructure using a declarative configuration language.

Create a new Terraform project and define your AWS resources, including your Lambda function and OpenSearch cluster. Be sure to use the provider alias feature of Terraform to manage resources in multiple AWS accounts.

Remember to include the necessary resource blocks, such as aws_lambda_function and aws_elasticsearch_domain, depending on your requirements.

Step 4: Establishing Trust Relationship

Establish a trust relationship between Account A (Lambda) and Account B (OpenSearch) using an IAM role in Account B. This role should have a trust policy that allows Account A to assume it, and a permissions policy that allows writing to OpenSearch. Use the ‘aws_iam_role’ and ‘aws_iam_role_policy’ Terraform resources to create this.

Step 5: Configure AWS Lambda to Send Logs

With the trust relationship set up, configure your Lambda function in Account A to send logs to CloudWatch Logs. From there, use a CloudWatch Logs subscription filter to redirect these logs to the OpenSearch cluster in Account B.

The subscription filter should use the ARN of the IAM role in Account B, allowing it to forward logs across accounts.

Step 6: GitLab CI/CD

The final step involves integrating our setup with GitLab’s CI/CD. We will create a pipeline that uses AWS-provisioned runners to apply our Terraform configuration, creating or updating our infrastructure.

In your GitLab repository, define a .gitlab-ci.yml file that describes the stages of your pipeline, including steps for “init”, “validate”, “plan”, and “apply”. Your runners will execute these steps to deploy your infrastructure.

Conclusion

This post has covered a high-level guide on shipping AWS Lambda logs from one AWS account to an OpenSearch cluster in another. By integrating AWS services, Terraform, and GitLab, you can create a seamless logging solution that spans across multiple AWS accounts. Remember that while this setup works, it’s crucial to tailor it to your organization’s needs, considering factors like security, scale, and compliance. Happy logging!

Terraform and GitLab CI/CD Configuration

Below is a basic outline of Terraform and GitLab CI/CD configurations that facilitate the shipping of AWS Lambda logs from one AWS account to an OpenSearch cluster in another.

Terraform Configuration

# Account A - provider.tf
provider "aws" {
  region  = "us-west-2"
  profile = "accountA"
}

# Account A - lambda.tf
resource "aws_lambda_function" "example" {
  function_name = "example"

  filename         = "lambda_function_payload.zip"
  source_code_hash = filebase64sha256("lambda_function_payload.zip")
  handler          = "exports.test"
  role             = aws_iam_role.lambda_exec.arn
  runtime          = "nodejs12.x"

  publish = true
}

resource "aws_iam_role" "lambda_exec" {
  name = "lambda_exec_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_cloudwatch_log_group" "example" {
  name              = "/aws/lambda/${aws_lambda_function.example.function_name}"
  retention_in_days = 14
}

# Account B - provider.tf
provider "aws" {
  region  = "us-west-2"
  profile = "accountB"
  alias   = "accountB"
}

# Account B - opensearch.tf
provider "aws" {
  alias  = "accountB"
  region = "us-west-2"
}

resource "aws_elasticsearch_domain" "example" {
  provider                  = aws.accountB
  domain_name               = "example"
  elasticsearch_version     = "7.1"
}

resource "aws_iam_role" "cross_account_role" {
  name = "cross_account_role"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<AccountA_ID>:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy" "policy" {
  name = "cross_account_policy"
  role = aws_iam_role.cross_account_role.id

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "es:*"
      ],
      "Resource": "${aws_elasticsearch_domain.example.arn}/*"
    }
  ]
}
POLICY
}

# GitLab - .gitlab-ci.yml
stages:
  - init
  - validate
  - plan
  - apply

variables:
  AWS_DEFAULT_REGION: "us-west-2"
  AWS_ACCESS_KEY_ID: "<access_key>"
  AWS_SECRET_ACCESS_KEY: "<secret_key>"

init:
  stage: init
  script:
    - terraform init

validate:
  stage: validate
  script:
    - terraform validate

plan:
  stage: plan
  script:
    - terraform plan

apply:
  stage: apply
  script:
    - terraform apply -auto-approve

Cross-Account Logging: Shipping AWS Lambda Logs to OpenSearch#

Prerequisites#

Step 1: Configuring Your AWS Accounts#

Step 2: Set Up OpenSearch in Account B#

Step 3: Setting Up Terraform#

Step 4: Establishing Trust Relationship#

Step 5: Configure AWS Lambda to Send Logs#

Step 6: GitLab CI/CD#

Conclusion#

Terraform and GitLab CI/CD Configuration#

Terraform Configuration#