iTranslated by AI
Using Terraform to Notify CloudWatch Logs Errors via Email and Slack
In this article, I will implement a setup using Terraform to notify application error logs output to CloudWatch Logs via Slack and Email.
Prerequisites
This article focuses on using Terraform to set up Email and Slack notifications, assuming the environment is already configured to output logs to CloudWatch Logs.
It serves as a continuation of this previous article:
What was done in the previous article:
- Running a very simple application on ECS (Fargate execution environment).
- An ECS task includes a Fluent Bit container for FireLens as a sidecar. It streams application standard output logs to S3, while sending error logs to CloudWatch Logs.
The architecture diagram is as follows:
What we will do
- Create an SNS topic for Email notifications.
- Add a subscription filter to CloudWatch Logs to trigger a Lambda function.
- Trigger the SNS topic from the Lambda function to send emails and Slack notifications.
What we will not do
- Managing the Lambda function code in a separate repository.
Notes
Regarding Slack notifications, although I used a Lambda function this time, it seems possible to achieve this without writing code by using AWS Chatbot.
Unfortunately, as of 2022/1/28, it cannot be handled using standard methods in Terraform (though there seem to be modules that utilize CloudFormation).
Therefore, I have implemented it with a Lambda function this time!
Implementation Details
The directory structure of the module this time is as follows.
I haven't included the contents of modules like network here, but the code is available in the following repository.
**.
├── README.md
├── main.tf
├── modules
│ ├── network
│ │ ├── main.tf
│ │ ├── output.tf
│ │ └── variables.tf
│ └── web_app
│ ├── alb.tf
│ ├── ecr.tf
│ ├── ecs.tf
│ ├── files
│ │ └── ecs_task_assume_policy.json
│ ├── fluentbit
│ │ ├── Dockerfile
│ │ └── extra.conf
│ ├── iam.tf
│ ├── lambda
│ │ ├── app_log_notificate
│ │ │ └── index.py
│ │ └── sns_publish.zip
│ ├── lambda.tf
│ ├── log.tf
│ ├── output.tf
│ ├── security_group.tf
│ ├── variables.tf
│ └── vpc_endpoint.tf
├── provider.tf
├── terraform.tfstate
├── terraform.tfvars
└── variables.tf
Additionally, you need to issue a Slack webhook_url in advance.
Please obtain it beforehand by referring to articles like this and specify it in tfvars.
Implementation of Terraform code
Creating the SNS topic
# Create SNS topic for email notifications
resource "aws_sns_topic" "notificate" {
name = "${var.project}-error-notification"
}
# Pass notification emails as an array like ["hoge@gmail.com", "fuga@gmail.com"]
resource "aws_sns_topic_subscription" "topic_email_subscription" {
for_each = { for email in var.emails : email => email }
topic_arn = aws_sns_topic.notificate.arn
protocol = "email"
endpoint = each.value
}
Preparing the CloudWatch Logs subscription filter
# Subscription to Lambda for delivering error logs to the SNS topic
resource "aws_cloudwatch_log_subscription_filter" "error_log_subscription" {
name = "${var.project}-error-subscription"
log_group_name = aws_cloudwatch_log_group.app.id
filter_pattern = "ERROR"
destination_arn = aws_lambda_function.sns_publish_func.arn
}
Preparing the Lambda function
# Package the Lambda function that publishes error logs to SNS
data "archive_file" "sns_publish_func" {
type = "zip"
source_dir = "${path.module}/lambda"
output_path = "${path.module}/lambda/sns_publish.zip"
}
resource "aws_lambda_function" "sns_publish_func" {
filename = data.archive_file.sns_publish_func.output_path
function_name = "${var.project}_sns_publish"
role = aws_iam_role.lambda_sns_publish.arn
handler = "app_log_notificate.index.lambda_handler"
source_code_hash = data.archive_file.sns_publish_func.output_base64sha256
runtime = "python3.9"
environment {
variables = {
SNS_TOPIC_ARN = aws_sns_topic.notificate.arn
ALARM_SUBJECT = "【Error Notification】${var.project}"
WEB_HOOK_URL = var.slack_webhook_url
}
}
depends_on = [
aws_cloudwatch_log_group.sns_publish_func,
aws_iam_role.lambda_sns_publish
]
}
# Allow execution from CloudWatch Logs
resource "aws_lambda_permission" "log_permission" {
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.sns_publish_func.arn
principal = "logs.ap-northeast-1.amazonaws.com"
source_arn = "${aws_cloudwatch_log_group.app.arn}:*"
}
# CloudWatch Logs log group for Lambda function log output
resource "aws_cloudwatch_log_group" "sns_publish_func" {
name = "/aws/lambda/${var.project}_sns_publish"
retention_in_days = 14
}
IAM role for the Lambda function
#####################
# IAM role for the Lambda function that publishes notifications to SNS
#####################
resource "aws_iam_role" "lambda_sns_publish" {
name = "${var.project}-publish-sns"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_iam_role_policy" "lambda_sns_publish" {
name = "${var.project}-publish-sns"
role = aws_iam_role.lambda_sns_publish.id
policy = <<EOF
{
"Statement": [
{
"Sid": "",
"Action": [
"SNS:Publish"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Sid": "",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*",
"Effect": "Allow"
}
]
}
EOF
}
Implementation of the Lambda function
It's a bit redundant, I'm embarrassed...
I've included the code that works for now.
import base64
import json
import zlib
import os
from urllib import request
import boto3
print('Loading function')
def lambda_handler(event, context):
data = zlib.decompress(base64.b64decode(
event['awslogs']['data']), 16+zlib.MAX_WBITS)
data_dict = json.loads(data)
log_entire_json = json.loads(json.dumps(
data_dict["logEvents"], ensure_ascii=False))
log_entire_len = len(log_entire_json)
region = context.invoked_function_arn.split(":")[3]
log_group_name = context.log_group_name
log_stream_name = context.log_stream_name
log_url = "https://"+region+".console.aws.amazon.com/cloudwatch/home?region="+region+"#logEvent:group="+log_group_name+";stream="+log_stream_name
post_sns_topic(log_entire_len, data_dict, log_url)
post_to_slack(log_entire_len, data_dict, log_url)
def post_sns_topic(log_entire_len: int, data_dict: dict, log_url:str):
for i in range(log_entire_len):
log_dict = json.loads(json.dumps(
data_dict["logEvents"][i], ensure_ascii=False))
try:
message_str = log_dict["message"]
message_dict = json.loads(message_str)
message_dict['log_url'] = log_url
message = json.dumps(message_dict)
sns = boto3.client('sns')
# SNS Publish
publishResponse = sns.publish(
TopicArn=os.environ['SNS_TOPIC_ARN'],
Message=message,
Subject=os.environ['ALARM_SUBJECT']
)
except Exception as e:
print("[sns_notice_exception: ]" + str(e))
def post_to_slack(log_entire_len: int, data_dict: dict, log_url:str):
for i in range(log_entire_len):
log_dict = json.loads(json.dumps(
data_dict["logEvents"][i], ensure_ascii=False))
try:
message_str = log_dict["message"]
message_dict = json.loads(message_str)
message_dict['log_url'] = log_url
message = json.dumps(message_dict)
send_data = {
"username": "notificate_bot",
"icon_emoji": "aws",
"text": message,
}
r = request.Request(
url=os.environ['WEB_HOOK_URL'],
data=json.dumps(send_data).encode('utf-8')
)
with request.urlopen(r) as response:
response_body = response.read().decode('utf-8')
print(response_body)
except Exception as e:
print("[slack_notice_exection: ]" + str(e))
Results
By intentionally triggering an error on the application side...
A notification arrives in Slack...
And there was a notification in the email too!
Since the CloudWatch Logs log stream URL is included in both, you can check the error logs with a single click.
When using it in practice, I think further implementation for formatting the message text, etc., would be necessary...
Discussion