AWS CloudWatch is helpful for storing and retrieving logs for various AWS resources. It tells us what is really happening with our infrastructure in the form of logs, metrics, alarms, and dashboards. Sometimes, there is a need to perform some action if something occurs within a resource e.g. an error or warning log. For this kind of work, we can use a native feature of AWS CloudWatch called Subscription Filter.

What is a Subscription Filter?

Subscription filters allow you to define rules for forwarding log events from AWS CloudWatch log groups to different AWS services such as Amazon Kinesis, AWS Lambda, or an Amazon OpenSearch service. This feature is useful for processing and analyzing log data in real time or for triggering specific actions based on log events.

Types of Subscription Filters

AWS supports different AWS services as subscription filters, which are as follows:

• AWS OpenSearch
• AWS Kinesis
• AWS Kinesis Firehose
• AWS Lambda

You can use any of the above services as your filter, depending on your use case e.g, data processing, storage, notifications etc. However, please note that you can use up to two filters for a single log group.

Use Case

For our use case, we will be using the AWS Lambda function as a destination for the subscription filter. AWS CloudWatch log group will forward the log events, matching a specific pattern and trigger the lambda function accordingly.

approach

• In the AWS management console, go to AWS CloudWatch service and select Log Groups.
• Create a new log group or choose an existing one.

• Once the log group has been created, open the log group and go to Subscription filters.

• On the right side, click on Create and choose Create Lambda subscription filter.

• If you have a lambda function ready, select that, otherwise, create a new Lambda function beforehand. make sure to include the CloudWatch permissions in the function’s default execution IAM role.

• If you are unsure about the log format, choose Other. Provide a filter pattern which will check the log stream for that specific matching pattern only.
• The filtered log data, when sent to the lambda function will be encoded. You need to decode or parse it in order to use it for your scenario. Here is an example code written in Python language to decode the data:

import gzip
import base64
def lambda_handler(event, context):
    encoded_zipped_data = event['awslogs']['data']
    zipped_data = base64.b64decode(encoded_zipped_data)
    data = gzip.decompress(zipped_data)

In the above code snippet, the decoded data is stored in the data variable. Now, you can parse it and get the desired response data and perform your custom operations accordingly.

• When the lambda function is ready, use it in the Subscription filter page.
• Once done with all the sections, click on Start streaming.
• Now, it’s time to create a test log stream. In the selected log group, got to Log Streams and select Create log stream.

• A popup window will appear, asking you to provide a name for the log stream. Give it a name and select Create.
• You can see you log stream in the Log Streams section.

• Open the log stream you just created, go to Actions and select Create log event.

• A popup will appear. Enter the pattern as this is a test error log and click on Create.

• As soon as this log event gets created, it will invoke our Lambda function, based on the log pattern that we specified, decode the log data and perform your operations accordingly. Your lambda function could include some code for sending AWS SNS notifications that a certain error log has occurred and sending an email to the concerned individual or team.

Conclusion

By using AWS CloudWatch Subscription filters, you can automate the processes within your AWS ecosystem based on log events and can monitor changes more closely. This will automate the error log checking more efficiently and will perform error log checks for you.