AWS CloudWatch is helpful for storing and retrieving logs for various AWS resources. It tells us what is really happening with our infrastructure in the form of logs, metrics, alarms, and dashboards. Sometimes, there is a need to perform some action if something occurs within a resource e.g. an error or warning log. For this kind of work, we can use a native feature of AWS CloudWatch called Subscription Filter.
What is a Subscription Filter?
Subscription filters allow you to define rules for forwarding log events from AWS CloudWatch log groups to different AWS services such as Amazon Kinesis, AWS Lambda, or an Amazon OpenSearch service. This feature is useful for processing and analyzing log data in real time or for triggering specific actions based on log events.
Types of Subscription Filters
AWS supports different AWS services as subscription filters, which are as follows:
- AWS OpenSearch
- AWS Kinesis
- AWS Kinesis Firehose
- AWS Lambda
You can use any of the above services as your filter, depending on your use case e.g, data processing, storage, notifications etc. However, please note that you can use up to two filters for a single log group.
For our use case, we will be using the AWS Lambda function as a destination for the subscription filter. AWS CloudWatch log group will forward the log events, matching a specific pattern and trigger the lambda function accordingly.
- In the AWS management console, go to AWS CloudWatch service and select
- Create a new log group or choose an existing one.
- Once the log group has been created, open the log group and go to
- On the right side, click on
Create Lambda subscription filter.
- If you have a lambda function ready, select that, otherwise, create a new Lambda function beforehand. make sure to include the CloudWatch permissions in the function’s default execution IAM role.
- If you are unsure about the log format, choose
Other. Provide a
filter patternwhich will check the log stream for that specific matching pattern only.
- The filtered log data, when sent to the lambda function will be encoded. You need to decode or parse it in order to use it for your scenario. Here is an example code written in Python language to decode the data:
import gzip import base64 def lambda_handler(event, context): encoded_zipped_data = event['awslogs']['data'] zipped_data = base64.b64decode(encoded_zipped_data) data = gzip.decompress(zipped_data)
In the above code snippet, the decoded data is stored in the
data variable. Now, you can parse it and get the desired response data and perform your custom operations accordingly.
- When the lambda function is ready, use it in the
- Once done with all the sections, click on
- Now, it’s time to create a test
log stream. In the selected log group, got to
Log Streamsand select
Create log stream.
- A popup window will appear, asking you to provide a name for the log stream. Give it a name and select
- You can see you log stream in the
- Open the log stream you just created, go to
Create log event.
- A popup will appear. Enter the pattern as
this is a test error logand click on
- As soon as this log event gets created, it will invoke our Lambda function, based on the log pattern that we specified, decode the log data and perform your operations accordingly. Your lambda function could include some code for sending AWS SNS notifications that a certain error log has occurred and sending an email to the concerned individual or team.
By using AWS CloudWatch Subscription filters, you can automate the processes within your AWS ecosystem based on log events and can monitor changes more closely. This will automate the error log checking more efficiently and will perform error log checks for you.