Creating a Centralized Logging Solution with AWS CloudWatch, Elasticsearch, and Kibana

In today’s fast-paced world of modern applications, centralized logging is a game-changer for managing logs and maintaining seamless operations. Businesses with centralized logging can reduce downtime by up to 40% during critical incidents, thanks to the unified visibility and real-time insights it provides. By consolidating logs from multiple sources, teams can troubleshoot faster, monitor application health effectively, and proactively address potential issues. Centralized logging simplifies system management, saving time and enhancing operational efficiency.

At SUDO Consultants, we recognize the transformative potential of centralized logging. As an AWS Partner, we specialize in building tailored logging frameworks that scale effortlessly with your business. Using services like CloudWatch and OpenSearch, we deliver solutions that enhance system visibility, reliability, and ease of use.

Why AWS CloudWatch, Elasticsearch, and Kibana?

Centralized logging becomes seamless when leveraging the right tools, and AWS CloudWatch, OpenSearch, and Kibana form an excellent trio for managing and visualizing log data. Here’s why:

• AWS CloudWatch: CloudWatch excels at collecting and monitoring logs and metrics from AWS services and applications. It provides real-time monitoring and alerts, enabling teams to quickly detect and act on operational issues. Additionally, its seamless integration with other AWS services ensures scalability and reliability for businesses of any size.
• Amazon Elasticsearch (OpenSearch): OpenSearch is a robust tool for querying, indexing, and analyzing log data. It supports complex searches and enables pattern identification, making it invaluable for troubleshooting and gaining actionable insights. Its scalability ensures that even businesses with large log volumes can maintain high performance.
• Kibana: As a user-friendly interface for OpenSearch, Kibana simplifies data visualization and analysis. It enables teams to create custom dashboards, track trends, and monitor key metrics effectively. With its intuitive design, even non-technical users can explore log data and derive meaningful insights.

Step-by-Step Case Example: Deploying a Centralized Logging Solution

Architecture Overview

Figure 1: Architecture overview

Step 1: Deploy the Sample Application

1.1 Launch an Ubuntu EC2 Instance

1. Go to the AWS Management Console → EC2.
2. Launch a new Ubuntu 20.04/22.04 instance:
   • Choose t2.micro for cost efficiency.
   • Create a new key pair or use an existing one.
   • Allow HTTP (port 80) and SSH (port 22) in the security group.
3. Connect to the instance using SSH:

ssh -i your-key.pem ubuntu@<instance-public-ip>

1.2 Set Up Node.js

Update the Ubuntu packages:

sudo apt update && sudo apt upgrade -y

Install Node.js and NPM:

sudo apt install -y nodejs npm
node -v
npm -v

1.3 Create the Node.js Application

1. Set up the app directory and initialize Node.js:

mkdir aws-logging-example
cd aws-logging-example
npm init -y
npm install express winston

2. Create the application file app.js:

Paste the following code:

const express = require('express');
const winston = require('winston');

const app = express();

const port = 3000;

// Logger configuration

const logger = winston.createLogger({

  transports: [

    new winston.transports.Console(),

    new winston.transports.File({ filename: 'app.log' })

  ]

});

// Sample route

app.get('/', (req, res) => {

  logger.info('Root endpoint accessed');

  res.send('Hello World! This is a sample node.js app!');

});

app.listen(port, () => {

  logger.info(`App running on http://localhost:${port}`);

});

3. Run the app:

node app.js

Figure 2: Terminal showing application startup logs

4. Test the app by accessing http://<instance-public-ip>:3000 in your browser.

Figure 3: Browser accessing the app via http://<instance-public-ip>:3000.

Step 2: Configure AWS CloudWatch Logs

2.1 Install CloudWatch Agent

1. Download and install the CloudWatch Agent:

sudo apt update
sudo apt install -y amazon-cloudwatch-agent

Pro-tip: If you run into a “Unable to locate package amazon-cloudwatch-agent” error, manually download and install the CloudWatch Agent .deb package from AWS S3. After downloading the .deb package, add the binary to your shell’s PATH. 

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb && sudo dpkg -i amazon-cloudwatch-agent.deb

sudo dpkg -i amazon-cloudwatch-agent.deb

export PATH=$PATH:/opt/aws/amazon-cloudwatch-agent/bin

2. Verify the installation:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status

2.2 Create the CloudWatch Agent Configuration File

Create a configuration file named cloudwatch-config.json:

vim cloudwatch-config.json

 Add the following JSON:

{

  "agent": {

    "metrics_collection_interval": 60

  },

  "logs": {

    "logs_collected": {

      "files": {

        "collect_list": [

          {

            "file_path": "/home/ubuntu/aws-logging-example/app.log",

            "log_group_name": "nodejs-logs",

            "log_stream_name": "app-log-stream",

            "timestamp_format": "%Y-%m-%d %H:%M:%S"

          }

        ]

      }

    }

  }

}

1. Start the CloudWatch Agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
-a fetch-config \
-m ec2 \
-c file:/home/ubuntu/cloudwatch-config.json \
-s

Figure 4: Command output showing the CloudWatch Agent starting successfully.

2. Verify that logs are streaming to CloudWatch:

• Go to AWS CloudWatch Console → Log Groups → nodejs-logs.

Figure 5: CloudWatch Log Group nodejs-logs with log entries.

Step 3: Set Up Amazon Elasticsearch (OpenSearch) Service

3.1 Create a New Elasticsearch Domain

1. Navigate to the Amazon OpenSearch Service console.
2. Configure the domain:
   • Domain name: app-logs-es
   • Instance type: t3.small.elasticsearch
   • Access Policy: Allow access from the EC2 instance.

Figure 6: Elasticsearch domain configuration wizard

3. Wait for the domain to be active and note the Kibana URL.

Figure 7: Domain dashboard showing the “Active” status.

Step 4: Stream CloudWatch Logs to Elasticsearch

4.1 Set Up a Subscription Filter

1. Create an IAM role with permissions:
   • AmazonOpenSearchServiceFullAccess
   • CloudWatchLogsFullAccess
2. Attach the role to the CloudWatch agent or Lambda (if needed).
3. Go to CloudWatch Console → Log Groups → nodejs-logs.
4. Select the log stream → Actions → Stream to Amazon Elasticsearch Service.
5. Choose:
   • Destination: Amazon Elasticsearch Service
   • Domain: app-logs-es
   • Filter Pattern: Leave blank to stream all logs.

Figure 8: Subscription Filter setup wizard

6. Verify that logs are being streamed to Elasticsearch.

Step 5: Visualize Logs in Kibana

5.1 Access the Kibana Dashboard

1. Use the Kibana URL from the Elasticsearch domain dashboard.

Figure 9: Kibana URL

5.2 Configure the Index Pattern

1. In Kibana, navigate to Stack Management → Index Patterns.
2. Create a new index pattern:
   • Pattern: nodejs-logs*
   • Select Time Filter: timestamp

5.3 Visualize the Logs

1. Go to the Discover tab and verify logs.
2. Create visualizations:
   • Bar Chart: Log over time in vertical bar graphs.
   • Line Graph: Logs over time in a linear distribution.

Figure 10: Logs in Elasticsearch

Step 6: Test and Validate

1. Access the Node.js app multiple times to generate logs:

curl http://<instance-public-ip>:3000

2. Verify:
   • Logs appear in CloudWatch Logs.

Figure 11: Cloudwatch logs

• Logs are indexed in Elasticsearch.

Figure 12: Indexed logs in Elasticsearch

• Logs are visualized in Kibana.

Figure 13: Line graph visualization of node.js endpoint access attempts

Figure 14: Vertical bar graph showing app’s endpoint access attempts

Summary of the Step-by-Step Guide

This guide provides a clear roadmap for building a centralized logging solution using AWS CloudWatch, OpenSearch, and Kibana. With this setup, you can monitor application logs in real-time, gain deeper insights, and streamline troubleshooting, all while leveraging AWS’s scalable and reliable infrastructure. The solution empowers proactive monitoring, enhances operational efficiency, and keeps costs in check with AWS’s pay-as-you-go model. By integrating these tools, you ensure your system is well-equipped for seamless log management and improved application visibility.

Advanced Enhancements to Logging Solutions

Once you’ve set up a centralized logging solution using AWS CloudWatch, OpenSearch, and Kibana, you can implement several advanced enhancements to optimize performance and insights. These enhancements not only improve operational efficiency but also provide deeper visibility into your applications and infrastructure.

1. Anomaly Detection with Machine Learning
   Machine learning in OpenSearch can help detect real-time anomalies, such as unusual spikes in traffic or errors. It identifies outliers in log data, enabling you to predict and prevent system failures. By setting up anomaly detection, you can receive alerts for deviations in log patterns and address issues proactively.
2. Log Transformation with AWS Lambda
   AWS Lambda is a powerful tool for transforming logs. It can filter out unnecessary data to reduce noise and storage costs, add metadata like geolocation for better context, and standardize logs into a consistent format for easier analysis. These transformations make your logs more actionable and useful for decision-making.
3. Custom Visualizations in OpenSearch Dashboards
   Customizing OpenSearch Dashboards enhances their utility for specific teams and roles. You can create tailored views for DevOps engineers, developers, or security teams, add drill-down options for exploring detailed log data, and configure alerts and trends to highlight critical issues. These customizations make it easier for teams to collaborate and act on insights.
4. Cost Optimization Strategies
   To manage costs effectively, you can configure log retention policies to archive or delete older logs, use compression to save storage space, and enable tiered storage in OpenSearch for infrequent access. These strategies ensure you optimize costs without compromising on functionality.

Enhancing your logging solution with these strategies makes it more robust and scalable. 

SUDO Consultants: Your Partner in Reliable Logging Solutions

Centralized logging is no longer a luxury but a necessity in today’s fast-paced digital landscape. Solutions built with AWS CloudWatch, OpenSearch, and Kibana not only improve system visibility but also simplify troubleshooting and enhance application reliability. By implementing a centralized logging framework, businesses can proactively monitor their systems, reduce downtime, and optimize operations for long-term success.

Need help setting up your own centralized logging solution or fine-tuning your AWS infrastructure? Reach out to SUDO Consultants today and let our AWS-certified experts design a solution tailored to your unique needs. Together, we’ll ensure your systems are streamlined, resilient, and ready for the future. 

Francis Kariuki