Introduction
IoT has the full form of The Internet of Things. It has revolutionized our interaction and gathering data from the outside world. IoT platforms are needed constantly when the number of connected devices grows. AWS is always at your service and provides various services for building reliable and secure IoT applications. AWS IoT Core and IoT Analytics are kept at the forefront. In this guide you will be familiarized with the architecture, key features and the practices for developing IoT applications using AWS services.
Understanding AWS IoT Core
Overview
Let us start by discussing AWS IoT Core. It is a cloud service that allows devices to perform a secure connection as well as interact with other cloud applications and devices. AWS IoT Core basically acts as a central device, hub, which manages all the communication between applications and IoT devices. Functionalities such as authentication, device registry and secure communication are provided as well.
Key Features
AWS IoT Core gives you the ease to create a device registry in order to manage information regarding IoT devices. Each device has its own unique identifier and metadata which includes attributes, state and type.
To ensure that only the authorized devices can connect, AWS IoT Core supports mutual authentication between cloud and devices. To have further control on your devices you can refer to fine-grained access control policies.
MQTT stands for Message Queuing Telemetry Transport. It is an efficient and light-weight protocol that is used for IoT applications so that devices can communicate with AWS IoT Core. Devices which cannot use MQTT protocol directly can take help of HTTP.
To know about the current state of your device in the cloud you can use the Device Shadow feature provided by AWS IoT Core. This feature becomes beneficial when the application wants to interact with the physical device but it is offline so instead it interacts with the device’s virtual representation.
You can use Rules Engine to define rules for the purpose of processing and routing messages from your devices to other services of AWS.
Architecture
The key components of AWS IoT Core architecture are as follows.
Device Gateway is used to manage the communication that takes place between AWS IoT Core and other devices. MQTT and HTTP show full compatibility with it.
To perform integration with AWS services and real-time analytics, Rules Engine is the best approach to use because it processes and routes messages based on predefined rules.
Authentication and authorization features allow only the trusted devices to perform a connection with AWS IoT Core. You can use X.509 certificates, policies and IAM roles to make it happen.
Device registry contains all the metadata that is associated with a device including attributes, ID and state.
If any application wants to interact with a device even if it is offline then it can do so by using Device Shadow feature.
Building IoT Applications with AWS IoT Core
Step 1: Device Onboarding
The first step in order to develop an IoT application using AWS IoT Core is to have all your devices onboard. You need to register each and every device in the device registry, get necessary credentials and perform the required configurations to allow communication with AWS IoT Core.
# Example code for device onboarding using AWS SDK for Python (Boto3)
import boto3
iot_client = boto3.client('iot')
# Create a new thing (device) in the device registry
response = iot_client.create_thing(
thingName='MyIoTDevice'
)
# Create X.509 certificate and key for the device
keys_and_cert = iot_client.create_keys_and_certificate()
# Attach the certificate to the thing (device)
iot_client.attach_thing_principal(
thingName='MyIoTDevice',
principal=keys_and_cert['certificateArn']
)
# Provide the certificate and other details to the device for communication
device_certificate = keys_and_cert['certificatePem']
device_private_key = keys_and_cert['keyPem']
iot_endpoint = iot_client.describe_endpoint()['endpointAddress']Step 2: Secure Communication
The best part of using AWS IoT Core is that it provides secure communication between cloud and devices with the help of mutual authentication. During the TLS handshake devices present their relevant certificates and AWS IoT Core verifies them accordingly.
# Example code for device communication using AWS SDK for Python (Boto3) and MQTT
from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
# Set up MQTT client
mqtt_client = AWSIoTMQTTClient('MyIoTDevice')
mqtt_client.configureEndpoint(iot_endpoint, 8883)
mqtt_client.configureCredentials(device_private_key, device_certificate, None)
# Connect to AWS IoT Core
mqtt_client.connect()
# Publish a message
mqtt_client.publish('my_topic', 'Hello, AWS IoT Core!', 0)
# Subscribe to a topic
mqtt_client.subscribe('another_topic', 1, lambda _, msg: print('Received message:', msg.payload))Step 3: Implementing Device Shadow
If you use the Device Shadow feature then you will be allowed to create a virtual representation of the device’s state and use it to communicate even when the device is offline.
# Example code for interacting with Device Shadow using AWS SDK for Python (Boto3)
shadow_client = boto3.client('iot-data')
# Update the reported state of the device
shadow_client.update_thing_shadow(
thingName='MyIoTDevice',
payload=json.dumps({
'state': {
'reported': {
'temperature': 25.5,
'humidity': 60.2
}}}))
# Retrieve the desired state from the shadow
response = shadow_client.get_thing_shadow(
thingName='MyIoTDevice')
desired_state = json.loads(response['payload']['body'])['state']['desired']Step 4: Implementing Rules for Data Processing
To enable real-time analytics and integrate IoT data with other parts of your application, you need to use Rules Engine in AWS IoT Core.
// Example rule for forwarding messages to Amazon S3
{
"sql": "SELECT * FROM 'my_topic'",
"ruleDisabled": false,
"actions": [
{
"s3": {
"roleArn": "arn:aws:iam::123456789012:role/service-role/S3AccessRole",
"bucketName": "my-iot-data-bucket",
"key": "iot-data/${year}/${month}/${day}/${hour}/${minute}/${second}/${clientid}/${timestamp}"
}}]}Leveraging AWS IoT Analytics
Overview
AWS IoT Core is responsible for handling the management and communication of IoT devices. On the other hand AWS IoT Analytics is responsible for the analysis and processing of data generated by the IoT devices. IoT data is required for further analysis and AWS IoT Analytics simplifies this process.
Key Features
- Ingestion of data from multiple sources is made possible due to AWS IoT Analytics. The sources include AWS IoT Core, Amazon S3 and other sources. Data from various IoT services and devices can be combined easily at one place.
- All the IoT data can be preprocessed using SQL queries with the help of AWS IoT Analytics. You can filter, transform and clean the data before storing it in the data store.
- Either it is raw data or processed data, AWS IoT Analytics gives you the flexibility to store the data. Amazon S3 and Amazon Redshift are some of the famous data stores.
- Amazon QuickSight enables you to visualize the IoT data and gain insights. AWS IoT Analytics offers integration with such tools. Creation of dashboards, performing ad-hoc analysis and making data-driven decisions are some of the important functions which you can easily perform.
- You can even build and deploy machine learning models used for predictive analytics based on IoT data with the help of Amazon SageMaker. AWS IoT Analytics can be integrated with this service.
Architecture
The architecture of AWS IoT Analytics has many important components. Let us have a look at them, shall we?
Channel
Collects data from different sources and then stores it in the data store (raw).
Pipeline
Data related activities such as filtering, transforming and enrichment are processed by pipeline. The processed form of data is stored in the data store (processed).
Data Store
As the name suggests, it is used to store the raw and processed form of data. Amazon S3, Amazon Redshift or any other custom data store can be used.
Analytics
Various SQL queries are run on processed data which enables you to derive insights in order to perform data analysis.
Integration
Integration with AWS services is enabled to visualize and implement machine learning models.
Building IoT Analytics with AWS IoT Analytics
Step 1: Set up a Channel for Data Ingestion
The first step includes setting up a channel. You need to specify source and destination for the ingested data.
// Example JSON for creating an IoT Analytics channel
{
"channelName": "MyIoTChannel",
"channelStorage": {
"serviceManagedS3": {}
},
"tags": {}
}Step 2: Create a Pipeline for Data Processing
After the data has been ingested successfully, you need to create a pipeline for processing the data.
// Example JSON for creating an IoT Analytics pipeline
{
"pipelineName": "MyIoTPipeline",
"tags": {},
"pipelineActivities": [
{
"channel": {
"name": "MyIoTChannel"
}
},
{
"lambda": {
"name": "MyLambdaActivity",
"lambdaName": "MyLambdaFunction",
"batchSize": 1,
"next": "store"
}
},
{
"datastore": {
"name": "MyProcessedDatastore",
"datastoreName": "MyProcessedDatastore",
"name": "MyProcessedDatastore",
"retentionPeriod": {
"unlimited": true
}}}]}Step 3: Analyze Data with IoT Analytics
After the channel has been set properly and the pipeline is created successfully, you can use Analytics to run various SQL queries on the processed data in order to derive insights.
-- Example SQL query for data analysis
SELECT
timestamp,
device_id,
AVG(temperature) AS avg_temperature,
MAX(humidity) AS max_humidity
FROM
MyProcessedDatastore
WHERE
timestamp >= TIMESTAMP '2024-01-01T00:00:00.000Z'
GROUP BY
timestamp, device_idStep 4: Visualize Data with Amazon QuickSight
Now if you want to create dashboards and have visualizations then you can integrate AWS IoT Analytics to perform an integration with Amazon QuickSight.
// Example JSON for creating an Amazon QuickSight dashboard
{
"Name": "MyIoTDashboard",
"SourceEntity": {
"SourceTemplate": {
"DataSetReferences": [
{
"DataSetPlaceholder": "MyDataSet",
"DataSetArn": "arn:aws:quicksight:us-east-1:123456789012:dataset/MyDataSet"
}
],
"Arn": "arn:aws:quicksight:us-east-1:123456789012:template/MyTemplate"
}
},
"ThemeArn": "arn:aws:quicksight:us-east-1:123456789012:theme/MyTheme",
"AwsAccountId": "123456789012",
"DashboardId": "MyIoTDashboard"
}Best Practices for Developing IoT Applications on AWS
- Your first concern should be the security. You need to implement an authentication mechanism using various communication protocols and frequently updating the device credentials.
- Although IoT Core and IoT Analytics is designed to handle a large number of devices along with data, having a proper architecture planned is essential.
- Keep a check and balance on device’s state and configurations using device registry. It will also streamline the operations.
- You need to choose those communication protocols which match your device’s capabilities. If you need to improve the performance then optimize the payload size.
- Carefully look at the data retention requirements and then choose the storage option for your data. However, Amazon S3 is the most popular data store among users.
- You can enable real-time analytics using Rules Engine in AWS IoT Core.
- AWS IoT Analytics offers seamless integration with Amazon SageMaker. Amazon SageMaker allows the building and deployment of machine learning models on IoT data.
- Only use those AWS services which are really required by your application. This will optimize the overall cost.
- In order to meet compliance and security requirements, implement encryption, audit trails and access controls.
Conclusion
All in all, using AWS IoT Core and IoT Analytics side by side helps in coming up with scalable solutions. This collaboration helps the developers to manage and analyze the data of devices in IoT. AWS IoT Core acts as a central hub and ensures that secure communication is going on using features such as authentication, device registry, real-time analytics and dynamic rules engine.
