Real-time Analytics using Data from IoT Devices - AWS Online Tech Talks

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Webinar Summary
Connected devices are everywhere and they are collecting
powerful data that can help companies gain actionable
insights and react promptly. In this webinar, we discuss
how to use Amazon Kinesis along with AWS IoT to quickly
and reliably stream data from devices into AWS for
analytics, and then use AWS Lambda to instruct those
devices what to do next, or to trigger some other action

Ryan Nienhuis
10/10/2017
Real-time Analytics using
Data from IoT Devices

Learning Objectives
• Learn the different options available to stream data from
IoT sensors to AWS
• Understand how to architect an analytics solution using
AWS services to ingest and process IoT data
• Take away best practices for building IoT applications
with scalability, cost-effectiveness, and security

AWS IoT App Challenge
https://aws.amazon.com/events/iot-app-challenge/

What is the end goal?

Why is solving this problem hard?
Designing for failure in global, real-time,
distributed systems is hard.
Infrastructure required to process billions of
devices sending trillions of messages is
expensive.
Management overhead and scale limitations
impede innovation.

What AWS services can help
you out?

AWS IoT Services: IoT Made Easy
AWS
Greengrass
AWS IoT
Platform

Amazon Kinesis: Streaming Data Made Easy
Amazon Kinesis
Streams
Amazon Kinesis
Analytics
Amazon Kinesis
Firehose

How do I get data from my
devices to the cloud?

AWS IoT Framework
AWS IoT
Messages
Device Gateway
Messages
Applications
Device Shadow
Message Broker
API
Registry
Messages
Messages
Rules Engine
AWS Services
Analytics
Kinesis
Artificial Intelligence
EMR
Messaging
SQS
SNS
Database
Redshift
DynamoDB
Manage
CloudWatch

Message Broker & Rules Engine
• Scalable, low-latency, reliable
message routing
• Publish/Subscribe for decoupled
devices and applications
• Two-way message streaming
between devices and
applications
• Data transformation, re-routing,
and enhancement with external
data sources

Message Broker & Rules Engine
• Route device data to and from
AWS services
• React to messages with Amazon
Simple Queue Service (SQS)
• Make machine learning
predictions in Amazon Elastic
Map Reduce (EMR)
• Analyze streaming data with
Amazon Kinesis
• Examine long-term trends in
data using Amazon Redshift
• Index data in real-time in
Amazon DynamoDB
MessagesMessages
Rules Engine
AWS Services

What does this look like in a real-world scenario?
Weather
Station AWS
IoT
IoT
rule
IoT
action
Rain sensor
Wind sensor
Temperature
sensor
Vibration
sensor
MQTT
Amazon
Kinesis

AWS IoT – Rule Setup
Incoming MQTT Topic
structure
weather/<state>/<city>/<station_id>/<sensor_type>/<sensor_id>

SQLStatement
SELECT * FROM
topic(6) AS sensor_id, topic(4) AS station_id,
topic(5) AS sensor, sensor_timestamp,
cast(sensor_value as float) AS sensor_value,
cast(sensor_value_smoothed as float) AS sensor_value_smoothed,
cast(direction as int) AS direction

SELECT * FROM
topic(6) AS sensor_id, topic(4) AS station_id,
topic(5) AS sensor, sensor_timestamp,
cast(sensor_value as float) AS sensor_value,
cast(sensor_value_smoothed as float) AS sensor_value_smoothed,
cast(direction as int) AS direction
References the AWS IoTMQTT
topic segment
<topic 1>/<topic 2>/…/<topic n>

AWS IoT – Rule Result
{
"value": 0.610802791886758,
"direction": -1,
"smoothed": 0.9843152123890655,
"timestamp": 1472611226005
}
Incoming payload

AWS IoT – Rule Result
Incoming payload{
"value": 0.610802791886758,
"direction": -1,
"smoothed": 0.9843152123890655,
"timestamp": 1472611226005
}
{
"sensor_id": "bQ7KcaMEas",
"station_id": "vzqHb8vghO",
"sensor": "vib",
"timestamp": 1472611226005,
"value": 0.610802791886758,
"value_smoothed": 0.9843152123890655,
"direction": -1
}
Transformed payload

How do I analyze IoT data in
real-time?

Simple Pattern for Streaming IoT Data
Continuously creates
data
Continuously writes
data to a stream
Can be almost
anything
Data Producer
Handles intermittent
connections
Durably stores data
in temporary buffer
Supports very high-
throughput
Ingest Layer
Continuously
processes data
Cleans, prepares, &
aggregates
Transforms data to
information
Data Consumer
IoT Device Amazon Kinesis app
AWS IoT
Amazon
Kinesis

Amazon Kinesis Analytics
• Apply SQL on streams
• Build real-time, stream processing applications
• Easy scalability
Connect to Kinesis streams,
Firehose delivery streams
Run standard SQL queries
against data streams
Kinesis Analytics can send processed data
to analytics tools so you can create alerts
and respond in real-time

SQL on streaming data, really?...
SQL is an API to your data;
• Ask for what you want, system decides how to get it;
• For all data, not just “flat” data in a database;
• Opportunity for novel data organization and algorithms;
• A standard (SQL 2011) and the most commonly used
data manipulation language used by data scientists to
business analysts and IT professionals.

Windowing Concepts
• Windows can be tumbling, sliding, session, etc.
• Windows can be fixed size (tumbling, sliding) or variable size
(session)
1 5 4 26 8 6 4
t1 t2 t5 t6t3 t4
Time
Window1 Window2 Window3
Aggregate
Function(Sum)
18 14OutputEvents

Amazon Elasticsearch Service
• Scalably deliver transformed data
• Build custom dashboards
• Ad-hoc search and analytics
Use a Firehose Delivery
to Deliver data to
ElasticSearch
Full text search
and log analytics
Use Kibana for real-time
dashboards
Elasticsearch

Kinesis Firehose Data Transformation
• Firehose buffers up to 3MB of ingested data
• When buffer is full, automatically invokes Lambda function,
passing array of records to be processed
• Lambda function processes and returns array of transformed
records, with status of each record
• Transformed records are saved to configured destination
[{"
"recordId": "1234",
"data": "encoded-data"
},
{
"recordId": "1235",
}
]
[{
"recordId": "1234",
"result": "Ok"
},
{
"recordId": "1235",
"result": "Dropped"
}
]

Amazon ES aggregations
Buckets – a collection of documents meeting some criterion
Metrics – calculations on the content of buckets
Bucket: time
Metric:count

host:199.72.81.55 with <histogram of verb>
1,
4,
8,
12,
30,
42,
58,
100
...
Look up
199.72.81.55
Field data
GET
GET
POST
GET
PUT
GET
GET
POST
Buckets
GET
POST
PUT
5
2
1
Counts

A more complicated aggregation
Bucket: ARN
Bucket: Region
Bucket: eventName
Metric: Count

What are some best practices
for analytics solutions on IoT
data?

Let’s pick a use case… continuous metrics
Examples
• Compute metrics from sensor logs
• Build leaderboards for key stats and device types
Key Requirements
• Produce accurate results with late and out of order data
• Combine results with historical data
• Quickly provide data to tech and non-tech end users

IoT sensors AWS IoT
RDS
MySQL DB
instance
Amazon
Kinesis
Streams
Amazon
Kinesis
Streams
Amazon
Kinesis
Analytics
AWS
Lambda function
Compute average
temperature every 10secIngest sensor data
Persist time series
analytic to database
Deep Dive: Time-Series Analytics
• Ingest and stream IoT data via MQTT
• Compute metrics like average temperature every 10
seconds

Best Practices for IoT Time-Series Analytics
• Handle data that arrives late and out of order from
IoT sensors that are not always online
• Use event or arrival time in your aggregation as a
key; this is the timestamp on your clients, check it for
accuracy
• Use processing time to trigger results; otherwise you
may only produce results when data arrives

Best Practices for IoT Time-Series Analytics
• Use linear interpolation or custom triggers; you are
going to have missing values and need to impute them
in some manner
• Perform upserts for your MySQL table, e.g. insert [..]
on duplicate key update [..]

Where to go next?

Try these use cases yourself
Many variations of these use cases have sample code on the AWS Big
Data and IoT Blogs.
Some good examples:
• Derive Insights from IoT in Minutes using AWS IoT, Amazon Kinesis
Firehose, Amazon Athena, and Amazon Quicksight
• Build a Visualization and Monitoring Dashboard for IoT Data with
Amazon Kinesis Analytics and Amazon Quicksight

Real-time Analytics using Data from IoT Devices - AWS Online Tech Talks

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Real-time Analytics using Data from IoT Devices - AWS Online Tech Talks

Similar to Real-time Analytics using Data from IoT Devices - AWS Online Tech Talks (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Real-time Analytics using Data from IoT Devices - AWS Online Tech Talks