Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ian Massingham, Chief Evangelist (EMEA), Amazon ...
AWS IoT
DEVICE SDK
Set of client libraries to
connect, authenticate and
exchange messages
DEVICE GATEWAY
Communicate with ...
Under the hood
Amazon
SQS
Amazon
DynamoDB
AWS IoT
Amazon
Kinesis
Amazon
EC2
Amazon
VPC
AWS IoT
Data storage
& analytics
Administration
Sensors
Actuators
Connected Farm
Control
automation
Telemetry & Analytics
Administration
Actuators
Control
automation
AWS IoT
Data storage
& analytics
Sensors
Connected Farm
AWS IoT Telemetry & Analytics
1. Connect devices
2. Send data
3. Collect & store the data
4. Do something with the data
AWS IoT Telemetry & Analytics
DEVICE GATEWAY
Communicate with devices via
MQTT and HTTP
AUTHENTICATION
AUTHORIZATION
Secur...
1) Connect the devices
1. Provision a certificate
2. Attach policy
3. Connect over MQTT
• Principle of Least Privilege
• L...
2) Send data
PUBLISH macdonald/sensors/123 (qos: 0)
{
"timestamp": "2016-01-29T10:00:00",
"temperature": 55
"humidity": 39...
3) Collect the data
AWS IoT
Data storage
& analytics
Sensors
?
How to get the data out of IoT, and where to put it?
Single consumer (don’t do this)
AWS IoT instance database
PUBLISH sensors/123
PUBLISH sensors/456
SUBSCRIBE sensors/+
PUBL...
Don’t do this: scalability
AWS IoT instance
SUBSCRIBE #
Don’t do this: availability
AWS IoT instance
Don’t do this: maintainability
AWS IoT
Store it in the device shadow (don’t do this)
Sensors
DEVICE SHADOWS
1. AWS Services
(Direct Integration)
Rules Engine
Actions
AWS IoT Rules Engine
LambdaSNS SQS
S3
Amazon
KinesisDDB RDS
Amaz...
Example rule
{
"rule": {
"sql": "SELECT * AS message FROM 'sensors/#'",
"description": "Store all sensor data into dynamod...
Now, solve the “where to put it” problem
Want to run a lot of queries constantly?
Use Amazon Kinesis Firehose to write int...
Takeaways
• Avoid single “firehose” MQTT consumer architecture
• Rules scalably route data into the rest of AWS
• Fork dat...
Cloud Control
Administration
AWS IoT
Data storage
& analytics
Sensors
Connected Farm
Actuators
Control
automation
Automated Sprinkler Service
Amazon
Kinesis
Amazon Machine
Learning
Amazon
Redshift
Rules
Engine
Device
Gateway
Sensor
Spri...
Talking back to the sprinklers
Amazon
Kinesis
Amazon Machine
Learning
Amazon
Redshift
Rules
Engine
Sensor
Device
Gateway
S...
Publish on/off to the sprinkler (don’t do this)
Device
Gateway
Sprinkler
Control
logic
SUBSCRIBE
macdonald/sprinkler-456
Publish on/off to the sprinkler (don’t do this)
Device
Gateway
Sprinkler
Control
logic
PUBLISH
macdonald/sprinkler-456
{ "...
Direct publishing: why not?
Sprinkler
Control
logic
on
off
Device
Gateway
off
on
Why aren’t messages ordered?
QoS 1
SQS Fanout Queue
Publisher SubscriberSubscriber
Dealer
Direct publishing: why not?
Device
Gateway
Sprinkler
(offline) Control
logic
PUBLISH
macdonald/sprinkler-456
{ "water": "o...
Direct publishing: why not?
• Messages aren’t ordered
• Connection blips
So then what?
Device Shadows
Shadow
State
Apps
offline
Device Shadows
Device Controller
reported
state
desired
state
Device Shadows
Device Controller
reported
state
desired
state
HTTP/REST
WebSockets
MQTT
AWS IoT Shadow - Simple Yet Powerful
{
"state" : {
“desired" : {
"lights": { "color": "RED" },
"engine" : "ON"
},
"reporte...
Device shadows and versioning
Sprinkler
Control
logic
on (version=1)
off (version=2)
Device
Gateway
off (version=2)
on (ve...
Device shadows: under the hood
Moonraker
Dealer
Publisher
Shadow
state table
Subscriber
Takeaways
• Plan for devices losing connectivity
• Send devices commands through shadows
• Query device state through shad...
Mobile Control
Data storage
& analytics
Sensors
Talking back to the sprinklers: manual override
Control
automation
AWS IoT
Administration...
AWS IoT
DEVICE SHADOW
Persistent thing state
during intermittent
connections
APPLICATIONS
Using Cognito with IoT
DEVICE SHADOW
Persistent thing state
during intermittent
connections
APPLICATIONS
AMAZON
COGNITO
PE...
end-user
(farmer)
Using Amazon Cognito with AWS IoT
DEVICE SHADOW
Persistent thing state
during intermittent
connections
A...
Policy for Amazon Cognito with AWS IoT
Amazon Cognito identity pool policy:
{
"Effect": "Allow",
"Action": "iot:*",
"Resou...
Policy for Amazon Cognito with AWS IoT
Amazon Cognito identity pool policy:
{
"Effect": "Allow",
"Action": "iot:*",
"Resou...
Policy for Amazon Cognito with AWS IoT
Amazon Cognito identity pool policy:
{
"Effect": "Allow",
"Action": "iot:*",
"Resou...
Overall Amazon Cognito “pairing” workflow
1. Create a Amazon Cognito identity pool.
2. Customer signs in using mobile app....
Managing fine-grained permissions
• One “farm owner” needs permissions to many shadows
• "arn:aws:iot:…:thing/sprinkler123...
Best practice: Thing name prefixing
• Prefix thing name with logical owner
• sensor123abc -> macdonald-sensor123abc
• Poli...
Takeaways:Amazon Cognito authorization
• Amazon Cognito enables secure human control
over IoT devices
• IoT scope-down pol...
WebSockets
• Amazon Cognito users now can do streaming
communication over AWS IoT
• Before: PUBLISH only over HTTP
• After...
WebSockets
AWS IoT
Managing software updates
Data storage
& analytics
Managing software updates
Control
automation
AWS IoT
Administration
Actuators
Sensors
Firmware on one topic (don’t do this)
• Have all devices subscribe to one topic
• Publish updated binaries to this topic
S...
Firmware on one topic (don’t do this)
Pros:
• Sending an update is easy
Cons:
• Large messages not supported
• Offline dev...
Firmware version shadow (don’t do this)
• One thing shadow for the current firmware version
• All devices subscribe to sha...
Firmware version shadow (don’t do this)
Pros:
• Sending an update is easy
• Offline devices eventually see updates
• Bulk ...
Firmware in devices own shadows
• Set each device’s shadow to its desired firmware version
• Devices subscribe to their ow...
Firmware in devices own shadows
SUBSCRIBE
$aws/shadow/sensor-abc123
PUBLISH $aws/shadow/sensor-abc123
{
"desired": {
"vers...
Firmware in devices own shadows
Pros:
• Full control over rollout / rollback
• Offline devices eventually see updates
• Bu...
Takeaway
• Be careful with wide fan out to millions of devices
• Wide fan out is supported, but slow
• Encourage safe devi...
Wrap-up
AWS IoT
Data storage
& analytics
Administration
Sensors
Actuators
Connected Farm
Control
automation
AWS IoT
DEVICE SDK
Set of client libraries to
connect, authenticate and
exchange messages
DEVICE GATEWAY
Communicate with ...
Key takeaways
• Messaging
• Be careful with wide fan out
• No message ordering guarantees
• Avoid large fan-in
• WebSocket...
Ian Massingham, Chief Evangelist (EMEA), Amazon Web Services
@IanMmmm
Upcoming SlideShare
Loading in …5
×

Deep Dive on AWS IoT

4,182 views

Published on

AWS IoT is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices.As an IoT developer, you will need to interact with AWS services like Amazon Kinesis, AWS Lambda, and Amazon Machine Learning to get the most from your IoT application. In this session, we will do a deep dive on how to define rules in the Rules Engine, or retrieve the last known and desired state of device using Device Shadows, routing data from devices to AWS services to leverage the entire cloud for your Internet of Things application.

Published in: Technology

Deep Dive on AWS IoT

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ian Massingham, Chief Evangelist (EMEA), Amazon Web Services @IanMmmm July 7, 2016 Deep Dive on AWS IoT Shadows, rules & more
  2. 2. AWS IoT DEVICE SDK Set of client libraries to connect, authenticate and exchange messages DEVICE GATEWAY Communicate with devices via MQTT and HTTP AUTHENTICATION AUTHORIZATION Secure with mutual authentication and encryption RULES ENGINE Transform messages based on rules and route to AWS services AWS Services - - - - - 3P Services DEVICE SHADOW Persistent thing state during intermittent connections APPLICATIONS AWS IoT API DEVICE REGISTRY Identity and management of your things
  3. 3. Under the hood Amazon SQS Amazon DynamoDB AWS IoT Amazon Kinesis Amazon EC2 Amazon VPC
  4. 4. AWS IoT Data storage & analytics Administration Sensors Actuators Connected Farm Control automation
  5. 5. Telemetry & Analytics
  6. 6. Administration Actuators Control automation AWS IoT Data storage & analytics Sensors Connected Farm
  7. 7. AWS IoT Telemetry & Analytics 1. Connect devices 2. Send data 3. Collect & store the data 4. Do something with the data
  8. 8. AWS IoT Telemetry & Analytics DEVICE GATEWAY Communicate with devices via MQTT and HTTP AUTHENTICATION AUTHORIZATION Secure with mutual authentication and encryption RULES ENGINE Transform messages based on rules and route to AWS services AWS Services - - - - - 3P Services
  9. 9. 1) Connect the devices 1. Provision a certificate 2. Attach policy 3. Connect over MQTT • Principle of Least Privilege • Limit what topics it can publish to (don’t impersonate other devices, talk to devices you’re not supposed to) • Limit what topics it can subscribe to (don’t read data you’re not supposed to / get data about other devices)
  10. 10. 2) Send data PUBLISH macdonald/sensors/123 (qos: 0) { "timestamp": "2016-01-29T10:00:00", "temperature": 55 "humidity": 39, "ph": 6.7 }
  11. 11. 3) Collect the data AWS IoT Data storage & analytics Sensors ? How to get the data out of IoT, and where to put it?
  12. 12. Single consumer (don’t do this) AWS IoT instance database PUBLISH sensors/123 PUBLISH sensors/456 SUBSCRIBE sensors/+ PUBLISH sensors/789
  13. 13. Don’t do this: scalability AWS IoT instance SUBSCRIBE #
  14. 14. Don’t do this: availability AWS IoT instance
  15. 15. Don’t do this: maintainability AWS IoT
  16. 16. Store it in the device shadow (don’t do this) Sensors DEVICE SHADOWS
  17. 17. 1. AWS Services (Direct Integration) Rules Engine Actions AWS IoT Rules Engine LambdaSNS SQS S3 Amazon KinesisDDB RDS Amazon Redshift Glacier EC2 3. External Endpoints (via Lambda and SNS) Rules Engine connects AWS IoT to External Endpoints and AWS Services. 2. Rest of AWS (via Amazon Kinesis, Lambda, S3, and more)
  18. 18. Example rule { "rule": { "sql": "SELECT * AS message FROM 'sensors/#'", "description": "Store all sensor data into dynamodb and firehose", "actions": [{ "dynamoDB": { "tableName": "sensor_data", "roleArn": "arn:aws:iam::123456789012:role/aws_iot_dynamoDB", "hashKeyField": "sensor_id", "hashKeyValue": "${topic(2)}", "rangeKeyField": "timestamp“ "rangeKeyValue": "${timestamp()}", } }, { "firehose": { "roleArn": "arn:aws:iam::123456789012:role/aws_iot_firehose", "deliveryStreamName": "my_firehose_stream" } }] } }
  19. 19. Now, solve the “where to put it” problem Want to run a lot of queries constantly? Use Amazon Kinesis Firehose to write into Amazon Redshift Need fast lookups, e.g. in Rules or Lambda functions? Write into DynamoDB, add indexes if necessary Have a need for heavy queries but not always-on? Use Amazon Kinesis Firehose & Amazon S3, process with Amazon EMR Want to analyze, search and visualize your device-generated data? Use AWS IoT Rules to route data into Elasticsearch domains
  20. 20. Takeaways • Avoid single “firehose” MQTT consumer architecture • Rules scalably route data into the rest of AWS • Fork data into multiple data stores simultaneously • Avoid the device shadow for analytics
  21. 21. Cloud Control
  22. 22. Administration AWS IoT Data storage & analytics Sensors Connected Farm Actuators Control automation
  23. 23. Automated Sprinkler Service Amazon Kinesis Amazon Machine Learning Amazon Redshift Rules Engine Device Gateway Sensor Sprinkler Amazon Kinesis– enabled app
  24. 24. Talking back to the sprinklers Amazon Kinesis Amazon Machine Learning Amazon Redshift Rules Engine Sensor Device Gateway Sprinkler Amazon Kinesis– enabled app
  25. 25. Publish on/off to the sprinkler (don’t do this) Device Gateway Sprinkler Control logic SUBSCRIBE macdonald/sprinkler-456
  26. 26. Publish on/off to the sprinkler (don’t do this) Device Gateway Sprinkler Control logic PUBLISH macdonald/sprinkler-456 { "water": "on" }
  27. 27. Direct publishing: why not? Sprinkler Control logic on off Device Gateway off on
  28. 28. Why aren’t messages ordered? QoS 1 SQS Fanout Queue Publisher SubscriberSubscriber Dealer
  29. 29. Direct publishing: why not? Device Gateway Sprinkler (offline) Control logic PUBLISH macdonald/sprinkler-456 { "water": "on" }
  30. 30. Direct publishing: why not? • Messages aren’t ordered • Connection blips So then what?
  31. 31. Device Shadows Shadow State Apps offline
  32. 32. Device Shadows Device Controller reported state desired state
  33. 33. Device Shadows Device Controller reported state desired state HTTP/REST WebSockets MQTT
  34. 34. AWS IoT Shadow - Simple Yet Powerful { "state" : { “desired" : { "lights": { "color": "RED" }, "engine" : "ON" }, "reported" : { "lights" : { "color": "GREEN" }, "engine" : "ON" }, "delta" : { "lights" : { "color": "RED" } } }, "version" : 10 } Thing Report its current state to one or multiple shadows Retrieve its desired state from shadow Mobile App Set the desired state of a device Get the last reported state of the device Delete the shadow Shadow Shadow reports delta, desired and reported states along with metadata and version
  35. 35. Device shadows and versioning Sprinkler Control logic on (version=1) off (version=2) Device Gateway off (version=2) on (version=1) (old message ignored by device)
  36. 36. Device shadows: under the hood Moonraker Dealer Publisher Shadow state table Subscriber
  37. 37. Takeaways • Plan for devices losing connectivity • Send devices commands through shadows • Query device state through shadows • Version numbers control concurrency
  38. 38. Mobile Control
  39. 39. Data storage & analytics Sensors Talking back to the sprinklers: manual override Control automation AWS IoT Administration Actuators
  40. 40. AWS IoT DEVICE SHADOW Persistent thing state during intermittent connections APPLICATIONS
  41. 41. Using Cognito with IoT DEVICE SHADOW Persistent thing state during intermittent connections APPLICATIONS AMAZON COGNITO PERMISSIONS APIs Configure device and Amazon Cognito user permissions end-user (farmer)
  42. 42. end-user (farmer) Using Amazon Cognito with AWS IoT DEVICE SHADOW Persistent thing state during intermittent connections APPLICATIONS AMAZON COGNITO PERMISSIONS APIs Configure device and Amazon Cognito user permissions
  43. 43. Policy for Amazon Cognito with AWS IoT Amazon Cognito identity pool policy: { "Effect": "Allow", "Action": "iot:*", "Resource": "*" } Specific policy for Old Macdonald Amazon Cognito user: { "Effect": "Allow", "Action": "iot:UpdateThingShadow", "Resource": "arn:aws:iot:…:thing/macdonald-sprinkler123" }
  44. 44. Policy for Amazon Cognito with AWS IoT Amazon Cognito identity pool policy: { "Effect": "Allow", "Action": "iot:*", "Resource": "*" } Specific policy for Old Macdonald Amazon Cognito user: { "Effect": "Allow", "Action": "iot:UpdateThingShadow", "Resource": "arn:aws:iot:…:thing/macdonald-sprinkler123" } Amazon Cognito
  45. 45. Policy for Amazon Cognito with AWS IoT Amazon Cognito identity pool policy: { "Effect": "Allow", "Action": "iot:*", "Resource": "*" } Specific policy for Old Macdonald Amazon Cognito user: { "Effect": "Allow", "Action": "iot:UpdateThingShadow", "Resource": "arn:aws:iot:…:thing/macdonald-sprinkler123" } AWS IoT
  46. 46. Overall Amazon Cognito “pairing” workflow 1. Create a Amazon Cognito identity pool. 2. Customer signs in using mobile app. 3. Associate their user with their “farm”. 4. Create a scope-down policy in AWS IoT for their user. 5. Attach that policy to their Amazon Cognito user in AWS IoT.
  47. 47. Managing fine-grained permissions • One “farm owner” needs permissions to many shadows • "arn:aws:iot:…:thing/sprinkler123abc" • "arn:aws:iot:…:thing/sprinkler456def" • … • Listing each is tedious
  48. 48. Best practice: Thing name prefixing • Prefix thing name with logical owner • sensor123abc -> macdonald-sensor123abc • Policy supports wildcards • "arn:aws:iot:…:thing/sensor123abc" • "arn:aws:iot:…:thing/sensor123abc" • "arn:aws:iot:…:thing/sensor456def" • … • "arn:aws:iot:…:thing/macdonald-*"
  49. 49. Takeaways:Amazon Cognito authorization • Amazon Cognito enables secure human control over IoT devices • IoT scope-down policy supports fine-grained control • Naming conventions simplify policy management • Setting permissions in practice is tricky, needs more innovation (pairing? Existing patterns?)
  50. 50. WebSockets • Amazon Cognito users now can do streaming communication over AWS IoT • Before: PUBLISH only over HTTP • After: PUBLISH and SUBSCRIBE over WebSockets!
  51. 51. WebSockets AWS IoT
  52. 52. Managing software updates
  53. 53. Data storage & analytics Managing software updates Control automation AWS IoT Administration Actuators Sensors
  54. 54. Firmware on one topic (don’t do this) • Have all devices subscribe to one topic • Publish updated binaries to this topic SUBSCRIBE sensor/firmware SUBSCRIBE sensor/firmware SUBSCRIBE sensor/firmware PUBLISH sensor/firmware 01100100 01101111 00100000 01101110 01101111 01110100 00100000 01100100 01101111 00100000 01110100 01101000 01101001 01110011
  55. 55. Firmware on one topic (don’t do this) Pros: • Sending an update is easy Cons: • Large messages not supported • Offline devices miss updates • No control over rollout
  56. 56. Firmware version shadow (don’t do this) • One thing shadow for the current firmware version • All devices subscribe to shadow updates • Messages include a CloudFront download URL SUBSCRIBE $aws/shadow/firmware-thing PUBLISH $aws/shadow/firmware-thing { "desired": { "version": “123.45" "url": “https://abc123.cloudfront.net/newversion" } } SUBSCRIBE $aws/shadow/firmware-thing
  57. 57. Firmware version shadow (don’t do this) Pros: • Sending an update is easy • Offline devices eventually see updates • Bulk download happens through CloudFront Cons: • No control over rollout • Shadow protocol is chatty
  58. 58. Firmware in devices own shadows • Set each device’s shadow to its desired firmware version • Devices subscribe to their own shadow • Messages include a CloudFront download URL
  59. 59. Firmware in devices own shadows SUBSCRIBE $aws/shadow/sensor-abc123 PUBLISH $aws/shadow/sensor-abc123 { "desired": { "version": “123.45" "url": “https://abc123.cloudfront.net/newversion" } } SUBSCRIBE $aws/shadow/sensor-def456 PUBLISH $aws/shadow/sensor-def456 { "desired": { "version": “123.45" "url": “https://abc123.cloudfront.net/newversion" } }
  60. 60. Firmware in devices own shadows Pros: • Full control over rollout / rollback • Offline devices eventually see updates • Bulk download happens through CloudFront Cons: • Sending updates requires sending multiple messages
  61. 61. Takeaway • Be careful with wide fan out to millions of devices • Wide fan out is supported, but slow • Encourage safe device management
  62. 62. Wrap-up
  63. 63. AWS IoT Data storage & analytics Administration Sensors Actuators Connected Farm Control automation
  64. 64. AWS IoT DEVICE SDK Set of client libraries to connect, authenticate and exchange messages DEVICE GATEWAY Communicate with devices via MQTT and HTTP AUTHENTICATION AUTHORIZATION Secure with mutual authentication and encryption RULES ENGINE Transform messages based on rules and route to AWS services AWS Services - - - - - 3P Services DEVICE SHADOW Persistent thing state during intermittent connections APPLICATIONS AWS IoT API DEVICE REGISTRY Identity and Management of your things
  65. 65. Key takeaways • Messaging • Be careful with wide fan out • No message ordering guarantees • Avoid large fan-in • WebSockets for Amazon Cognito authentication • Rules • Send data to multiple data stores at the same time • Manage device lifecycle events • Shadows • Designed for the real world: poor connectivity, out of order messages • Fine-grained control over software rollouts • Not ideal for storing time-series analytics data • Security • One cert per device • Set fine-grained permissions for devices and Amazon Cognito users • Naming conventions can simplify policy management
  66. 66. Ian Massingham, Chief Evangelist (EMEA), Amazon Web Services @IanMmmm

×