Internet of Things is creating a tidal wave of new data including events, correlations, business value, and much more. With the proliferation of new data sets, it also introduces more potential issues, errors, and spurious values. In this session, we will explore using Amazon Machine Learning to analyse and understand the new data collected within your IoT solution. In addition, we will learn how to discover patterns, trends, anomalies, and correlations by demonstrating the capabilities of Amazon Machine Learning and SparkML running on AWS Cloud. Speaker: Simon Elisha, Solutions Architect, Amazon Web Services

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simon Elisha – Head of Solution Architecture, ANZ Public Sector, AWS
April 2016
Anomaly Detection in IoT Applications
Using Amazon Machine Learning to Identify Trends in IoT Data
Technical 201

2. What We’ll Cover Today
• Overview of AWS IoT and Amazon Machine Learning
• Anomaly Detection – What? Why? How?
• How to Build an IoT Solution using Machine Learning
based Anomaly Detection

3. AWS IoT
Securely connect one or one billion devices to AWS,
so they can interact with applications and other devices

4. AWS IoT
DEVICE GATEWAY
AUTHENTICATION
AUTHORISATION
RULES ENGINE
DEVICE SHADOW
DEVICE REGISTRY
AWS SERVICES
APPLICATIONS
AWS IoT
API

5. PUBLISH myiotapp/cars/bcd123 (qos: 0)
{
"timestamp": "2016-­04-­27T10:14:32.429",
“vehicle”: “bcd123”,
”lat": -­33.867537904,
”long": 151.207044258,
”elev": 25.7,
“vel”: 1.04,
“heading”: 225.1
}

6. Amazon Machine Learning
A predictive analytics service that makes it easy for developers
of all skill levels to use machine learning technology.

7. Amazon Machine Learning
Based on What You
Know about the User:
Will they Use Your
Product?

8. Amazon Machine Learning
Based on What You
Know about an Order:
Is this Order
Fraudulent?
Based on What You
Know about the User:
Will they Use Your
Product?

9. Amazon Machine Learning
Based on What You Know
about a News Article:
What Other Articles are
Interesting?
Based on What You
Know about an Order:
Is this Order
Fraudulent?
Based on What You
Know about the User:
Will they Use Your
Product?

10. Amazon Machine Learning

11. Amazon Machine Learning

12. IoT and ML are a Phenomenal Combination
When it comes to working with Complex Real World Systems.

15. Anomaly Detection

16. Awesome Traffic Reduction Application
Drive at
25
km/h

17. Data
PUBLISH myiotapp/cars/bcd123 (qos: 0)
{
"timestamp": "2016-­04-­27T10:14:32",
“vehicle”: “bcd123”,
”lat": -­33.867537904,
”long": 151.207044258,
”elev": 25.7,
“vel”: 1.04,
“heading”: 225.1
}
PUBLISH myiotapp/tlights/a45d0 (qos: 0)
{
"timestamp": "2016-­04-­27T10:10:05",
“tlight”: “a45d0”,
”state": “red”,
”duration": “72”
}

18. Behaviour Modes
• Entering Road
• Exiting Road
• Driving Between Intersections
• Stopped at Lights
• Waiting for Clear Path to Turn
• Waiting for Pedestrian
• Parking at Kerb

19. Unexpected Behaviours
• Car off Road
• Cars Stopped for Long Periods
• Erratic Driving
• More Traffic or More Messages than Expected

20. Supervised Machine Learning
The model is trained using historical data (or observations)
that are labeled with accurate answers for the problem
under analysis.
lat long elev vel heading activity
33.63078996 153.2408174 65.962 14.6 88.1 driving
33.61899274 153.4826805 64.732 0.6 163.8 parking
33.67753203 153.2248169 67.682 1.58 321.4 entering
33.10896801 153.0650639 61.006 27.52 145.9 driving
33.91719004 153.952396 53.305 0.0 60.0 at_lights
33.08905011 153.7517515 64.59 0.05 122.9 waiting
33.44729954 153.8196027 48.619 22.02 349.6 driving
exiting
Target
column

21. Supervised Machine Learning
lat long elev vel heading activity
33.63078996 153.2408174 65.962 14.6 88.1 driving
33.61899274 153.4826805 64.732 0.6 163.8 parking
33.67753203 153.2248169 67.682 1.58 321.4 entering
33.10896801 153.0650639 61.006 27.52 145.9 driving
33.91719004 153.952396 53.305 0.0 60.0 at_lights
33.08905011 153.7517515 64.59 0.05 122.9 waiting
33.44729954 153.8196027 48.619 22.02 349.6 driving
33.85352192 153.4845429 48.265 5.49 251.1 exiting
... … … … ... …

22. Supervised Machine Learning
33.61899274 153.4826805 64.732 0.6 163.8
driving: 0.007642,
parking:0.908068,
exiting:0.00581,
entering:0.01754,
waiting:0.06094

23. Supervised Machine Learning
33.61899274 153.4826805 64.732 0.6 163.8
driving: 0.2046,
parking:0.2027,
exiting:0.1085,
entering:0.3249,
waiting:0.1593

24. How Do We Build The Training Data?
lat long elev vel heading activity
33.63078996 153.2408174 65.962 14.6 88.1 ?
33.61899274 153.4826805 64.732 0.6 163.8 ?
33.67753203 153.2248169 67.682 1.58 321.4 ?
33.10896801 153.0650639 61.006 27.52 145.9 ?
33.91719004 153.952396 53.305 0.0 60.0 ?
33.08905011 153.7517515 64.59 0.05 122.9 ?
33.44729954 153.8196027 48.619 22.02 349.6 ?
33.85352192 153.4845429 48.265 5.49 251.1 ?
... … … … ... …

25. Trivial to get the data
RULES ENGINE
iotTrafficApp
SELECT * FROM ’myiotapp/cars'
"actions":
[{
”s3": {
”bucketName": ”iotTrafficApp",
“key” : “car-data/${timestamp()}”
"roleArn":"arn:aws:iam:…:role/aws_iot_s3”
}
}]

26. Unsupervised Machine Learning
x_da x_os p_e1 p_e2 q_e1 q_e2
3 413 -­‐0.568 -­‐0.848 1.132 1.633
4 989 0.062 -­‐0.062 3.6847 1.868
4 537 -­‐0.034 -­‐0.756 3.3729 1.0858
2 233 0.408 0.722 4.5867 3.1139
2 407 -­‐0.272 -­‐0.93 1.01 4.998
5 200 0.482 0.802 0.0952 2.6146
2 976 0.88 -­‐0.088 2.3031 0.4448
1 497 -­‐0.446 -­‐0.086 0.8797 3.4498
2 450 0.332 0.82 2.5444 4.5703

27. Clustering

28. Clustering
lat long elev vel heading activity
33.63078996 153.2408174 65.962 14.6 88.1 A
33.61899274 153.4826805 64.732 0.6 163.8 B
33.67753203 153.2248169 67.682 1.58 321.4 C
33.10896801 153.0650639 61.006 27.52 145.9 A
33.91719004 153.952396 53.305 0.0 60.0 D
33.08905011 153.7517515 64.59 0.05 122.9 E
33.44729954 153.8196027 48.619 22.02 349.6 A
33.85352192 153.4845429 48.265 5.49 251.1 F
33.6616973 153.3943277 64.205 0.66 338 B
33.9324867 153.6082797 50.684 21.96 282.2 A
33.64924421 153.5149211 50.97 1.4 16.1 C
33.58459572 153.1589644 62.842 0 287.5 E

29. Clustering
lat long elev vel heading activity
33.63078996 153.2408174 65.962 14.6 88.1 driving
33.61899274 153.4826805 64.732 0.6 163.8 parking
33.67753203 153.2248169 67.682 1.58 321.4 entering
33.10896801 153.0650639 61.006 27.52 145.9 driving
33.91719004 153.952396 53.305 0.0 60.0 stopped
33.08905011 153.7517515 64.59 0.05 122.9 waiting
33.44729954 153.8196027 48.619 22.02 349.6 driving
33.85352192 153.4845429 48.265 5.49 251.1 entering
33.6616973 153.3943277 64.205 0.2 338 parking
33.9324867 153.6082797 50.684 21.96 282.2 driving
33.64924421 153.5149211 50.97 1.4 16.1 entering
33.58459572 153.1589644 62.842 0 287.5 waiting

30. Observations
Anomalies
Conform with
known Behaviours

31. What Do Anomalies Represent?
Malfunction New Mode
or Extension of
Existing Mode
Intrusion, Malware
or DDOS

32. Blocking Access
AUTHENTICATION
AUTHORISATION
AWS IoT
API
ANOMALY
DETECTION
update-­certificate
REVOKE

34. Building Secure Applications: A Reminder
def updateCar(id, data):
assert ID_re.matches(id)
assert isinstance(data.ma
assert isinstance(data.mo

35. Six Steps to Getting it Done
Cluster Analysis Build Predictive Model Run Predictions
Handle Critical Risks Assess Anomalies Iterate
driving: 0.007642,
parking: 0.908068,
exiting: 0.00581
33.6189 153.482 64.732 0.6 163.8

36. AWS Training & Certification
Intro Videos & Labs
Free videos and labs to
help you learn to work
with 30+ AWS services
– in minutes!
Training Classes
In-­person and online
courses to build
technical skills –
taught by accredited
AWS instructors
Online Labs
Practice working with
AWS services in live
environment –
Learn how related
services work
together
AWS Certification
Validate technical
skills and expertise -­
identify qualified IT
talent or show you
are AWS cloud ready
Learn more: aws.amazon.com/training

37. Your Training Next Steps:
ü Visit the AWS Training & Certification pod to discuss your
training plan & AWS Summit training offer
ü Register & attend AWS instructor led training
ü Get Certified
AWS Certified? Visit the AWS Summit Certification Lounge to pick up your swag
Learn more: aws.amazon.com/training

38. Thank You!