Datapalooza: A Music Festival Themed ML & IoT Workshop

Welcome
Datapalooza:
A Music festival themed
ML & IoT Workshop

Agenda
Overview—30 mins
Prep Challenge—1 hour
Challenge 1—2.5 hours
Lunch
Challenge 2—2.5 hours

The Challenge
DataPalooza—A music festival themed ML & IoT Workshop
Scenario: Your bold startup has taken the challenge of providing a new type of EDM music festival
experience. At venues with multiple stages, festival-goers are always looking to identify which DJ stage
areas are the liveliest. This causes them to constantly move around between different stages and miss
out. You are looking to use Machine Learning and IoT to come up with a connected
fan experience that takes the music festival scene to the next level. From your initial research there are
existing ML models that you can leverage to do face and emotion detection, but there are two ways that
the predictions (inference) can be done; on the cloud and on the camera itself, but which one will work the
best for your needs at the festival? You are going to test both approaches and find out!
In this workshop you will use AWS and Intel technologies including Amazon SageMaker with Intel C5
Instances, AWS DeepLens, AWS Greengrass, Amazon Rekognition, AWS Lambda, along with Intel IoT
hardware kits. The objective of the workshop is to learn how to build and deploy a machine learning model
and then run inference on it from the cloud and from the edge device.
By the time you’re done with these challenges, EDM DJ’s will be able to tell whether the crowd is enjoying
their set by the looks on their faces.

Amazon Machine Learning Stack
Platform Services
Application Services
Frameworks & Interfaces
Caffe2 CNTK
Apache
MXNet
PyTorch TensorFlow Chainer Keras Gluon
AWS Deep Learning AMIs
Infrastructure
EC2 GPUs EC2 CPUs IoT Edge
AWS
DeepLens
Education

Machine Learning Process is Hard…
Fetch
data
Clean &
format data
Prepare &
transform
data
Train
model
Evaluate
model
Integrate
with prod
Monitor/
debug/
refresh
Data wrangling
• Set up and manage
Notebook environments
• Get data to notebooks securely
Experimentation
• Set up and manage clusters
• Scale/distribute ML algorithms
Deployment
• Set up and manage
inference clusters
• Manage and auto scale
inference APIs
• Testing, versioning,
and monitoring

Amazon’s fast, scalable algorithms
Distributed Apache MXNet and TensorFlow
Bring your own algorithm
Hyperparameter optimization
Building HostingTraining
Amazon SageMaker Components

Building Hosting (C/P)Training

Resizable as
you need
Common tools
pre-installed
Easy access to
your data sources
No servers
to manage
Zero Setup for Data Exploration

Clusters of GPU
or powerful CPU

Distributed Training that Works with You
Amazon-optimized
algorithms using the
AWS SDK…
… or Apache Spark
IM Estimators
Bring your own deep
learning script…
… or your custom
algorithm Docker image

More than Just General Purpose Algorithms
XGBoost, FM, and
Linear for classification
and regression
Kmeans and PCA
for clustering and
dimensionality reduction
Image classification
with convolutional
neural networks
LDA and NTM for
topic modeling, seq2seq
for translation

Amazon ECS
Bring Your Own Algorithm
... publish to Amazon ECS... add algorithm code
to a Docker container...
Choose your own framework

Elastic Clusters
CPU or GPU
instances

Hosting
One-step
deployment
Low latency
High throughput
High reliability
A/B
testing
Use your
own model

Modular Architecture So You Can Use What You Need
Training
algorithm
Model
artifacts
Inference
code
Client
application
Model
Data Inference
Ground
truth
Amazon SageMaker
Past
Data

Pay As You Go and Inexpensive
ML compute by the
second starting
at $0.0464/hr
ML storage by the
second at $0.14
per GB-month
Data processed in
notebooks and hosting
at $0.016 per GB
Free trial to
get started quickly

Amazon EC2 C5 Instances
Cost effective CPUs, e.g., for models using INT8
• Powered by 3.0 GHz Intel Xeon (Skylake) platinum processors
• 72 vCPUs and 155-GB RAM (25% price/performance improvement versus C4)
• Nitro Hypervisor for larger instance sizes
Ideal for running ML inference as GPU based instances would be overkill
(cost saving)
Suitable for training simple ML algorithms (text or CSV data) or during
dev/test mode and proof-of-concepts

Can we do more to put ML in the
hands of all developers (literally)?

AWS Deeplens
is not a video camera…
…it’s the worlds first
Deep Learning Enabled
Developer kit

Artistic style
transfer
Object
detection
Face detection
/recognition
Hot dog/
not hot dog
Cat versus
dog
Activity
detection
add custom functionality
or
create your own project
Get Started with Sample Projects

Deeplens Specifications
• Intel Atom Processor
• Gen9 graphics
• Ubuntu OS- 16.04 LTS
• 100 GFLOPS performance
• Dual band Wi-Fi
• 8 GB RAM
• 16 GB Storage (eMMC)
• 32 GB SD card
• 4 MP camera with MJPEG
• H.264 encoding at 1080p resolution
• 2 USB ports
• Micro HDMI
• Audio out
• AWS Greengrass preconfigured
• clDNN Optimized for MXNet

AWS Deeplens Architecture
Video out
Data out
Inference
Deploy projects
Manage device
Security
Console Project
Management
AWS Cloud
Intel: Model Optimizer
cIDNN and Driver

AWS and Intel
Amazon Web Services (AWS) and Intel technologies are designed to provide
a more secure, scalable edge-to-cloud solution for IoT applications
• Operate locally and on the cloud
• Easily manage and update devices
• Connect fleets of devices, gateways, and cloud environments

Customer Pain Points with IoT Implementation
Security
• Securing data transport to the cloud
with encryption
• Enabling devices to communicate
with one another without
introducing vulnerabilities
• Ensuring devices have not been
tampered with before sending data
to the cloud
• Authenticating device identity without
sending credentials over the wire
Deployment and Management
• Managing large numbers of
simultaneous connections to devices
connecting via different networks
• Updating device software, patching, and
sending configurations to device fleets
• Incorporating legacy and proprietary
protocols with IoT deployments
• Bandwidth and storage costs of
sending device data to the cloud when
local hardware has sufficient resources
for local analytics
• Ongoing security management over
life of implementation
Scale
• Managing large numbers of
simultaneous connections to devices
connecting via different networks
• Updating device software, patching, and
sending configurations to device fleets
• Incorporating legacy and proprietary
protocols with IoT deployments
• Bandwidth and storage costs of
sending device data to the cloud when
local hardware has sufficient resources
for analytics

Benefits of Using AWS with Intel IoT Hardware
Easy to deploy
and manage
Whether making existing
things smart or deploying
new connected devices,
AWS and Intel make it easy
to get started
Security enabled
Intel hardware and software
solutions are tightly
integrated with the robust
AWS cloud infrastructure to
deliver enhanced security,
from to device, to network,
to cloud
Scalable
Start with minimal or no
upfront investment and
easily scale to millions
of devices and billions
of messages
Cost-effective
Leverage pay-as-you-go
pricing, the flexibility to use
local and cloud resources,
and flexible and low-cost IT
resources powered by Intel
technology to reduce the
costs of IoT deployments

AWS and Intel Strategies to Maximize Value
of IoT Deployments
Act locally on device data
at the edge. Use the cloud
for management, analytics,
and durable storage
Operate offline in
circumstances when
latency requirements or
intermittent connectivity
that make a round trip to
the cloud unfeasible
Execute AWS Lambda
functions locally using AWS
Greengrass, reducing the
complexity of developing
embedded software
Increase the quality of the
data you send to the cloud
through filtering device data
locally and only transmitting
the data you need so you
can achieve rich insight at
a lower cost

Where Do I Want To Process Data?
Infrastructure CloudPoPIoT Endpoint Gateway Appliance
Common Programming Model
Onboard
AWS
Cloud
Lambda
@Edge
Amazon
FreeRTOS
Greengrass

Features of Greengrass
Security
AWS-grade
security
Data and
state sync
Local
Device Shadows
Local
triggers
Local
Message Broker
Local
actions
Local
Lambda Functions
Machine
Inference
Local Execution
of ML Models
Protocol
Adapters
Local messaging
with other devices
Over the
Air Updates
Easily Update
Greengrass Core
Local
Resource Access
Lambdas interact
with peripherals
Amazon
FreeRTOS
Works together
out of the box

Benefits AWS Greengrass
Respond quickly
to local events
Operate
offline
Simplified device
programming
Reduce the cost of
IoT applications
AWS-grade
security

Amazon Rekognition
A Music festival themed
ML & IoT Workshop

Images—Universal, Ubiquitous, and Essential
There are 3,700,000,000 internet users in 2017
1,200,000,000 photos will be taken in 2017 (9% YoY Growth)
Source: InfoTrends Worldwide

Amazon Rekognition
Extract rich metadata from visual content
Object and Scene
Detection
Facial
Analysis
Face
Comparison
Facial
Recognition
Celebrity
Recognition
Image
Moderation

Why Use Rekognition?
Object & Scene Detection
• Photo-sharing apps can power smart searches and
quickly find cherished memories, such as weddings, hiking,
or sunsets
Facial Analysis
• Retail businesses can understand the demographics and
sentiment of in-store customers
Face Comparison
• Hotels & hospitality businesses can provide seamless
access for guests and VIPs
Facial Recognition
• Provide secondary authentication for existing applications

Object and scene detection makes it easy for you to add features
that search, filter, and curate large image libraries
DetectLabels
Flower
Arrangement
Chair
Coffee Table
Living Room Indoors
Furniture
Cushion
Vase
Maple
Villa
Plant
Garden
Water
Swimming Pool
Tree
Potted Plant
Backyard
Patio
Object & Scene Detection
Identify objects and scenes and provide confidence scores

Emotion Expressed
General Attributes
Facial Pose
Facial Landmarks
EyeLeft,EyeRight,Nose
RightPupil,LeftPupil
MouthRight,LeftEyeBrowUp
Bounding Box...
Happy
Surprised
Smile:True
EyesOpen:True
Beard:True
Mustache:True
Pitch
Roll
Yaw
Demographic Data
Age Range
Gender:Male
29–45
96.5%
Facial Analysis
Analyze facial characteristics in multiple dimensions
DetectFaces
Image Quality
Brightness
Sharpness
23.6%
99.9%
83.8%
0.65%
23.6%
99.8%
99.5%
99.9%
1.446
5.725
4.383

CompareFaces
Face Comparison
Measure the likelihood that faces are of the same person
Similarity 93%
Similarity 0%

Search
Index
Collection
SearchFacesByImage
Facial Recognition
Find similar faces in a large collection of images

Detect explicit and suggestive contentRecognize thousands of famous individuals
DetectModerationLabelsRecognizeCelebrities
Celebrity Recognition & Image Moderation
Newly released Rekognition features

Interfacing with Rekognition
Optimizing your input & requests for best performance
• S3 input for API calls – max image size of 15MB
• 5MB limit for non-S3 (Base64 encoded) API calls
• Minimum image resolution (x or y) of 80 pixels
• Image data supported in PNG or JPG format
• Max number of faces in a single face collection is 1 million
• The max matching faces the search API returns is 4096
• Size of face should occupy 5%+ of image for detection
• Collections are for faces!
…
Use Amazon CloudWatch to observe & issue alerts on Rekognition metrics

https://console.aws.amazon.com/rekognition/home

Rekognition APIs—Overview
Rekognition’s computer vision API operations can be grouped into
Non-storage API operations, and Storage-based API operations
Non-storage API Operations Storage-based API Operations
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
{
"FaceMatches": [
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces

What Can You Do with Amazon Rekognition?
Search for people, objects, scenes, and concepts across millions of images
Filter inappropriate or specific content
Redact identities from images of faces
Verify identities by matching against reference faces
Recognize individuals by matching faces to a collection
Analyze user traffic hotspots and journey paths by demographics and sentiment

Searchable Image Library
Real Estate Property Search
Property Search Amazon Elasticsearch
User captures an image
for their property listing
Mobile app uploads
the image to S3
A Lambda function is triggered
and calls Rekognition
Rekognition retrieves the image from S3 and
returns labels for the property and amenities
Lambda pushes the labels and
confidence scores to Elasticsearch
Other users can search properties
by landmarks, category, etc.
Photo Upload Amazon S3 AWS Lambda Detect Objects & Scenes

Searchable Image Library
Real Estate Property Search
• Optimize the client
• Event based, decoupled infra
• Buffering—SQS, SNS, Kinesis
• Rate Control—high volume S3 image ingest
• DynamoDB—scale label storage
• Elasticsearch—operational & performance statistics
• CloudFront—search cache
AWS
Lambda
Amazon
S3
Amazon
SQS
AWS
CloudFormation
Amazon
CloudWatch
Amazon
Kinesis
Amazon
CloudFront
Amazon
DynamoDB
Amazon
ElasticSearch
1
2
3

Face-Based User Verification
Confirm user identities by comparing their live image with a reference image
Authenticated User
Image Capture
Amazon S3
Compare Faces
Rekognition compares the live image
and the badge image—and returns
a similarity score
The application retrieves the
user’s badge from S3
Application
If the similarity score is over 92%,
the application returns a green status.
If not, an alert is issued to security staff
The application captures a live
image of each employee as they
scan their access card

Face-Based User Verification
Confirm user identities by comparing their live image with a reference image
• S3 Encryption of badge images—
SSE-S3, SSE-KMS, SSE-C
• Prevent tampering with bucket
policies & IAM RO permissions
• Extend by using Rek collections
• Cloudtrail—Logging & Auditing
with tamper-proof log signatures
• Tie notification into SNS/SES,
Custom CloudWatch Logs metrics,
or ElasticSearch with alerts
AWS
KMS
AWS
CloudTrail
AWS
Lambda
Amazon
S3
Amazon
SNS
AWS
CloudFormation
Amazon
CloudWatch
Amazon
SES
1
2
3

Facial Recognition
Identify individuals by matching a live image to a collection of
images of known persons
#0123 #0123
5426 128762
78426 45871
286546 26751
3861 945
Images SearchFacesByImage Face Collection
Person Details Table
Photo AppEnd User Amazon S3
Rekognition searches the face collection for
matches to the reference image and returns an
array of face metadata for potential face
matches, ordered by similarity
If source images are required,
they are retrieved from S3
The photo app displays search
results to the end user

Collections and Access Patterns
Logging—visitor logs, digital libraries
• Easily find specific images from a digital library
• Find certain images by using a reference image
Social Tagging—photo storage and sharing
• One collection per application user
• Automated friend tagging
Person Verification—employee gate check
• One collection for each person to be verified
• Detection of stolen/shared IDs

Rekognition APIs—Advanced Usage
Decision trees and processing pipelines
Why?
• Many use cases require more than a single operation
to arrive at actionable data
How?
• S3 event notifications, Lambda, Step Functions
• DynamoDB for persistent pipeline storage
• Augmenting results with 3rd Party AI/ML
• OpenCV, MXNet, etc. on EC2 Spot, ECS, AI/ML AMI
Sample Use Cases
• Person of interest near a celebrity
• Multi-pass motion detection enhancement
• Subjects leaving a location without possessions
IndexFaces
DetectLabels
“person”

aws rekognition recognize-celebrities –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
aws rekognition search-faces-by-image –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
--collection-id “persons-of-interest"
aws rekognition create-collection --collection-id “persons-of-interest”
aws rekognition index-faces --image “S3Object={Bucket=mybucket,Name=subject.jpg}”
--collection-id “persons-of-interest”
Rekognition APIs—Advanced Usage
Person of Interest Near a Celebrity
{
"FaceMatches": [
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
2
{
"FaceMatches": [
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
3
1

Automating Footage Tagging with
Amazon Rekognition
• Built in three weeks
• Indexed against 99,000 people
• Index created in one day
• Saved ~9,000 hours a year in
manual curation costs
• Live video with frame sampling
Previously, only about half of all footage
was indexed due to the immense time
requirements required by manual processes

Automating Footage Tagging with
Amazon Rekognition
Solution Architecture
EncodersStills Extraction & Feeds
Results
Cache Bucket
R3
Amazon
Rekognition
users
Stills FramesSQS Trigger
1
2
3
4
1

Visual Search
Open Influence is a market leader in the
influencer marketing space and enables
global brands and agencies to identify
relevant influencers
• Real-time visual search, powered by
Rekognition, enables Open Influence to
tag millions of social images accurately
• Using Rekognition allowed Open Influence to
cut down the time it takes to source relevant
influencers from 2–3 days to minutes

Metadata Tagging
Scripps Networks Interactive is a leading
developer of engaging lifestyle content
• Instead of manually tagging media
assets, Rekognition enables Scripps
Networks Interactive to save time and
increase productivity with automated
metadata tagging

Resources
Product information:
• Product page: https://aws.amazon.com/deeplens/
• Blog posts: https://aws.amazon.com/blogs/machine-learning/category/artificial-intelligence/aws-deeplens/
• Developer community projects: https://aws.amazon.com/deeplens/community-projects/
Help Getting started with DeepLens:
10-Minute tutorials: they can already access the step by step guides now. Note: Friday 5/4 there will be video versions
on You Tube for each of these that are easier to follow:
1. How to Configure Your New AWS DeepLens
https://aws.amazon.com/getting-started/tutorials/configure-aws-deeplens/
2. How to Create and Deploy a Deep Learning Project With AWS DeepLens
https://aws.amazon.com/getting-started/tutorials/create-deploy-project-deeplens/
3. How to Extent a Deep Leaning Project with AWS DeepLens
https://aws.amazon.com/getting-started/tutorials/extend-deeplens-project/
4. How to Build an AWS DeepLens project using Amazon SageMaker
https://aws.amazon.com/getting-started/tutorials/build-deeplens-project-sagemaker/
General questions:
Check out the AWS DeepLens FAQs or AWS DeepLens Developer Forum

Thank you!
Closing Video, AWS & Intel Better Together : https://youtu.be/ocJx_h_v1DY

Datapalooza: A Music Festival Themed ML & IoT Workshop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Datapalooza: A Music Festival Themed ML & IoT Workshop

Similar to Datapalooza: A Music Festival Themed ML & IoT Workshop (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Datapalooza: A Music Festival Themed ML & IoT Workshop