Apache MXNet GLUON implementation of state of the art FER+ paper by Barsoum et. al. for facial emotion recognition. You can see the final demo at - http://bit.ly/mxnet-fer. In this implementation, we use majority voting (MV) technique illustrated in the paper and achieve 81.25% validation accuracy.(paper achieves 83.85%) We export the trained facial emotion recognition model and learn how to deploy at scale in production using MXNet Model Server. By end of this session, you will learn about Apache MXNet and its powerful and easy to use GLUON interface in Python, basics of Convolutional Neural Network (CNN), Training a state of the art CNN model for Facial Emotion Recognition, Productionizing deep learning models at scale using MXNet Model Server.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Emotion recognition in images
From idea to a model in production
Hagay Lupesko
Sandeep Krishnamurthy
©2018 Amazon Web Services, Inc. or its affiliates, All rights reserved

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Brief Intro to Deep Learning
AI
Machine
Learning
Deep
Learning
Can machines think?
Can machines do what we can?
(Turing, 1950)
Machine
Learning
Data
Answers Rules
Traditional
Programming
Data
Rules Answers

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inspired by the brain’s Neurons
We have ~100B of them, and ~1Q Synapses
w1
w2
wn
x1
x2
xn
Σ φ
Inputs Weights Non-Linear
Activation
!
…
! = #(%
&'(
)
*+,+)
Brief Intro to Deep Learning – Artificial Neurons

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Brief Intro to Deep Learning – Neural Networks
Output
Layer
Input
Layer
Hidden
Layers
Many
More…
• Non linear
• Hierarchical
feature learning
• Scalable
architecture
• Computationally
intensive

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Training Neural Networks
Forward Pass
Backwards Pass
Input Data
Neural
Network
Inference
Loss
Back
Propagate
Update
Weights
Backwards Pass is where the magic of learning happens,
leveraging Gradient Descent.

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep Learning is a Big Deal
It has a growing impact on our lives
Personalization Logistics Voice Autonomous
Vehicles

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep Learning is a Big Deal
It’s able to do better than humans (and ML)

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Facial Emotion Recognition

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Misconception - AI First Strategy
• Start with the problem statement!
• Problem -> Is AI necessary -> Use AI
• I want to use AI -> Let me solve this problem with AI ->
How can I solve it now?

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Problem Statement
XYZ bank wants to improve the customer experience rating. They
recognize bank customer care representative is one of the main
factor to improve the experience for customers visiting the bank
branch offices. XYZ bank decides to analyze the emotion {Happy,
Stressed, Tired …} of its customer facing employees in different
settings {Time of the day, location, before and after event …} and
strategize on –
Happy employees, happy customers!

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Stage 1 - Problem Formulation
• Scalable solution is to automate.
• Capture photos of the customer care representatives and analyze
the emotions.
• We want a machine to be able to See, learn and classify.
• A good problem to be solved with AI using Computer Vision (CV)
techniques.
“We want to build a Facial Emotion Recognition model”

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Stage 2 – Do I need to build my model?
SOTA
• State of the Art Research (SOTA)
Model Zoo
• Is the pre-trained model available in Model Zoo - https://github.com/onnx/models,
https://gluon-cv.mxnet.io/model_zoo/index.html, http://gluon-nlp.mxnet.io/scripts/index.html
Transfer
Learning / Fine
Tuning
• Extend the pre-trained model – Transfer Learning, Fine Tuning?
Build the Model
• Build and Train the model for your problem.

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SOTA – FER+
• FER+ – Training Deep Networks for Facial Expression Recognition
with Crowd-Sourced Label Distribution by Barsoum et. al.
https://arxiv.org/pdf/1608.01041.pdf
• 8 emotion types: {neutral, happiness, surprise, sadness, anger,
disgust, fear, and contempt}
• Pre-trained model in ONNX model zoo -
https://github.com/onnx/models/tree/master/emotion_ferplus
For this talk, let us implement the FER+ in Apache MXNet GLUON

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apache MXNet - Background
• Apache (incubating) open source project
• Framework for building and training
DNNs
• Created by academia (CMU and UW)
• Adopted by AWS as DNN framework of
choice, Nov 2016
http://mxnet.io

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apache MXNet - Highlights
• Imperative, Symbolic and Dynamic APIs
• APIs in Python, Scala, C, C++, R (and more)
• Examples and tutorials
Ease of Use
• Optimized for CPU, GPU, ARM (and more)
• Highly scalable distributed training
• Quantization, Sparse, NCCL, and more…
Performance
• Train on cloud, predict on edge
• Model serving framework
• ONNX support
Portability

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Model Training Lab
https://github.com/TalkAI/facial-emotion-recognition-gluon

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Model
Model Server
Mobile
Desktop
IoT
Internet
So what does a deployed model looks like?

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance
Availability
Networking
Monitoring
Model Decoupling
Cross Framework
Cross Platform
The Undifferentiated
Heavy Lifting of
Model Serving
Model Server for
MXNet

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MXNet Model Server
• Machine learning model server
• Serves MXNet and ONNX models
• Automated HTTP endpoints setup
• Auto-scales to all available CPUs and GPUs
• Pre-built and configured containers
• CLI to package model artifacts for serving
• Open source project under AWS Labs
http://modelserver.io

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Trained
Network
Model
Signature
Custom
Code
Auxiliary
Assets
Model Archive
Model Export CLI
Model Archive
Back

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serving Our Model

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Containerization
• Production-ready orchestration tools: ECS, Docker, Kubernetes
• Easy to scale out
• Robust and scalable images
• Automatically leverages all GPUs and CPUs on host
• Pre-configured images on DockerHub:
awsdeeplearningteam/mms_cpu
Back

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MMS
Docker
Image
Pull or Build
Run
Containerization
Container Cluster
MMS Container
MMS ContainerMMS Container
MXNet NGINX
MXNet Model Server
Load
Balancer

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deploying Models with Containers

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo - http://bit.ly/mxnet-fer

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apache MXNet Resources
• http://mxnet.incubator.apache.org/
• Learn Deep Learning with Gluon - https://gluon.mxnet.io/
• GitHub Repo - https://github.com/apache/incubator-mxnet
• Medium: https://medium.com/apache-mxnet
• Twitter: @ApacheMXNet
• Wiki: https://cwiki.apache.org/confluence/display/MXNET
• Subscribe to dev list – dev@mxnet.incubator.apache.org

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apache MXNet Evaluation
• Want to learn more and start using MXNet?
• Our experts can review your use cases
• We will help you with an evaluation of MXNet
• We can walk you through the steps to get to a production ready state
• Partner with AWS!
• Become a customer reference
• Write a blog and receive market and peer recognition
• Customer case study to promote what you’re doing in Deep Learning
• Contact Marcelo Bossi to get started
• Email: mbbossi@amazon.com
• Cell: (650) 796-1781

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Try it out, Star and Contribute!
http://mxnet.io
http://modelserver.io