The Advent of Deep Learning Deep Learning Applications Apache MXNet Overview Apache MXNet Demos (there will be code, ha!)

1. Julien Simon, AI/ML Evangelist, EMEA
@julsimon
November 8th, 2017
An introduction to Deep Learning
with Apache MXNet

2. Agenda
• The Advent of Deep Learning
• Deep Learning Applications
• Apache MXNet Overview
• Apache MXNet Demos (there will be code, ha!)

3. Artificial Intelligence: design software applications which
exhibit human-like behavior, e.g. speech, natural language
processing, reasoning or intuition
Machine Learning: teach machines to learn without being
explicitly programmed
Deep Learning: using neural networks, teach machines to
learn from complex data where features cannot be explicitly
expressed

4. The Advent of
Deep Learning
Algorithms

5. The Advent of
Deep Learning
Data
Algorithms

6. The Advent of
Deep Learning
Data
GPUs
& Acceleration
Algorithms

7. The Advent of
Deep Learning
Data
GPUs
& Acceleration
Programming
models
Algorithms

8. Selected customers running AI on AWS

9. Infrastructure CPU
Engines MXNet TensorFlow Caffe Theano Pytorch CNTK
Services
Amazon Polly
Chat
Platforms
IoT
Speech
Amazon Lex
Mobile
Amazon AI: Artificial Intelligence In The Hands Of Every Developer
Amazon
ML
Spark &
EMR
Kinesis Batch ECS
GPU
Amazon Rekognition
Vision
FPGA

10. Hardware innovation for Deep Learning
https://aws.amazon.com/blogs/aws/new-amazon-ec2-instances-with-up-to-8-nvidia-tesla-v100-gpus-p3/
https://devblogs.nvidia.com/parallelforall/inside-volta/
https://aws.amazon.com/fr/blogs/aws/now-available-compute-intensive-c5-instances-for-amazon-ec2/
Intel Skylake CPU
November 6th
Nvidia Volta GPU
October 25th

11. Deep Learning Applications

12. Image Classification
Same breed?
Humans: 5,1%

13. https://news.developer.nvidia.com/expedia-ranking-hotel-images-with-deep-learning/
• Expedia have over 10M images from
300,000 hotels
• Using great images boosts conversion
• Using Keras and EC2 GPU instances,
they fine-tuned a pre-trained Convolutional
Neural Network using 100,000 images
• Hotel descriptions now automatically feature the
best available images

14. • 17,000 images from Instagram
• 7 brands
• Inception v3 model, pre-trained on ImageNet
• Fine-tuning with TensorFlow and EC2 GPU
instances
• Additional work on color extraction
https://technology.condenast.com/story/handbag-brand-and-color-detection

15. Object Detection
https://github.com/precedenceguo/mx-rcnn https://github.com/zhreshold/mxnet-yolo

16. Object Segmentation
https://github.com/TuSimple/mx-maskrcnn

17. https://www.oreilly.com/ideas/self-driving-trucks-enter-the-fast-lane-using-deep-learning
Last June, tuSimple drove an autonomous
truck
for 200 miles from Yuma, AZ to San Diego,

18. Text Detection and Recognition
https://github.com/Bartzi/stn-ocr

19. Face Detection
https://github.com/tornadomeet/mxnet-face

20. Real-Time Pose Estimation
https://github.com/dragonfly90/mxnet_Realtime_Multi-Person_Pose_Estimation

21. Machine Translation
https://aws.amazon.com/blogs/ai/train-neural-machine-translation-models-with-sockeye/

22. Amazon Echo
Natural Language Processing & Text-to-Speech

23. Apache MXNet overview

24. Apache MXNet
Programmable Portable High Performance
Near linear scaling
across hundreds of GPUs
Highly efficient
models for mobile
and IoT
Simple syntax,
multiple languages
Most Open Best On AWS
Optimized for
Deep Learning on
AWS
Accepted into the
Apache Incubator

25. Input Output
1 1 1
1 0 1
0 0 0
3
mx. sym. Convol ut i on( dat a, ker nel =( 5, 5) , num_f i l t er =20)
mx. sym. Pool i ng( dat a, pool _t ype=" max" , ker nel =( 2, 2) ,
st r i de=( 2, 2)
l st m. l st m_unr ol l ( num_l st m_l ayer , seq_l en, l en, num_hi dden, num_embed)
4 2
2 0
4=Max
1
3
...
4
0.2
-0.1
...
0.7
mx. sym. Ful l yConnect ed( dat a, num_hi dden=128)
2
mx. symbol . Embeddi ng( dat a, i nput _di m, out put _di m = k)
0.2
-0.1
...
0.7
Queen
4 2
2 0
2=Avg
Input Weights
cos(w, queen ) = cos(w, k i n g) - cos(w, m an ) + cos(w, w om an )
mx. sym. Act i vat i on( dat a, act _t ype=" xxxx" )
" r el u"
" t anh"
" si gmoi d"
" sof t r el u"
Neural Art
Face Search
Image Segmentation
Image Caption
“ People Riding
Bikes”
Bicycle, People,
Road, Sport
Image Labels
Image
Video
Speech
Text
“ People Riding
Bikes”
Machine Translation
“ Οι άνθρωποι
ιππασίας ποδήλατα”
Events
mx. model . FeedFor war d model . f i t
mx. sym. Sof t maxOut put

26. CPU or GPU: your choice
mod = mx.mod.Module(lenet)
mod = mx.mod.Module(lenet, context=mx.gpu(0))
mod = mx.mod.Module(lenet,
context=(mx.gpu(7), mx.gpu(8), mx.gpu(9)))

27. Ideal
Inception v3
Resnet
Alexnet
88%
Efficiency
1 2 4 8 16 32 64 128 256
Distributed Training with MXNet

28. Apache MXNet demos

29. 1. Image classification: using pre-trained models
Imagenet, multiple CNNs, MXNet
2. Image classification: fine-tuning a pre-trained model
CIFAR-10, ResNet-50, Keras + MXNet
3. Image classification: learning from scratch
MNIST, MLP & LeNet, MXNet
4. Machine Translation: translating German to English
News, LSTM, Sockeye + MXNet

30. Demo #1 – Image classification: using a pre-trained model
*** VGG16
[(0.46811387, 'n04296562 stage'), (0.24333163,
'n03272010 electric guitar'), (0.045918692, 'n02231487
walking stick, walkingstick, stick insect'),
(0.03316205, 'n04286575 spotlight, spot'),
(0.021694135, 'n03691459 loudspeaker, speaker, speaker
unit, loudspeaker system, speaker system')]
*** ResNet-152
[(0.8726753, 'n04296562 stage'), (0.046159592,
'n03272010 electric guitar'), (0.041658506, 'n03759954
microphone, mike'), (0.018624334, 'n04286575
spotlight, spot'), (0.0058045341, 'n02676566 acoustic
guitar')]
*** Inception v3
[(0.44991142, 'n04296562 stage'), (0.43065304,
'n03272010 electric guitar'), (0.067580454, 'n04456115
torch'), (0.012423956, 'n02676566 acoustic guitar'),
(0.0093934005, 'n03250847 drumstick')]
https://medium.com/@julsimon/an-introduction-to-the-mxnet-api-part-5-9e78534096db

31. Demo #2 – Image classification: fine-tuning a model
CIFAR-10 data set
• 60,000 images in 10 classes
• 32x32 color images
Initial training
• Resnet-50 CNN
• 200 epochs
• 82.12% validation
Cars vs. horses
• 88.8% validation accuracy
https://medium.com/@julsimon/keras-shoot-out-part-3-fine-tuning-7d1548c51a41

32. Demo #2 – Image classification: fine-tuning a model
• Freezing all layers but the last one
• Fine-tuning on « cars vs. horses » for 10 epochs
• 2 minutes on 1 GPU (p2)
• 98.8% validation accuracy
Epoch 10/10
10000/10000 [==============================] - 12s
loss: 1.6989 - acc: 0.9994 - val_loss: 1.7490 - val_acc: 0.9880
2000/2000 [==============================] - 2s
[1.7490020694732666, 0.98799999999999999]

33. Demo #3 – Image classification: learning from scratch
MNIST data set
70,000 hand-written digits
28x28 grayscale images
https://medium.com/@julsimon/training-mxnet-part-1-mnist-6f0dc4210c62

34. Multi-Layer Perceptron vs. Handmade-Digits-From-Hell™
784/128/64/10, Relu, AdaGrad, 100 epochs  97.51% validation accuracy
[[ 0.839 0.034 0.039 0.009 0. 0.008 0.066 0.002 0. 0.004]]
[[ 0. 0.988 0.001 0.003 0.001 0.001 0.002 0.003 0.001 0.002]]
[[ 0.006 0.01 0.95 0.029 0. 0.001 0.004 0. 0. 0.]]
[[ 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]]
[[ 0. 0.001 0.005 0.001 0.982 0.001 0. 0.007 0. 0.002]]
[[ 0.001 0.001 0. 0.078 0. 0.911 0.01 0. 0. 0.]]
[[ 0.003 0. 0.019 0. 0.005 0.004 0.863 0. 0.105 0.001]]
[[ 0.001 0.008 0.098 0.033 0. 0. 0. 0.852 0.004 0.004]]
[[ 0.001 0. 0.006 0. 0. 0.001 0.002 0. 0.991 0.]]
[[ 0.002 0.158 0.007 0.117 0.082 0.001 0. 0.239 0.17 0.224]]

35. LeNet CNN vs. Handmade-Digits-From-Hell™
ReLu instead of tanh, 10 epochs, AdaGrad  99.20% validation accuracy
[[ 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
[[ 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]
[[ 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]]
[[ 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]]
[[ 0. 0. 0.001 0. 0.998 0. 0. 0.001 0. 0.]]
[[ 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]]
[[ 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]]
[[ 0. 0. 0. 0.001 0. 0. 0. 0.999 0. 0.]]
[[ 0. 0. 0.006 0. 0. 0. 0. 0. 0.994 0.]]
[[ 0. 0. 0. 0.001 0.001 0. 0. 0.001 0.001 0.996]]

36. Demo #4 – Machine Translation: German to English
• Sockeye
• 5.8M sentences (news headlines), 5 hours of training on 8 GPUs (p2)
./translate.sh "Chopin zählt zu den bedeutendsten Persönlichkeiten der
Musikgeschichte Polens .”
Chopin is one of the most important personalities of Poland’s history
./translate.sh "Hotelbetreiber müssen künftig nur den Rundfunkbeitrag
bezahlen, wenn ihre Zimmer auch eine Empfangsmöglichkeit bieten .”
in the future , hotel operators must pay only the broadcasting fee if their
rooms also offer a reception facility .
https://aws.amazon.com/blogs/ai/train-neural-machine-translation-models-with-sockeye/

37. Anything you dream is fiction, and anything
you accomplish is science, the whole history
of mankind is nothing but science fiction.
Ray Bradbury

38. Resources
https://aws.amazon.com/ai/
https://aws.amazon.com/blogs/ai/
https://mxnet.io
https://github.com/gluon-api/
https://reinvent.awsevents.com/ watch this space ;)
https://medium.com/@julsimon/

39. https://aws.amazon.com/fr/events/semaine-ia/

40. Merci!
Julien Simon, AI/ML Evangelist, EMEA
@julsimon
https://aws.amazon.com/evangelists/julien-simon/