Build Deep Learning Applications Using Apache MXNet - Featuring Chick-fil-A (AIM407-R1) - AWS re:Invent 2018

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Information Extraction for Enterprise
Documents
A I M 4 0 7
Cyrus M Vahid
Principal Evangelist, AWS AI Labs
cyrusmv@amazon.com
Vivek Srivastava
ML Product Manager
Workday
Henry Zhang
ML Engineer
Workday

Agenda
Apache MXNet and why we use it
Amazon SageMaker and distributed
training
Amazon SageMaker and DevOps
style deep learning
Business Case
Framework
Model and Algorithms
Results
Productization
Key Learnings

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Computational Dependency/Graph
• 𝑧 = 𝑥 ⋅ 𝑦
• 𝑘 = 𝑎 ⋅ 𝑏
• 𝑡 = 𝜆𝑧 + 𝑘
x y
𝑧
x
𝜆
𝑢
x
a
x
b
k
𝑡
+
1 1
2
3

Computational Dependency/Graph
net = mx.sym.Variable('data')
net = mx.sym.FullyConnected(net, name='fc1', num_hidden=64)
net = mx.sym.Activation(net, name='relu1', act_type="relu")
net = mx.sym.FullyConnected(net, name='fc2', num_hidden=10)
net = mx.sym.SoftmaxOutput(net, name='softmax')

Training
import logging
logging.getLogger().setLevel(logging.DEBUG) # logging to stdout
# create a trainable module on compute context
mlp_model = mx.mod.Module(symbol=mlp, context=ctx)
mlp_model.fit(train_iter,
eval_data=val_iter,
optimizer='sgd',
optimizer_params={'learning_rate':0.1},
eval_metric='acc',
batch_end_callback = mx.callback.Speedometer(batch_size, 100),
num_epoch=10)

Training Efficiency – 92%
https://mxnet.incubator.apache.org/tutorials/vision/large_scale_classification.html

End-to-End Machine
Learning Platform
Zero setup Flexible Model
Training
Pay by the second
$
Amazon SageMaker
Build, train, and deploy machine learning models at scale

Amazon SageMaker and Distributed Training
• Faster training through SageMaker Streaming for Custom Algorithms.
• Boilerplate code for your algorithms to train over a cluster.
PCA Bemchmark
if len(hosts) == 1:
kvstore = 'device' if num_gpus > 0 else 'local’
else:
kvstore = 'dist_device_sync' if num_gpus > 0 else 'dist_sync’
trainer = gluon.Trainer(net.collect_params(), 'sgd’,
{'learning_rate': learning_rate, 'momentum': momentum},
kvstore=kvstore)

Training code
• Matrix Factorization
• Regression
• Principal Component Analysis
• K-Means Clustering
• Gradient Boosted Trees
• And More!
Amazon provided Algorithms
Bring Your Own Script (IM builds the Container)
Bring Your Own Algorithm (You build the Container)
Fetch Training data
Save Model Artifacts
Fully
managed –
Secured
–
Amazon ECR
Save Inference Image
IM Estimators in Apache
Spark
CPU GPU HPO
Distributed Training

Automatic Model Tuning
Training code
• Factorization Machine
• Regression/classification
• Principal Component Analysis
• K-Means Clustering
• XGBoost
• DeepAR
• And More
SageMaker built-in Algorithms Bring Your Own Script (prebuilt containers) Bring Your Own Algorithm
Fetch Training data Save Model Artifacts
Fully
managed –
Secured–
Automatic Model Tuning

Evolution of DL Frameworks

Why Gluon
Simple, Easy-to-
Understand Code
Flexible, Imperative
Structure
Dynamic Graphs High Performance

Gluon Code – Network Definition
net = gluon.nn.HybridSequential()
with net.name_scope():
net.add(gluon.nn.Dense(units=64, activation='relu'))
net.add(gluon.nn.Dense(units=10))
softmax_cross_entropy = gluon.loss.SoftmaxCrossEntropyLoss()
net.initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx, force_reinit=True)
trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.02})

Gluon Code – Training
smoothing_constant = .01
for e in range(10):
cumulative_loss = 0
for i, (data, label) in enumerate(train_data):
data = data.as_in_context(model_ctx).reshape((-1, 784))
label = label.as_in_context(model_ctx)
with autograd.record():
output = net(data)
loss = softmax_cross_entropy(output, label)
loss.backward()
trainer.step(data.shape[0])

What’s New – GluonCV
• A Deep Learning Toolkit for Computer Vision
• Features:
• Training scripts that reproduces SOTA results reported in latest papers
• A large set of pre-trained models
• Carefully designed APIs and easy-to-understand implementations
• Community support

What’s New - GluonNLP
• A deep learning toolkit for natural language processing
• Features:
• Training scripts to reproduce SOTA results reported in research papers.
• Pre-trained models for common NLP tasks.
• Carefully designed APIs that greatly reduce the implementation
complexity.
• Community support.

What’s New – Keras Backend
Instance Type GPUs Batch Size
Keras-MXNet
(img/sec)
Keras-
TensorFlow
(img/sec)
C5.18X Large 0 32df 13 4
P3.8X Large 1 32 194 184
P3.8X Large 4 128 764 393
P3.16X Large 8 256 1068 261
Instance Type GPUs Batch Size
Keras-MXNet
(img/sec)
Keras-
TensorFlow
(img/sec)
C5.X Large 0 32 5.79 3.27
C5.8X Large 0 32 27.9 18.2
https://github.com/awslabs/keras-apache-mxnet/tree/master/benchmark

What’s New - Sockeye
• A seq2seq toolkit based on Apache MXNet.
• Features:
• Beam search inference
• Easy assembling of multiple models
• Residual connections between RNN layers (Wu et al., 2016) [Deep LSTM with parallelism]
• Lexical biasing of output layer predictions (Arthur et al., 2016) [Low Frequency Words]
• Modeling coverage (Tu et al., 2016) [Keeping attention history to reduce over and under translation]
• Context gating (Tu et al., 2017) [Improving adequacy of translation by controlling rations of source and target
context]
• Cross-entropy label smoothing (e.g., Pereyra et al., 2017)
• Layer normalization (Ba et al, 2016) [improving training time]
• Multiple supported attention mechanisms [dot, mlp, bilinear, multihead-dot, encoder last state, location]
• Multiple model architectures (Encoder-Decoder Wu et al., 2016, Convolutional Gehring et al, 2017,
Transformer Vaswani et al, 2017,)

Inference Efficiency - TensorRT
Model Name Relative TRT Sppedup HArdware
Resnet 101 1.99x Titan V
Resnet 18 1.54x Jetson TX1
cifar_resnext29_16x64d 1.26x Titan V
cifar_resnet20_v2 1.21x Titan V
Alexnet 1.4x Titan V
https://cwiki.apache.org/confluence/display/MXNET/How+to+use+MXNet-TensorRT+integration

Inference Efficiency – NNVM
https://aws.amazon.com/blogs/machine-learning/introducing-nnvm-compiler-a-new-open-end-to-end-compiler-for-ai-frameworks/

Portability - NNVM
https://aws.amazon.com/blogs/machine-learning/introducing-nnvm-compiler-a-new-open-end-to-end-compiler-for-ai-frameworks/

Portability - ONNX
Model Parameters
Hyper Parameters

Deployment with Amazon SageMaker
I
ML Hosting Service
Amazon ECR
30 50
10 10
ProductionVariant
Model Artifacts
Inference Image
Model versions
Versions of the same
inference code saved in
inference containers.
Prod is the primary one,
50% of the traffic must be
served there!
One-Click!
EndpointConfiguration
Inference Endpoint
Amazon Provided Algorithms
Amazon SageMaker
InstanceType: c3.4xlarge
InitialInstanceCount: 3
ModelName: prod
VariantName: primary
InitialVariantWeight: 50

Recap

Agenda
Business Case
Framework
Model and Algorithms
Results
Productization
Key Learnings

Expense Report Automation

AP Invoice Automation
Machine Learning Machine Learning

Document scanning
CV and NLU to extract and understand text in varied documents
and pre-populate the appropriate fields

Framework Requirement
• Fast and agile prototyping and experiments
• Option to productize via Python and JVM language by exporting models
• Best accuracy
• Ideally - active support

Framework Choice (2017)
• Agile Python prototyping, production JVM language support
• MXNet
• Imperative
• ~50% faster training and inference speed + less memory consumption vs Tensorflow
• Better accuracy for our problem
• Sockeye (S2S)
• Easy to use
• Quick support
• Not Expected: Strong and active support from AWS MXNet team

MXNet vs. Tensorflow Accuracy (2017)
MXNet Tensorflow
Perfect Matching
Error Rate
11.6% 21.36%
Edit Distance
Error Rate
4% 7.75%
• Note: For our problem and models

Problems
Where are
the texts?
What do the
texts mean?
What are
the texts?

Where are the texts?
• Customized model based on You Only
Look Once (YOLO) network
• Added angle of tilt
• Used ResNet instead of CNN

What are the texts?
• Customized model based on
WaveNet (Originally intended for
speech)
• ResNet + Dilated Convolution
• Able to scan forward and backward

What do the texts mean?
(Rules-based system)
• Rules-Based System
• Looks into surrounding context
• Fuzzy Regex

Encoder-Decoder Seq2seq Approach
• Encoder-decoder model with
attention mechanism
• Character embedding to word
embedding
• Vocabulary built within training data
• Future work to use embeddings from
FastText/Word2vec

Benefit of Encoder-Decoder
Approach
• This approach is good at inferring
from context
• However, it requires you to have seen
the merchant in training data

CNN-based Box Classification Approach
• CNN-based model to detect
probability of each box
• Leverages word2vec word
embedding
• This approach considers both
language context and approximate
location of each field

Ensemble Strategy
• Easier fields – rules based only
• More difficult fields – perplexity
value decision
• Combined two 0.60 perfect
match model to get 72% on test
set with only 40k training data

Transferred Learning
• With limited real document data,
we trained on generated data.

Sample Business Documents

Results
• We have competitive results that’s at least on par, and on average performs
better when comparing to major public cloud providers on Vision part of the
model.
• Test Set metrics: ~75–85% accuracy
• Production metrics: ~95% accuracy

Productization

Key Learnings
• Value in building models specific for the product/use case
• Prototyping/Experimenting mindset vs. production mindset
• Importance of working with the framework team for challenges

Thank you!
Vivek Srivastava
Henry Zhang
Cyrus M Vahid
cyrusmv@amazon.com

Build Deep Learning Applications Using Apache MXNet - Featuring Chick-fil-A (AIM407-R1) - AWS re:Invent 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Build Deep Learning Applications Using Apache MXNet - Featuring Chick-fil-A (AIM407-R1) - AWS re:Invent 2018

Similar to Build Deep Learning Applications Using Apache MXNet - Featuring Chick-fil-A (AIM407-R1) - AWS re:Invent 2018 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Build Deep Learning Applications Using Apache MXNet - Featuring Chick-fil-A (AIM407-R1) - AWS re:Invent 2018