The document discusses several problems that are created by deep learning related to data operations and logistics, including a lack of support for the AI software development lifecycle, handling different workloads beyond just deep learning models, difficulties in putting machine learning models into production, running models in multiple locations, data dependencies being more costly than code dependencies, and changes in conditions over time. It provides recommendations on how to address these problems through approaches like stream-based architectures and containerization.

1. Data Operations Problems Created by Deep
Learning, and How to Fix Them!
Jim Scott
@kingmesal

2. 2 © 2018 MapR Technologies, Inc. // MapR Confidential
Public Service Announcement
You may see Artificial Intelligence, Machine
Learning and Deep Learning used interchangeably
within this presentation please feel free to
mentally substitute the phrase of your choice if
it is more applicable to you.
I’m not trying to split hairs on terminology.
Thanks for understanding!

3. 3 © 2018 MapR Technologies, Inc. // MapR Confidential
Terminology
Data Science
Artificial Intelligence
Machine Learning
Deep
Learning
Data Science
Artificial Intelligence (AI)
Machine Learning (ML)
The use of algorithms to extract knowledge and
insights from data in various forms in order to
obtain insights.
Some Subfields: Statistics, Artificial Intelligence
(AI), Computational Math
The simulation of intelligent human behavior for
problem-solving and decision-making.
Some Subfields: Robotics, Natural Language
Processing (NLP), Machine Learning.
The process by which machines are taught to make
calculated suggestions and/or predictions by
examining large amounts of input data.
Some Subfields: Logistic Regression, Deep Learning,
Reinforcement Learning

4. 4 © 2018 MapR Technologies, Inc. // MapR Confidential
90% of the effort in successful
machine learning isn’t in the
training or model development…
It’s the logistics!

5. 5 © 2018 MapR Technologies, Inc. // MapR Confidential
“Machine learning offers a
fantastically powerful toolkit for
building complex systems
quickly.… it is remarkably easy
to incur massive ongoing
maintenance costs at the
system level when applying
machine learning.”
The Importance of Data Logistics

6. 6 © 2018 MapR Technologies, Inc. // MapR Confidential
Why?
Just getting the training data is hard:
● Which data? How to make it accessible? Multiple sources!
● New kinds of observations force restarts
● Requires a ton of domain knowledge
The myth of a single model:
● You cannot train just one
● You will have dozens of models, likely hundreds or more
● Handoff to new versions is tricky
● You have to get runtime to be sure about which is better

7. 7 © 2018 MapR Technologies, Inc. // MapR Confidential
Problem 1
Lack of support for the
Artificial Intelligence
Software Development Lifecycle
(AI-SDLC)

8. 8 © 2018 MapR Technologies, Inc. // MapR Confidential
Building a Machine Learning Solution
1. Identify the data sources
2. Identify the tools
3. Write some code
4. Train a model
5. Test the model
6. Analyze the output of the tests
7. Repeat steps 3 through 6 until happy-ish
a. Maybe swap out a tool if your cannot achieve happiness
8. Figure out how to get this solution into production

9. 9 © 2018 MapR Technologies, Inc. // MapR Confidential
Choosing the Best Machine Learning Tool
Most successful groups keep several “favorite” machine learning tools on hand:
● No single tool is the best in every situation
The most important tool is a platform that supports logistics well:
● Everything does not need to be done at the application level
● Lots of what matters can be handled at the platform level
A good design for the logistics can make a big difference

10. 10 © 2018 MapR Technologies, Inc. // MapR Confidential
Problem 2
The deep learning
workload is only one
type of workload

11. 11 © 2018 MapR Technologies, Inc. // MapR Confidential
Massaging your data
● This will normally include cleansing, normalizing, and even optimizing data formats
for downstream consumption for GPU based workloads
● Distributed compute is often the best approach for this type of activity due to the
volume of data and variety of data types
● Be sure to keep your original data, in case of mistakes
Separate your training and holdback data sets
● This is based off of the massaged data
Analyze model outputs to determine the quality of your models
● Especially valuable over time to know that the models are moving the right way
● Great candidate for a distributed workload
Workloads

12. 12 © 2018 MapR Technologies, Inc. // MapR Confidential
Problem 3
Putting machine learning into
production is not quite the same
as other enterprise software

13. 13 © 2018 MapR Technologies, Inc. // MapR Confidential
Gotchas with Making it to Production
● Ops-oriented people will not necessarily “get it” regarding modeling subtleties
● Data scientists will not necessarily “get it” regarding operational realities
● Therefore, modelers have to deliver self-contained models
● And, ops has to provide pre-wired structure

14. 14 © 2018 MapR Technologies, Inc. // MapR Confidential
Handling Real-time
Stream instead of database as the shared “truth”
Image © 2016 Ted Dunning & Ellen Friedman from Chap 6 of O’Reilly book Streaming Architecture used with permission

15. 15 © 2018 MapR Technologies, Inc. // MapR Confidential
Real-time can be started on
your schedule, that is the key

16. 16 © 2018 MapR Technologies, Inc. // MapR Confidential
Problem 4
Running machine learning
models in more than
one location is tough

17. 17 © 2018 MapR Technologies, Inc. // MapR Confidential
Streaming Isolates Services

18. 18 © 2018 MapR Technologies, Inc. // MapR Confidential
With MapR, Geo-Distributed Data Appears Local

19. 19 © 2018 MapR Technologies, Inc. // MapR Confidential
With MapR, Geo-Distributed Data Appears Local
Global Data Center
Regional Data Center

20. 20 © 2018 MapR Technologies, Inc. // MapR Confidential
Features of Good Streaming
Persistent
● Messages stick around for other consumers
● Consumers don’t affect producers
● Consumer doesn’t have to be online when message arrives
Performant
● You should NEVER need to worry if a stream can keep up
Pervasive
● It is there whenever you need it, no need to deploy anything
● How much work is it to create a new file? Why harder for a stream?

21. 21 © 2018 MapR Technologies, Inc. // MapR Confidential
Improving Machine Learning Logistics
Stream first architecture is a powerful approach with surprisingly widespread
advantages
● Innovative technologies emerging to for streaming data
Microservices approach provides flexibility
● Streaming supports microservices (if done right)
Containers remove surprises
● Predictable environment for running models

22. 22 © 2018 MapR Technologies, Inc. // MapR Confidential
Problem 5
Data dependencies cost more
than code dependencies,
a lot more!

23. 23 © 2018 MapR Technologies, Inc. // MapR Confidential
Data dependencies cost more than code dependencies!
● Code dependencies are easy to track, because it is a well known and a well
practiced discipline
● The data may be unstable
Undeclared consumers can wreak havoc on your models
● Downstream users may create a data dependency on the data from your model
● Updates to your model may break their system, if they made an assumption on
the function of your model
● Who do you think will suffer?
Top Reason to Use a Streaming Architecture

24. 24 © 2018 MapR Technologies, Inc. // MapR Confidential
First Look with Streams

25. 25 © 2018 MapR Technologies, Inc. // MapR Confidential
Then Rendezvous

26. 26 © 2018 MapR Technologies, Inc. // MapR Confidential
Faster Throughput Through Failure
Suppose we have one model that can handle 10,000 t/s @ 2ms
● But this isn’t the most accurate model. Not bad, but not the best.
And our champion model can handle 1,000 t/s @ 10ms
● Then imagine a burst of 2,000 t/s for several minutes
Champion can only evaluate half of all requests
● Should skip to keep up
● Fast model will cover for champion

27. 27 © 2018 MapR Technologies, Inc. // MapR Confidential
Rendezvous – Mainly for Making Decisions
Decisioning models
● Looking for a “right answer”
● Simpler than reinforcement learning
Examples
● Fraud detection
● Predictive analytics / market prediction
● Churn prediction (as in telecommunications)
● Yield optimization
● Deep learning in form of speech or image recognition, in some cases

28. 28 © 2018 MapR Technologies, Inc. // MapR Confidential
Some Key Points
● Note that all models see identical inputs
● All models run in production setting
● All models send scores to same stream
● The rendezvous server decides which scores to ignore
● Roll forward, roll back, correlated comparison are all now trivial

29. 29 © 2018 MapR Technologies, Inc. // MapR Confidential
Problem 6
Wash,
Rinse,
Repeat!

30. 30 © 2018 MapR Technologies, Inc. // MapR Confidential
Are you performing all of these steps in your AI-SDLC manually?
● Consider a workflow tool
○ e.g. Airflow, Kubeflow, Argo, etc…
Is all of your data in static files or will there be real-time data?
● Prepare for real-time in development to be ready for production
Version everything!
● I’m sorry, this isn’t a job for GIT!
● Includes source data: static and real-time
● Also includes models and their output
● Ensures sanity checks
● Long-term performance analytics
Concerns About Repeatability

31. 31 © 2018 MapR Technologies, Inc. // MapR Confidential
Quality & Reproducibility of Input Data is Important!
Recording raw-ish data is really a big deal
● Data as seen by a model is worth gold
● Data reconstructed later often has time-machine leaks
● Databases were made for updates, streams are safer
Raw data is useful for non-ML cases as well (think flexibility)
Decoy model records training data as seen by models under development &
evaluation

32. 32 © 2018 MapR Technologies, Inc. // MapR Confidential
A Quick Review

33. 33 © 2018 MapR Technologies, Inc. // MapR Confidential
The Proxy Talks to the Outside World

34. 34 © 2018 MapR Technologies, Inc. // MapR Confidential
The Input Stream Feeds All Models Identically

35. 35 © 2018 MapR Technologies, Inc. // MapR Confidential
The Scores Stream Contains All Results

36. 36 © 2018 MapR Technologies, Inc. // MapR Confidential
The Rendezvous Picks A Result

37. 37 © 2018 MapR Technologies, Inc. // MapR Confidential
Results Return Via A Stream and Return Address

38. 38 © 2018 MapR Technologies, Inc. // MapR Confidential
Problem 7
In the real world
conditions may
(will) change!

39. 39 © 2018 MapR Technologies, Inc. // MapR Confidential
Not Such Bad Ideas
Keep models running “in the wings”
● Do not wait until conditions change to start building the next model
● Keep new short-history models ready to roll, some graybeards as well
Hot hand-off
● With rendezvous, stop ignoring the new best model
Deploy a canary server
● Keep an old model active as a reference
● If it was 90% correct, difference with any better model should be small
● Score distribution should be roughly constant

40. 40 © 2018 MapR Technologies, Inc. // MapR Confidential
Prepare for Scaling Up
Model Variety
● Multiple rendezvous frameworks for different tasks
Throughput
● Fast default models
● Partition input stream to allow parallel model evaluation
● Input batching
Extreme Volumes
● Cannibalize fancy models to run more fast/simple models
● Speed before beauty

41. 41 © 2018 MapR Technologies, Inc. // MapR Confidential
Making Improvements
1. Data + the right question + domain knowledge, matters!
2. Prioritize – put serious effort into infrastructure
a. DataOps requires more than just data science
3. Persist – use streams to keep data around
4. Measure – everything, and record it
5. Analyze – understand and see what is happening
6. Containerize – make deployment predictable, repeatable and easy

42. 42 © 2018 MapR Technologies, Inc. // MapR Confidential
Problems 8, 9 and 10
Copying data from your
streaming system,
data lake,
and edge systems to your
machine learning environment

43. 43 © 2018 MapR Technologies, Inc. // MapR Confidential
PLEASE, PLEASE, PLEASE…
...tell me you are not copying
all your data between these systems

44. 44 © 2018 MapR Technologies, Inc. // MapR Confidential
Storage
Appliance
Traditional Storage Vendor Solution
Edge
Copy
Ingest
Core Cloud
Unified Data
Lake
Data Prep
Training
+
Testing
Production
Training
Cluster Deployment
Copy
Storage
Appliance
ServersServers w/
GPU
Does NOT support real-time workflows
Doesn’t support distributed data preparation workloads
Copy
Copy

45. 45 © 2018 MapR Technologies, Inc. // MapR Confidential
Hadoop Based Solutions
Edge
Copy
Core Cloud
Unified Data
Lake
Data Prep
Training
+
Testing
Production
Training
Cluster Deployment
HDFS
Cluster
ServersServers w/
GPU
Minimum of seven non-homogeneous environments to administer and secure
Full data copies without versioning, lineage control or multi-master support
Copy
Kafkain-motion
Copy
Copy
Copy
in-motion Kafka
in-motion
Copy
Copy
Copy
Ingest
Kafka
Where does the
master copy of
the data live?

46. 46 © 2018 MapR Technologies, Inc. // MapR Confidential
MapR Solution
Data Fabric
Global Namespace
Core CloudEdge
Data Prep
Training
+
Testing
Deployment
One homogeneous environment to manage and secure
Supports real-time processing with data protection, lineage, and versioning
Runs directly on DGX servers to create a unified DGX cluster

47. 47 © 2018 MapR Technologies, Inc. // MapR Confidential
MapR AI + RAPIDS
Document
DB
Events
Structured
Data
Unstructured
Data
Inference
Typical Training and Evaluation Workflow
Events
Production DeploymentData Management
Applications
RAPIDS
Apache
Arrow GPU Memory
cuGRAPH
Graph Analytics
cuML
Machine Learning
cuDF
Analytics
Data
Preparation
Training
Data Set
Model
Training
Evaluate/
Visualize

48. 48 © 2018 MapR Technologies, Inc. // MapR Confidential
How Data is Accessed
Advantages of the MapR Data Fabric
● Linear Scalability
● Architected for performance, scale,
and reliability
● Distributed metadata in the fabric
How Data is Stored
How Data is Accessed
● Distributed location support
● Multi-master Replication
● Location awareness
How Data is Distributed
● Capability to serve as a system of record
● Data security and governance within the
fabric
● Mixed Data access from multiple
protocols
● Distributed Multi-tenancy
● Global Namespace
● Integrated data streaming for AI

49. 49 © 2018 MapR Technologies, Inc. // MapR Confidential
On-premise or Cloud Infrastructure
• Combines Distributed
Compute, AI, HPC, and
general purpose
workloads
• MapR provides
complementary data
logistics to better manage
and deploy deep learning
across the entire ecosystem
• Enables deployment agility
with data management
extending from on-premise,
to cross-cloud, to the edge
Architecture Matters

50. 50 © 2018 MapR Technologies, Inc. // MapR Confidential
Simplified administration and security models
● One and done - no need for a different model in each location
● GDPR “compliant”!
Scales linearly with customer needs
● No reason to create a bunch of separate clusters
Sustainability - All data, files, database and event streaming
● Both at-rest and in-motion
An enabling and flexible architecture
● Only way to bring distributed data and GPUs together
● Easy to meet customers needs
● Supports both kubernetes and containers
Low cost of entry and linear cost of scaling
MapR Advantages for AI

51. 51 © 2018 MapR Technologies, Inc. // MapR Confidential
Same platform and architecture in all locations:
● On-premise works the same as the cloud
● Second cloud works the same as a first cloud
● Data mirroring between locations is built-in
● Real-time event management and lineage is built-in
○ Scale out streaming applications without rearchitecting them
● Kubernetes is a simple way to inject MapR storage and GPU support into a
container
○ Leverage resources anywhere with Global Namespace
○ Application portability across all locations, no rework required
On-Premise, Cloud or Both

52. 52 © 2018 MapR Technologies, Inc. // MapR Confidential
Complex data pipelines, large data volumes serving GPUs
● Mixed workloads - distributed data prep plus real-time
Simultaneous data and model versioning
● Data at-rest and in-motion
Model output lands in a stream
● Creates pluggable model flow
Works across on-premise and cloud infrastructures, simultaneously
Simplifying Model Development and Deployment

53. 53 © 2018 MapR Technologies, Inc. // MapR Confidential
“90+% of Machine Learning
Success is Data Logistics”
https://mapr.com/ebook/machine-learning-logistics
The Key is Data Logistics

54. 54 © 2018 MapR Technologies, Inc. // MapR Confidential
● Over 35 FREE on-demand training courses for AI and analytic development, data
engineering and administration
● Certification tracks for developers, administrators, and data scientists
● Expanded support portal and knowledge base
● Containerized clusters, for free download, solution templates and code examples
for hands-on experience
https://mapr.com/training/
Need Help Solving Your Data Logistics Problems?

55. Thank you!