Distributed Deep Learning: Methods and Resources This document discusses distributed deep learning methods and resources. It provides an overview of deep learning and stochastic gradient descent (SGD), and how they can be parallelized using data and model parallelism. It describes Neuromation, a platform developing a worldwide marketplace for knowledge mining using distributed computational resources. Neuromation will use blockchain and its TokenAI token to combine synthetic data generation, distributed training of neural networks, and payment for computational work into a single decentralized platform.

1. Distributed Deep Learning:
Methods and Resources
Sergey Nikolenko
Chief Research Officer, Neuromation
Researcher, Steklov Institute of Mathematics at St. Petersburg
September 23, 2017, AI Ukraine, Kharkiv
Maxim Prasolov
CEO, Neuromation

2. Outline
● Bird’s eye overview of deep learning
● SGD and how to parallelize it
● Data parallelism and model parallelism
● Neuromation: developing a worldwide marketplace
for knowledge mining

3. ● 10 years ago machine learning underwent a deep learning revolution
● Neural networks are one of the oldest techniques in ML
● But since 2007-2008, we can train large and deep neural networks
(in part due to distributed computations)
● And now deep NNs yield state of the art results in many fields

4. What is a deep neural network
● A deep neural network is a huge
composition of simple functions
implemented by artificial neurons
● Usually linear combination
followed by nonlinearity, but can be
anything as long as you can take
derivatives
● These functions are combined into a
computational graph that computes
the loss function for the model

5. Backpropagation
● To train the model (learn the weights),
you take the gradient of the loss
function w.r.t. weights
● Gradients can be efficiently computed
with backpropagation
● And then you can do (stochastic)
gradient descent and all of its
wonderful modifications, from
Nesterov momentum to Adam

6. FEEDFORWARD NETWORKS CONVOLUTIONAL NETWORKS RECURRENT NETWORKS
Gradient descent is used for all kinds of neural networks

7. Distributed Deep Learning: The Problem
● One component of the DL revolution
was the use of GPUs
● GPUs are highly parallel (hundreds of cores)
and optimized for matrix computations
● Which is perfect for backprop (and fprop too)
● But what if your model does not fit on a GPU?
● Or what if you have multiple GPUs?
● Can we parallelize further?

8. What Can Be Parallel
● Model parallelism vs. data parallelism
● We will discuss both
● Data parallelism is much more common
● And you can unite the two:
[pictures from (Black, Kokorin, 2016)]

9. Examples of data parallelism
● Make every worker do its thing and
then average the results
● Parameter averaging: average w from all workers
○ but how often?
○ and what do we do with advanced SGD variants?
● Asynchronous SGD: average updates from workers
○ much more interesting without synchronization
○ but the stale gradient problem
● Hogwild (2011): very simple asynchronous SGD,
just read and write to shared memory, lock-free;
whatever happens, happens

10. Examples of data parallelism
● FireCaffe:
○ DP on a GPU cluster
○ communication through
reduction trees

11. Model parallelism
● In model parallelism, different weights are distributed
● Pictures from the DistBelief paper (Dean et al., 2012)
● Difference in communication:
○ DP: workers exchange weight updates
○ MP: workers exchange data updates
● DP in DistBelief: Downpour SGD
vs. Sandblaster L-BFGS
● Now, DistBelief has been
completely replaced by...

12. Distributed Learning in TensorFlow
● TensorFlow has both DP (right) and MP (bottom)
● Workers and parameter servers
● MP usually works as a pipeline between layers:

13. First specify the structure of the cluster: Then assign (parts of) computational graph
to workers and weights to parameter servers:
Example of Data Parallelism in TensorFlow

14. Interesting variations
● (Zhang et al., 2016) – staleness-aware SGD: add weights depending on
the time (staleness) to updates
● Elephas: distributed Keras that runs on Spark
● (Xie et al., 2015) – sufficient factor broadcasting:
represent and send only u and v
● (Zhang et al., 2017) – Poseidon: a new architecture with
wait-free backprop and hybrid communication

15. ● Special mention: reinforcement learning; async RL is great!
● And standard (by now) DQN tricks are perfect for parallelization:
○ experience replay: store experience in replay memory
and serve them for learning
○ target Q-network is separate from the
Q-network which is learning now, updates are rare
Distributed reinforcement learning

16. Gorila from DeepMind: everything is parallel and asynchronous

17. Recap
● Data parallelism lets you process lots of data in parallel, copying the model
● Model parallelism lets you break down a large model into parts
● Distributed architectures are usually based on parameter servers and workers
● Especially in reinforcement learning, where distributed architectures rule
● And this all works out of the box in TensorFlow and other modern frameworks
● But how is it relevant to us? Isn’t that for the likes of Google and/or DeepMind?
● Where do we get the computational power and why do we need so much data?
Distributed deep learning works

18. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR
Chris
NOT ENOUGH LABELED DATA
FOR NEURAL NETWORK
TRAINING
“BOTTLENECK” OF AUTOMATION
OF EVERY INDUSTRY:

19. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR
IMAGE RECOGNITION IN RETAIL

20. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR
170.000 OBJECTS
ABOUT 40 BLN IMAGES PER YEAR
TO AUTOMATE THE RETAIL INDUSTRY
MUST BE RECOGNIZED ON THE SHELVES
IMAGE RECOGNITION IN RETAIL BY ECR RESEARCH:
OSA HP CONTRACTED NEUROMATION TO PRODUCE
LABELED DATA AND TO RECOGNIZE
30% OF THE COST IS COMPUTATIONAL POWER.
MORE THAN 7 MLN EURO REVENUE

21. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR
Chris
LABELED PHOTOS ARE REQUIRED
TO TRAIN IMAGE RECOGNITION MODELS
MORE THAN
1 BLN
WHERE CAN WE GET THIS HUGE AMOUNT OF LABELED DATA?

22. Chris
1 MAN = 8 HOURS x 50 IMAGES,
x $0.2 PER IMAGE
YEARS OF MECHANICAL WORK
1 BLN LABELED PHOTOS = $240 MLN
DATA LABELING
HAS BEEN MANUAL WORK TILL NOW

23. WE KNOW HOW TO GENERATE SYNTHETIC
LABELED DATA FOR DEEP LEARNING

24. ● Labeled data with 100% accuracy
● Automated data generation with no limits
● Cheaper and faster than manual labor
SYNTHETIC DATA:
A BREAKTHROUGH IN
DEEP LEARNING
BUT REQUIRES HUGE COMPUTATIONAL POWER
TO RENDER DATA AND TRAIN NEURAL NETWORKS

25. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR
BITCOIN OR ETHER
MINING
AMAZON DEEP
LEARNING
$7-8
per DAY
$3-4
per HOUR
The AI industry is ready to
pay miners for their
computational resources
more than they can ever get
from mining Ether.
KNOWLEDGE MINING IS MORE PROFITABLE.
DEEP LEARNING NEEDS YOUR COMPUTATION POWER!
WE CAN BRIDGE THIS GAP
GPU
x6
GPU

26. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR
BLOCKCHAIN + DEEP LEARNING

27. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR
NEUROMATION
PLATFORM
TokenAI
will combine in one place all the components
necessary to build deep learning solutions
with synthetic data
THE UNIVERSAL MARKETPLACE OF
NEURAL NETWORK DEVELOPMENT

28. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR

29. ● the network must be geographically distributed and keep track of
massive amounts of transactions
● the payment method for completed work should be highly liquid and
politically independent
● network nodes have to understand the model of “mining” a resource for a bounty:
transparency is required to build trust
● transactions need to be transparently auditable to prevent fraud and mitigate dispute
NEUROMATION PLATFORM will be extending Etherium with TokenAI.
Blockchain is the only technology that can realistically accomplish this.
Extending Ethereum instead of building our own blockchain is an obvious first step.
Neuromation needs to quickly deploy
a massive network of computation nodes (converted from crypto miners).

30. BITCOIN OR
ETHER MINING
AMAZON DEEP
LEARNING
$7-8 USD
per DAY
$3-4 USD
HOUR
DEEP LEARNING
RESEARCH GRANTS
● for R&D teams and start-up’s of DL/ML
industry
● In cooperation with frontier institutions
WE ARE OPEN FOR COOPERATION
partnerships@neuromation.io
● 1000 GPUs pool (+100,000GPU are
coming)

31. VAST APPLICATIONS OF SYNTHETIC DATA
NEUROMATION LABS:
RETAIL AUTOMATION
LAB
PHARMA AND
BIOTECH LAB
synthetic data for:
ENTERPRISE
AUTOMATION LAB
synthetic data for:
● +170 000+ items for the
Eastern European Retail
Market only
● about 50 euro per object
● contract for >7mln euro in
revenue
● medical imaging (classify
tumors and melanomas)
● health applications (smart
cameras)
● training flying drones,
self-driving cars, and
industrial robots in virtual
environments
● manufacturing and
supply-chain solutions
(live)

32. OUR TEAM:
Maxim Prasolov
CEO
Fedor Savchenko
CTO
Sergey Nikolenko
Chief Research Officer
Denis Popov
VP of Engineering
Constantine Goltsev
Investor / Chairman
Andrew Rabinovich
Adviser
Yuri Kundin
ICO Compliance
Adviser
Aleksey Spizhevoi
Researcher
Esther Katz
VP Communication
Kiryl Truskovskyi
Lead Researcher

33. OCTOBER 15th, 2017 Presale of TOKENAI starts
NOVEMBER, 2017 Public sale starts
UNKNOWN DATE Secret cap is reached, and token sale ends in 7 days
Jan 1st, 2018 Token sale ends (if secret cap is not reached)
ICO ROADMAP

34. ● Tokens Minted: 1,000,000,000
● Issued in ICO: Up to 700,000,000
● Reserve: from 300,000,000
● Price per Token: 0.001 ETH
ICO DETAILS
Jurisdiction: Estonia, EU
* Neuromation is fully compliant with Estonian crowdfunding legislation

35. KNOWLEDGE MINING - A NEW ERA OF
DISTRIBUTED COMPUTING
THANK YOU!
neuromation.io