This project aims to scale up deep learning by extending the BIDMach machine learning framework to take advantage of computer clusters. The goals are to implement and benchmark novel deep learning algorithms, and adapt BIDMach's distributed communication to utilize multiple machines for faster performance. The project will work with industry partner OpenChai to bring machine learning to enterprises and consumers by adapting BIDMach to OpenChai's hardware.

1. Scaling Up
Deep Learning
on Clusters
Max Xie
Qualia Li
Aleks Kamko

2. Who are we?
Aleks did his undergrad in EECS at
Berkeley. He’s back for some Data
Science experience.
Max has a B.S. in Math and would like
to explore the world of data with it.
Qualia majored in Software
Engineering at ZJU. He is interested
in how big data works.
Qualia Aleks
Max

3. 2. What is “Machine Learning”!?
a. What is “Deep Learning”?
3. History + Status of Project
4. Scaling Up - What and How
5. Stakeholders
6. Roadmap
Agenda

4. Project
Overview
Scaling Up Deep Learning on Clusters
Extend the BIDMach machine learning
framework
Implement + benchmark novel deep
learning algorithms
Extend framework to take advantage of
clusters. More machines → faster

5. Project
Overview
Scaling Up Deep Learning on Clusters
Extend the BIDMach machine learning
framework
Implement + benchmark novel deep
learning algorithms
Extend framework to take advantage of
clusters. More machines → faster
Work with OpenChai to bring machine learning
to enterprise + consumers
Adapt BIDMach to OpenChai’s bleeding

6. Machine Learning 101
- Computer algorithms automatically learning
from data and information.

7. Machine Learning 101
- Computer algorithms automatically learning
from data and information.
History:
1950 - Alan Turing creates the “Turing Test”
1952 - First ML program, for checkers.
1990s - Shifts from knowledge-driven to data-
driven
1997 - IBM’s Deep Blue beats world champion
at chess.

8. So... Deep Learning?
First, Neural Networks:
Inspired by the structure and
functional aspects of biological
neural networks, modeling complex
relationships

9. So... Deep Learning?
First, Neural Networks:
Inspired by the structure and
functional aspects of biological
neural networks, modeling complex
relationships
1957 - The first neural network for
computers.

10. Deep Learning!!
= Deeper Neural Networks!
Focusing on different levels of
abstraction in representing objects.

11. Deep Learning!!
= Deeper Neural Networks!
Focusing on different levels of
abstraction in representing objects.
Applications: speech recognition,
computer vision, robotics, planning, etc.
BIDMach doesn’t have much
implementation for Deep Learning.

12. History of Project:
Overview
Target
Make the BID Data Suite the fastest Big Data
tool on the Internet
Components
Storage, CPU and GPU
BIDMat -- Matrix Algebra, Data Manipulation
BIDMach -- Machine Learning
Scaling Up
Butterfly Mixing
Sparse AllReduce

13. History of Project:
BIDMach and the BID Data Suite
BIDMach is a machine learning framework that is part of the larger BID Data
Suite. Motives for the suite include:
1. Exploratory data analysis → quickly sifting through data, making
hypotheses about structure, rapidly testing

14. History of Project:
BIDMach and the BID Data Suite
BIDMach is a machine learning framework that is part of the larger BID Data
Suite. Motives for the suite include:
1. Exploratory data analysis → quickly sifting through data, making
hypotheses about structure, rapidly testing
2. Rapid deployment and live tuning of models in commercial setting → very
high performance in both prototype and production setting

15. History of Project:
BIDMach and the BID Data Suite
BIDMach is a machine learning framework that is part of the larger BID Data
Suite. Motives for the suite include:
1. Exploratory data analysis → quickly sifting through data, making
hypotheses about structure, rapidly testing
2. Rapid deployment and live tuning of models in commercial setting → very
high performance in both prototype and production setting
3.Make single-machine algorithms fast first, improve using clusters later

16. 300%
BIDMach offers up to 300% gain in performance,
compared with state-of-art.
(John Canny et. al, 2015)

17. BIDMach offers up to 300% gain in performance,
compared with state-of-art.
(John Canny et. al, 2015)
Netflix Movie Recommendation
(A Matrix Factorization Problem)

18. Scaling Up
“Make single-machine algorithms fast first, improve using clusters later”
“The Cloud”

19. Scaling Up
“Make single-machine algorithms fast first, improve using clusters later”
Using only a single machine, BIDMach already beats most cluster-
based frameworks.

20. Scaling Up
“Make single-machine algorithms fast first, improve using clusters later”
Using only a single machine, BIDMach already beats most cluster-
based frameworks.
Our capstone project aims to extend BIDMach’s existing
algorithms to utilize the compute power of a cluster, making
BIDMach even faster!

21. “Scaling Up” to an
Enterprise Solution
Our main industry partner,
OpenChai, is working with us
to bring BIDMach and Machine
Learning to enterprise

22. “Scaling Up” to an
Enterprise Solution
Our main industry partner,
OpenChai, is working with us
to bring BIDMach and Machine
Learning to enterprise
● Retail
● Banking
● Genetics
● Mobile
● Home

23. “Scaling Up” to a
Consumer Solution
We’re also trying to bring
machine learning to your home
and local businesses!
Using mobile-phone GPUs and
CPUs → affordable, powerful,
and power-efficient

24. “Scaling Up” Goal 1:
Bring BIDMach to clusters.
“The Cloud”

25. “Scaling Up” Goal 2:
Bring clusters to your home!
“The Cloud”

26. Stakeholders
Co-developers
nVidia
ARM
OpenChai
Potential-Users
Google
Facebook
Twitter

27. Roadmap
1. Get familiar with the various systems
2. Implement Random Forests & K-Means
3. Improve BIDMach distributed
communication system
4. Working on model-parallel algorithms:
- Logistic regression
- Recurrent Neural Networks
- Convolutional Neural Networks
- Translational Model
Oct.
Dec.
Feb.
May
Technical Business

28. Roadmap
1. Get familiar with the various systems
2. Implement Random Forests & K-Means
3. Improve BIDMach distributed
communication system
4. Working on model-parallel algorithms:
- Logistic regression
- Recurrent Neural Networks
- Convolutional Neural Networks
- Translational Model
- Meet with OpenChai
- Get BIDMach working on OpenChai
hardware
- Running benchmarks
- Marketing “OpenChai + BIDMach”
Oct.
Dec.
Feb.
May
Technical Business

29. We will generally be co-working.
Aleks: Deal with the machines on cloud and how
they communicate.
Qualia: Learn Scala and write codes.
Max: Understand how variables are interrelated in
model-parallel algorithms
Roles
Qualia Aleks
Max

30. Revisiting
Project
Overview
Scaling Up Deep Learning on Clusters
Extend the BIDMach machine learning
framework
Implement + benchmark novel deep
learning algorithms
Extend framework to take advantage of
clusters. More machines → faster

31. Revisiting
Project
Overview
Scaling Up Deep Learning on Clusters
Extend the BIDMach machine learning
framework
Implement + benchmark novel deep
learning algorithms
Extend framework to take advantage of
clusters. More machines → faster
Work with OpenChai to bring machine learning
to enterprise + consumers
Adapt BIDMach to OpenChai’s bleeding

32. “A breakthrough
in machine
learning would be
worth ten
Microsofts. ”
- Bill Gates

33. Questions?