Apache SystemML is a machine learning framework that allows users to write machine learning algorithms in a declarative way using a high-level language similar to R or Python. It includes a compiler and optimizer that can translate the high-level code into optimized low-level operations. The framework supports both single-machine and distributed execution using Spark. Current work is focused on improving usability for deep learning tasks and adding support for GPUs and new backends like Flink.

1. Apache SystemML
Mike Dusenberry
Engineer, Machine Learning & SystemML
Spark Technology Center
@dusenberrymw
Datapalooza, Denver - 05.19.16

2. Apache
SystemML
1. Background
a. Machine Learning
b. Declarative ML
2. SystemML
a. Overview
b. Language
c. Compiler/Optimizer
d. Runtime
3. Demo
4. Current Work
a. Deep Learning: SystemML-NN
5. Questions
Agenda

3. Links
● Main Website:
systemml.apache.org
● Code:
github.com/apache/incubator-systemml
● Documentation:
apache.github.io/incubator-systemml
● JIRA:
issues.apache.org/jira/browse/SYSTEMML

4. Machine Learning

5. Machine Learning
● Data
○ Multiple “examples”
○ Multiple “features” per “example”
○ “Label(s)” for each “example” (supervised)
● Model
○ Construct/select a model that fits the problem.
○ Examples:
■ Linear/Logistic Regression
■ SVM
■ Neural Networks
● Loss
○ An “evaluation” of how well the model fits the data.
● Optimizer
○ Minimize “loss” by adjusting model to better fit the data.

6. Declarative Machine Learning

7. Laptop
Exploratory Data Analysis Today
R
Python
Others
Data
Scientist
DataR
Python
Others
Data
Scientist

8. Laptop
Exploratory Data Analysis Today
R
Python
Others
Data
Scientist
R
Python
Others
Data
Scientist

9. Current Best Practice for Big Data Analysis
Data
Scientist
Data
Scientist
Data
Scientist
Hadoop
Engineer
Spark
Engineer
MPI
Engineer
R
Python
Others

10. Laptop
Data
Scientist
Scale-up
Cluster
R
Python Query
Optimization
Others
Vision: Declarative Machine Learning

11. Common patterns:
•Changes in feature set
•Changes in data size
•Algorithm customization
•Quick iteration
Declarative Machine Learning

12. Classification by level of abstraction (different target user)
Landscape of Existing Work
Distributed Systems w/ DSLs
Large-Scale ML Libraries (fixed plan)
Declarative ML (fixed algorithm)
Declarative ML++ (fixed task)
Spark, Flink, REEF, GraphLab,
(R, Matlab, SAS)
MLlib, Mahout MR, MADlib, ORE,
Rev R, HP Dist R, Custom alg.
SystemML, (Mahout Samsara,
Tupleware, Cumulon, Dmac, SimSQL)
Mlbase*, Specific sys.

13. Requirements to Support Declarative ML
• Goal: Write ML algorithms independent of input data and cluster characteristics.
• R1: Full flexibility
▪ Specify new / customize existing ML algorithms.
▪ ➔ ML DSL
• R2: Data independence
▪ Hide physical data representation (sparse/dense, row/column-major, blocking
configs, partitioning, caching, compression).
▪ ➔ Abstract data types and coarse-grained logical operations.
• R3: Efficiency and scalability
▪ Very small to very large use-cases.
▪ ➔ Automatic optimization and hybrid runtime plans.
• R4: Specified algorithm semantics
▪ Understand, debug, and control algorithm behavior.
▪ ➔ Optimization for performance only, not accuracy.

14. Apache SystemML

15. Sidenote: Fun Stuff - Neural Art
-A Neural Algorithm of Artistic Style, L.A.
Gatys, A.S. Ecker, M. Bethge
-https://github.com/jcjohnson/neural-style

16. Apache SystemML

17. Apache SystemML
● High-level language
○ DML -> R-like
○ PyDML -> Python-like
○ Focus is on matrices and
linear algebra.
● Engine
○ Compiler/Optimizer
○ Lots of optimizations, such as
rewrites.
● Runtime
○ Laptop
○ Spark
○ (also Hadoop)
(DML) (PyDML)
Engine

18. Apache SystemML
● High-level language
○ DML -> R-like
○ PyDML -> Python-like
○ Focus is on matrices and
linear algebra.
● Engine
○ Compiler/Optimizer
○ Lots of optimizations, such as
rewrites.
● Runtime
○ Laptop
○ Spark
○ (also Hadoop)
(DML) (PyDML)
Engine

19. SystemML - Example: Logistic Regression (DML)

20. SystemML - Example: Sigmoid Function (DML)

21. Apache SystemML
● High-level language
○ DML -> R-like
○ PyDML -> Python-like
○ Focus is on matrices and
linear algebra.
● Engine
○ Compiler/Optimizer
○ Lots of optimizations, such as
rewrites.
● Runtime
○ Laptop
○ Spark
○ (also Hadoop)
(DML) (PyDML)
Engine

22. Apache SystemML
● High-level language
○ DML -> R-like
○ PyDML -> Python-like
○ Focus is on matrices and
linear algebra.
● Engine
○ Compiler/Optimizer
○ Lots of optimizations, such as
rewrites.
● Runtime
○ Laptop
○ Spark
○ (also Hadoop)
(DML) (PyDML)
Engine

23. SystemML - Compilation Chain

24. SystemML - Compilation Chain
24

25. SystemML - Compilation Chain
25

26. SystemML - Compilation Chain
26

27. SystemML - Compilation Chain
27

28. Apache SystemML
● High-level language
○ DML -> R-like
○ PyDML -> Python-like
○ Focus is on matrices and
linear algebra.
● Engine
○ Compiler/Optimizer
○ Lots of optimizations, such as
rewrites.
● Runtime
○ Laptop
○ Spark
○ (also Hadoop)
(DML) (PyDML)
Engine

29. Apache SystemML
● High-level language
○ DML -> R-like
○ PyDML -> Python-like
○ Focus is on matrices and
linear algebra.
● Engine
○ Compiler/Optimizer
○ Lots of optimizations, such as
rewrites.
● Runtime
○ Laptop
○ Spark
○ (also Hadoop)
(DML) (PyDML)
Engine

30. More Fun...
https://github.com/google/deepdream

31. Apache SystemML
● High-level language
○ DML -> R-like
○ PyDML -> Python-like
○ Focus is on matrices and
linear algebra.
● Engine
○ Compiler/Optimizer
○ Lots of optimizations, such as
rewrites.
● Runtime
○ Laptop
○ Spark
○ (also Hadoop)
(DML) (PyDML)
Engine

32. SystemML - Compilation Chain
32

33. SystemML - Compilation Chain
33
Spark
CP + b sb _mVar1
SPARK mapmm X.MATRIX.DOUBLE _mvar1.MATRIX.DOUBLE
_mVar2.MATRIX.DOUBLE RIGHT false NONE
CP * y _mVar2 _mVar3

34. Apache SystemML
● High-level language
○ DML -> R-like
○ PyDML -> Python-like
○ Focus is on matrices and
linear algebra.
● Engine
○ Compiler/Optimizer
○ Lots of optimizations, such as
rewrites.
● Runtime
○ Laptop
○ Spark
○ (also Hadoop)
(DML) (PyDML)
Engine

35. SystemML Architecture (APIs and runtime)
35
Command
Line
JMLC
Spark
MLContext
Spark
ML
APIs
High-Level Operators (HOPs)
Parser/Language
Low-Level Operators (LOPs)
Compiler
Runtime
Control Program
Runtime
Prog
Buffer Pool
ParFor Optimizer/Runtime
MR InstSpark
Inst
CP
Inst
Recompiler
Cost-based
optimizations
DFS IOMem/FS IO
Generic
MR
MatrixBlock Library
(single/multi-threaded)

36. SystemML Architecture (APIs and runtime)
36
Command
Line
JMLC
Spark
MLContext
Spark
ML
APIs
High-Level Operators (HOPs)
Parser/Language
Low-Level Operators (LOPs)
Compiler
Runtime
Control Program
Runtime
Prog
Buffer Pool
ParFor Optimizer/Runtime
MR InstSpark
Inst
CP
Inst
Recompiler
Cost-based
optimizations
DFS IOMem/FS IO
Generic
MR
MatrixBlock Library
(single/multi-threaded)

37. Demo

38. Current Work

39. Current Work
● Usability / Applications:
○ Deep Learning (SYSTEMML-540)
○ Embedded Scala/Python/R DSL with sufficient optimization scope (SYSTEMML-451)
● Optimizer:
○ Cost-model enhancement (SYSTEMML-416)
○ Global program optimization (SYSTEMML-421)
○ Source code generation for automatic operator fusion (SYSTEMML-448)
● Runtime:
○ Add GPU backend (SYSTEMML-445) => CUDA / OpenCL
○ Frame support / Sparse block representation
○ Integrate Apache Flink as additional backend for SystemML (SYSTEMML-636 / PR-119)
○ NUMA-aware single node backend (SYSTEMML-406)

40. Deep Learning - Plans
● Deep Learning library for SystemML written in DML (SYSTEMML-618).
○ SystemML-NN [https://github.com/dusenberrymw/systemml-nn]
● Built-in DML functions for computationally-intensive layers.
○ Convolution (2D), Max Pooling
● GPU acceleration for these built-in functions (SYSTEMML-445).
● Integration with existing deep learning libraries (Keras, TensorFlow, Torch,
etc.)?

41. Deep Learning - SystemML-NN Library
● Deep learning library written in DML (and
PyDML soon…).
● Multiple layers:
○ Core:
■ Affine, 2D Convolution, Max Pooling
○ Nonlinearity/Transfer:
■ Sigmoid, Tanh, Softmax, ReLU
○ Regularization:
■ Dropout, L1, L2
○ Loss:
■ Log-loss, Cross-entropy, L1, L2
● Multiple optimizers:
○ SGD, SGD w/ momentum, SGD w/
Nesterov momentum, Adagrad, RMSprop,
Adam
https://github.com/dusenberrymw/systemml-nn

42. Deep Learning - SystemML-NN Library (cont.)
https://github.com/dusenberrymw/systemml-nn
● Each layer type has a simple `forward(...)
` and `backward(...)` API.
○ `forward(...)` computes the output of the
function based on the inputs.
○ `backward(...)`computes the partial
derivatives (gradient) of the inputs to the
function w.r.t. some function deeper in the
network (usually the loss function at the
end).
● Each optimizer has a simple `update(...)`
API.
○ `update(...)` adjusts the given parameters
based on their partial derivatives.
● Includes test code in DML.
○ Gradient checks, unit tests

43. Deep Learning - SystemML-NN Library (cont.)
SystemML-NN
SystemML
Engine

44. Apache
SystemML
1. Background
a. Machine Learning
b. Declarative ML
2. SystemML
a. Overview
b. Language
c. Compiler/Optimizer
d. Runtime
3. Demo
4. Current Work
a. Deep Learning: SystemML-NN
5. Questions
Agenda Revisited

45. Links
● Main Website:
systemml.apache.org
● Code:
github.com/apache/incubator-systemml
● Documentation:
apache.github.io/incubator-systemml
● JIRA:
issues.apache.org/jira/browse/SYSTEMML

46. Questions?

47. Thanks!