GraphPipe is an open source protocol and collection of software designed to simplify machine learning model deployment and decouple it fromframework-specific model implementations.

2. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 2
Vish Abrams
Architect, Cloud Develompent, Oracle
October 9, 2018
Blazingly Fast Machine Learning Inference

3. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Machine Learning Model Serving
What is GraphPipe?
Advantages
Protocol Deep Dive
Real World Demo
More Info
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 3
5
6

4. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Machine Learning Inference
What is GraphPipe?
Performance
Protocol Deep Dive
Real World Demo
More Info
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 4
5
6

5. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Machine Learning Inference (Model Serving)
• Building machine learning models has become much easier due to open
source frameworks like TensorFlow and Pytorch
• Serving machine learning models means putting your trained model onto a
server so that it can be accessed by client applications
• This involves two components: the ML client and the ML server.
• The client talks to the server using some kind of communication protocol:
often JSON over HTTP.
Confidential – Oracle Internal/Restricted/Highly Restricted 5

6. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
ML Client
Confidential – Oracle Internal/Restricted/Highly Restricted 6

7. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
ML Server
Confidential – Oracle Internal/Restricted/Highly Restricted 7

8. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Machine Learning Model Serving
What is GraphPipe?
Advantages
Protocol Deep Dive
Real World Demo
More Info
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 8
5
6

9. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What is GraphPipe?
GraphPipe is an open source protocol and collection of software designed
to simplify machine learning model deployment and decouple it from
framework-specific model implementations.
Confidential – Oracle Internal/Restricted/Highly Restricted 9

10. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What is GraphPipe?
In other words, it turns this:
Confidential – Oracle Internal/Restricted/Highly Restricted 10
mxnet servertensorflow serving custom server
standard jsoncustom protocolprotocol buffers
custom clientautogenerated client custom client

11. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
What is GraphPipe?
Confidential – Oracle Internal/Restricted/Highly Restricted 11
Into this:
graphpipe-onnxgraphpipe-tf

12. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
GraphPipe Features
• A minimalist machine learning transport specification based on flatbuffers
• Simple reference model servers for Tensorflow, Caffe2, and ONNX.
• Efficient client implementations in Go, Python, and Java.
Confidential – Oracle Internal/Restricted/Highly Restricted 12

13. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Why Did we Make It?
• Production deployments of AI agents are around the corner
• Model Serving is an important part of production solutions
• Existing solutions suffer from various problems:
– Inconsistent
– Inefficient
– Custom Clients
• A standard along with simple implementations moves the industry forward
Confidential – Oracle Internal/Restricted/Highly Restricted 13

14. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Machine Learning Model Serving
What is GraphPipe?
Advantages
Protocol Deep Dive
Real World Demo
More Info
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 14
5
6

15. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Ease of Development
• Model Servers are written in Go – a very accessible language
• Flatbuffer code generation makes it easy to produce new clients
• Open spec makes it possible to integrate with existing servers
Confidential – Oracle Internal/Restricted/Highly Restricted 15

16. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Protocol Performance
Confidential – Oracle Internal/Restricted/Highly Restricted 16

17. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Serving Performance
Confidential – Oracle Internal/Restricted/Highly Restricted 17

18. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Machine Learning Model Serving
What is GraphPipe?
Advantages
Protocol Deep Dive
Real World Demo
More Info
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 18
5
6

19. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Flatbuffers
• Extensible protocol
• Small code footprint
• Near-zero deserialization overhead
Confidential – Oracle Internal/Restricted/Highly Restricted 19

20. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Protocol Summary
Confidential – Oracle Internal/Restricted/Highly Restricted 20

21. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Machine Learning Model Serving
What is GraphPipe?
Advantages
Protocol Deep Dive
Real World Demo
More Info
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 21
5
6

22. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
AlphaZero
• Algorithm for training a machine to play any game*
• Any game that can be represented with a Markov Process
• Trained without human information through self play
• Needs a structured representation of the game state
• Needs rules for transitioning from one state to the next
Confidential – Oracle Internal/Restricted/Highly Restricted 22

23. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
AlphaZero Timeline
AlphaGo
Beats Fan Hui
Oct-16
Paper
published in
Nature Jan-
16
AlphaGo
Beats Lee
Sedol Mar-16
AlphaGo
Beats Ke Jie
May-17
AlphaGoZero
published
Oct-17
AlphaZero
published
Dec-17
Confidential – Oracle Internal/Restricted/Highly Restricted 23

24. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
The Game Playing Black Box
Confidential – Oracle Internal/Restricted/Highly Restricted 24
Neural
NetworkPosition Move

25. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Training the Network
Confidential – Oracle Internal/Restricted/Highly Restricted 25
TrainingLabeled
Data
Neural
Network

26. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Generating Data
Confidential – Oracle Internal/Restricted/Highly Restricted 26
Self-Play
(MCTS)
Neural
Network
Labeled
Data

27. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
AlphaZero In a nutshell
Confidential – Oracle Internal/Restricted/Highly Restricted 27
Self-PlayTraining
Neural Network
Labeled Data

28. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
AlphaZero for Connnect Four
• We trained a network to play Connect Four using 150 cycles of this process
(and playing about 1,000,000 games during self-play)
• The network finds the correct move in about 99% of positions
• We used GraphPipe as part of the training process because we were
generating games across a cluster of 5 machines with GPUs
• But GraphPipe is even more useful for deploying this model so that people
can use it
• How do we deploy our model for use in an application? GraphPIpe!
Confidential – Oracle Internal/Restricted/Highly Restricted 28

29. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Serving the AlphaZero Trained Network
Confidential – Oracle Internal/Restricted/Highly Restricted 29
Neural
Network
GraphPipe
Position
Move
GraphPipe
Web
Frontend

30. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Live Demo!
Confidential – Oracle Internal/Restricted/Highly Restricted 30

31. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Actual Architecture
Confidential – Oracle Internal/Restricted/Highly Restricted 31

32. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Machine Learning Model Serving
What is GraphPipe?
Advantages
Protocol Deep Dive
Real World Demo
More Info
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 32
5
6

33. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
GraphPipe
• https://oracle.github.io/graphpipe/
• https://github.com/oracle/graphpipe
• https://github.com/oracle/graphpipe-go
• https://github.com/oracle/graphpipe-py
• https://github.com/oracle/graphpipe-tf-py
• https://hub.docker.com/r/sleepsonthefloor/
• https://hackernoon.com/machine-learning-model-pipelines-part-i-
e138b7a7c1ef
Confidential – Oracle Internal/Restricted/Highly Restricted 33

34. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
AlphaZero
• https://azfour.com/
• https://medium.com/oracledevs/lessons-from-implementing-alphazero-
7e36e9054191
• https://medium.com/@sleepsonthefloor/azfour-a-connect-four-webapp-
powered-by-the-alphazero-algorithm-d0c82d6f3ae9
• https://medium.com/applied-data-science/alphago-zero-explained-in-one-
diagram-365f5abf67e0
• https://deepmind.com/documents/119/agz_unformatted_nature.pdf
• https://arxiv.org/abs/1712.01815
Confidential – Oracle Internal/Restricted/Highly Restricted 34

35. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Questions and Answers
Confidential – Oracle Internal/Restricted/Highly Restricted 35

36. Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Confidential – Oracle Internal/Restricted/Highly Restricted 36