1. Deep Dive into deeplearn.js
Kai Sasaki
Treasure Data Inc

2. About me
Kai Sasaki - 佐々木 海 (@Lewuathe)
Software Engineer at Treasure Data
Apache Hivemall Committer
Hadoop, Spark, deeplearnjs Contributor

3. What is deeplearn.js?
deeplearn.js is a deep learning library
running on your browser accelerated by
WebGL.
Mainly developed by Google PAIR project.
https://research.googleblog.com/2017/08/harness-power-of-machine-learning-in.html

4. Demo
https://deeplearnjs.org/
- SqueezeNet
- Teachable Machine

5. Similar Products
WebDNN
- https://mil-tokyo.github.io/webdnn/
Keras.js
- https://github.com/transcranial/keras-js

6. Technology
- Written in TypeScript
- Available in TypeScript and ES6 project
- Accelerated by WebGL
- Tested on Chrome and Firefox
- Can run both training and inference
https://www.npmjs.com/ types

7. Architecture
Graph: Representing computation data flow
much like graph in TensorFlow etc
Op: A graph node that represents a
computation on tensor
NDArray: Real data copied from/to CPU
and GPU backed by Typed Array

8. Computation Graph

9. Graph
Node
Node
Node Tensor
Tensor
Tensor

10. Graph
Node
Node
Node
Node
Node Tensor
Tensor
Tensor

11. Graph
Node
Node
Node
Node
Node
Node
Tensor
Tensor
Tensor
Tensor
Tensor
add
multiply
conv2d
relu
etc

12. Operation
Op
Op
Op
Op
Op
Op
NDArray
NDArray
NDArray
NDArray
NDArray

13. Kernel Implementation

14. math
Web Browser
WebGL

15. math
NDArrayMathCPU
NDArrayMathGPU
Web Browser
WebGL

16. math
math
NDArrayMathCPU
NDArrayMathGPU
Web Browser
WebGL
kernelNDArray

17. math
math
NDArrayMathCPU
NDArrayMathGPU
Web Browser
WebGL
kernelNDArray
Conv2d softmax etc…
Graph
matmul

18. Kernel Example
import {Array1D, NDArrayMathGPU, Scalar} from 'deeplearn';
const math = new NDArrayMathGPU();
const a = Array1D.new([1, 2, 3]);
const b = Scalar.new(2);
const result = math.add(a, b);
// Option 1: With async/await.
// Caveat: in non-Chrome browsers you need to put this in an async
function.
console.log(await result.data()); // Float32Array([3, 4, 5])
// Option 2: With a Promise.
result.data().then(data => console.log(data));
// Option 3: Synchronous download of data.
// This is simpler, but blocks the UI until the GPU is done.
console.log(result.dataSync());

19. TensorFlow like Example
const graph = new Graph();
// Make a new input in the graph, called 'x', with shape [] (a
Scalar).
const x: Tensor = graph.placeholder('x', []);
// Make new variables in the graph, 'a', 'b', 'c' with shape []
and random
// initial values.
const a: Tensor = graph.variable('a', Scalar.new(Math.random()));
const b: Tensor = graph.variable('b', Scalar.new(Math.random()));
const c: Tensor = graph.variable('c', Scalar.new(Math.random()));
//…
// At this point the graph is set up, but has not yet been
evaluated.
// **deeplearn.js** needs a Session object to evaluate a graph.
const math = new NDArrayMathGPU();
const session = new Session(graph, math);

20. TensorFlow Integration

21. Port TensorFlow model
deeplearn.js can import TensorFlow model
from checkpoint.
Need to port the weights from checkpoint
files.
https://deeplearnjs.org/demos/mnist/mnist.html

22. Porting weights
$ python scripts/dump_checkpoints/dump_checkpoint_vars.py
—model_type=tensorflow
—output_dir=demos/mnist/
—checkpoint_file=/tmp/tensorflow/mnist/logs/
fully_connected_feed/model.ckpt-1999
# Will save a lot of files
# (one file per variable and manifest.json)

23. Porting weights
import {CheckpointLoader} from 'deeplearn';
const varLoader = new CheckpointLoader('.');
varLoader.getAllVariables().then(async vars => {
const math = new NDArrayMathGPU();
const hidden1W = vars[‘hidden1/weights’] as Array2D;
const hidden1b = vars[‘hidden1/biases’] as Array1D;
//…
}

24. GPU Acceleration

25. WebGL
WebGL (Web Graphics Library) is a
JavaScript API for rendering interactive 3D
and 2D graphics within any compatible web
browser without the use of plug-ins.
- Mozilla Developer Network

26. WebGL
- Provides the standard API
- Implemented by major web browsers such
as Chrome, Firefox, Safari and Opera
- Hardware Agnostic

27. WebGL Graphic Pipeline
https://www.tutorialspoint.com/webgl/webgl_graphics_pipeline.htm

28. WebGL Graphic Pipeline
http://math.hws.edu/graphicsbook/c6/s4.html

29. WebGL
1. deeplearnjs compiles GLSL shader
2. Send input NDArray as texture into GPU
3. Run a fragment shader
4. Fetch result NDArray from GPU

30. 1. Compile Shader
float unaryOperation(float x) {
return (x < 0.0) ? 0.0 : x;
}
void main() {
float x = getAAtCoords(); // Sample a value from a texture
float y = unaryOperation(x);
setOutput(y);
}

31. 2. Upload NDArray to GPU
gl.bindTexture(gl.TEXTURE_2D, texture);
gl.texSubImage2D(
gl.TEXTURE_2D, 0, 0, 0, width, height, textureFormat,
getTextureType(gl), data);
// Texture format is RGBA
// Texture type is gl.UNSIGNED_BYTE or gl.FLOAT
gl.bindTexture(gl.TEXTURE_2D, null);

32. 3. Run shader program
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
// Map buffer data to texture coordination
gl.vertexAttribPointer(
loc, arrayEntriesPerItem, gl.FLOAT, false,
itemStrideInBytes, itemOffsetInBytes);
// Draw something
gl.drawElements(gl.TRIANGLES, 6, gl.UNSIGNED_SHORT, 0);

33. 4. Download the result
getBufferSubDataAsyncExtension.getBufferSubDataAsync(
gl2.PIXEL_PACK_BUFFER, 0, downloadTarget);
// Download as Typed Array -> NDArray
// Typed Array is backed by raw buffer memory (ArrayBuffer)
// accessed through views (e.g. Float32Array)
https://www.khronos.org/registry/webgl/extensions/WEBGL_get_buffer_sub_data_async/

34. Future Works

35. Automatic TensorFlow to
deeplearn.js
Direct importing from TensorFlow
GraphDef format makes model importing
easier.
https://deeplearnjs.org/docs/roadmap.html

36. Eager Mode
Eager execution mode enables us to run
computation without Graph definition.
Similar to TensorFlow eager mode.

37. Decoupling NDArray from
storage mechanism
NDArray is now tightly coupled with storage
layer (GPU texture)
Tracking NDArray to avoid memory leak
often becomes hard work.

38. Great Demos
One of the reason we use deeplearn.js is
that it enables us to create interesting
application by using deep learning easily.
deeplearn.js is now enhancing demos to
show the potential of deep learning on Web
Browser.

39. Patches Welcome!