Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ML in the Browser: Interactive Experiences with Tensorflow.js

20 views

Published on

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/39SddUL.

Victor Dibia provides a friendly introduction to machine learning, covers concrete steps on how front-end developers can create their own ML models and deploy them as part of web applications. He discusses his experience building Handtrack.js - a library for prototyping real time hand tracking interactions in the browser. Filmed at qconsf.com.

Victor Dibia is a Research Engineer with Cloudera’s Fast Forward Labs. Prior to this, he was a Research Staff Member at the IBM TJ Watson Research Center, New York. His research interests are at the intersection of human computer interaction, computational social science, and applied AI.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

ML in the Browser: Interactive Experiences with Tensorflow.js

  1. 1. 1 ML in the Browser Victor Dibia, PhD Cloudera Fast Forward Labs @vykthur Interactive Experiences with Tensorflow.js
  2. 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ ml-browser-tensorflow-js/
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  4. 4. 2 - Background in CS, Human Computer Interaction (HCI), Applied AI. - Previously Research Scientist at IBM Research New York. - Community Contributions - GDE for Machine Learning - Research Engineer (ML) at Cloudera Fast Forward Labs Research Engineer Cloudera Fast Forward Labs, Brooklyn. @vykthur | @victordibia
  5. 5. 4
  6. 6. 5
  7. 7. 6 experiments.fastforwardlabs.com
  8. 8. 7 Agenda • Why Machine Learning in the Browser? • Overview of the Tensorflow.js library API with examples • The Ops API • The Layers API • Building Handtrack.js - a javascript library for prototyping gesture based interactions in the browser. •
  9. 9. 8 Why Machine Learning in the Browser?
  10. 10. Artificial Intelligence Machine Learning Deep Learning The broad field that describes efforts to make machines intelligent. Algorithms that enable machines independently learn from data and make predictions. Decision trees, random forests, reinforcement learning, neural networks, Neural Networks (NN): a set of stacked computation units. - High accuracy, scales with data - Advances in NN algorithms - Amenable to GPU computation But What is Machine Learning? Let's review some terminology ..
  11. 11. Artificial Intelligence Machine Learning Deep Learning The broad field that describes efforts to make machines intelligent. Algorithms that enable machines independently learn from data and make predictions. Decision trees, random forests, reinforcement learning, neural networks, Neural Networks (NN): a set of stacked computation units. - High accuracy, scales with data - Advances in NN algorithms - Amenable to GPU computation
  12. 12. Artificial Intelligence Machine Learning Deep Learning The broad field that describes efforts to make machines intelligent. Algorithms that enable machines independently learn from data and make predictions. Decision trees, random forests, reinforcement learning, neural networks, Neural Networks (NN): a set of stacked computation units. - High accuracy, scales with data - Advances in NN algorithms - Amenable to GPU computation For today, each time I refer to Machine Learning or AI, what I am referring to is Deep Learning
  13. 13. Fast Forward Labs Perceptron
  14. 14. Dense layer (nodes in one layer connected to all nodes in the following layer). Fast Forward Labs Multilayer Perceptron
  15. 15. 14 The Languages of Machine Learning
  16. 16. 15 The Languages of Machine Learning
  17. 17. 16 The Languages of Machine Learning
  18. 18. 17 The Languages of Machine Learning Backend programming languages .. and they work well! .. but ..
  19. 19. 18 • Cloudera Fast Forward Labs • Intro to Transfer Learning for NLP • Applications • Appendix: Positioning & FAQ (internal audience only) 2019 Stackoverflow survey of 90k developers What programming language do developers use?
  20. 20. 19 2019 Stackoverflow survey of 90k developers What programming language do developers want to learn?
  21. 21. 20 Github 2018 Top Programming languages between 2008 and 2018.
  22. 22. 21 dev.to Anthony Delgado
  23. 23. 22 Backend Node.js + Front End Vanilla js, Frameworks
  24. 24. 23 • Wait .. thats a bad idea Tenor Bad Idea
  25. 25. 24 Can I really train a neural network in Javascript? Orienta Boy thinking
  26. 26. 25 Yes!! Tensorflow.js - a library for building/training ML models in Javascript.
  27. 27. 26 Will it meet my speed requirements?
  28. 28. 27 Yes!! Tensorflow.js, is accelerated in the browser using WebGL (CPU + GPU) and in Node.js using Tensorflow C API (CPU + GPU).
  29. 29. 28 How much effort is this? Can I re-use my existing models/pipelines?
  30. 30. 29 Not much! - Tensorflowjs API is Similar to Keras - Tensorflow.js converter allows you import pretrained Keras (hd5, savedModel), Tensorflow SavedModel, Tensorflow Hub Module
  31. 31. 30 Oprah “You get a car”
  32. 32. 31 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity
  33. 33. 32 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity • Strong Notion of Privacy We do not see your data, it never gets to our servers. • Example use case: sensitive documents identification, domain specific text autocomplete for chat systems etc
  34. 34. 33 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity • No installs, drivers and dependency issues. • Navigate to a URL and your application is available to end users. • Free model asset hosting/versioning/distribution (NPM/jsdelvr), excellent for end user applications.
  35. 35. 34 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity • ML models can be optimized for speed and can run fast on mobile and in the browser • In some cases, it is faster to run your model in the browser, rather than send make round trip requests to a server. • Avoid reliability issues with intermittent connectivity environments
  36. 36. 35 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity • The Browser is designed for interactive experiences. ML can supercharge this! • Build models on the fly using rich user data available in the browser (camera, file uploads), run inference, retrain existing models, enable dynamic behaviour. • ML education, retail, advertising, arts, entertainment, gaming.
  37. 37. 36 Privacy Preserving Sensitive Content Detection.
  38. 38. 37 Example Use Cases Client-side indecent content checking nsfw.js.com
  39. 39. 38 Handtrack.js
  40. 40. 39
  41. 41. 40 Tensorflow.js A library for machine learning in JavaScript
  42. 42. 41 What can you do with Tensorflow.js • Author models • Import pre-trained models for inference • Re-train imported models Online Flow • Design/Train/Test/ Inference, all in your javascript code.
  43. 43. 42 What can you do with Tensorflow.js • Author models • Import pre-trained models for inference • Re-train imported models Offline Flow • Design/Train/Test model on your preferred hardware (GPU, TPU clusters) • Import into javascript for inference
  44. 44. 43 What can you do with Tensorflow.js • Author models • Import pre-trained models for inference • Re-train imported models Hybrid Flow • Design/Train/Test model on your preferred hardware (GPU, TPU clusters) • Import into javascript application • Retrain/Run inference
  45. 45. 44 Tensorflow.js - Backend and Frontend Browser (Front end) Server (Back end, Node.js)
  46. 46. 45 Tensorflow.js - Backend and Frontend Browser (Front end) • Usage • Vanilla js script tag import (like jquery) • Import - NPM install ( build tools, frameworks such as React, Vue, etc). • Acceleration • WebGL • Tensors implemented as Shader programs (will use GPUs where available)
  47. 47. 46 Tensorflow.js - Backend and Frontend Server (Back end, Node.js) • Usage • NPM install • (tfjs-node, tfjs-node-gpu) • Acceleration • Tensorflow C binary
  48. 48. WebGL Browser TF CPUTF GPU Node.js Runtime API Ops API (Eager) Layers API TF TPU Keras Model TensorFlow SavedModel
  49. 49. 48 Ops API (Eager) Layers API Fit data to a polynomial function Build and train a two layer autoencoder in the browser Import a trained model for inference in the browser (Handtrack.js) API
  50. 50. 49 Tensorflow.js OPS API Use with caution
  51. 51. 50 The Ops API - Fit a Polynomial f(x) = ax2 + bx + c
  52. 52. 51 import * as tf from '@tensorflow/tfjs';
  53. 53. 52 import * as tf from '@tensorflow/tfjs'; const a = tf.tensor(0.1).variable(); const b = tf.tensor(0.1).variable(); const c = tf.tensor(0.1).variable();
  54. 54. 53 import * as tf from '@tensorflow/tfjs'; const a = tf.tensor(0.1).variable(); const b = tf.tensor(0.1).variable(); const c = tf.tensor(0.1).variable(); // f(x) = a*x^2 + b^x + c const f = x => a.mul(x.square()).add(b.mul(x)).add(c); // ...
  55. 55. 54 // ... // Mean-squared error const loss = (preds, label) => preds.sub(label).square().mean();
  56. 56. 55 // ... // Mean-squared error const loss = (preds, label) => preds.sub(label).square().mean(); const optimizer = tf.train.sgd(learningRate);
  57. 57. 56 // ... // Mean-squared error const loss = (preds, label) => preds.sub(label).square().mean(); const optimizer = tf.train.sgd(learningRate); for (let i = 0; i < EPOCHS; i++) { optimizer.minimize(() => loss(f(data.xs), data.ys)); }
  58. 58. 57 Tensorflow.js Layers API Very much like keras
  59. 59. 58 The Layers API • Recommended API for neural networks • Similar in spirit to the Keras Library. Keras is a well designed api standard for composing and training neural networks.
  60. 60. 59 An Autoencoder Dimensionality reduction • Neural Network with two parts. • Given some high dimension data (e.g. tabular data with many fields), find a smaller representation. • This representation should support reconstruction.15 7 2 15 7
  61. 61. 60 An Autoencoder Encoder • Take a high dimensional input. • Learn to compress it to a meaningful smaller representation (bottleneck) 15 7 2 15 7
  62. 62. 61 An Autoencoder Decoder • Takes a bottleneck vector • Learn to reconstruct original input from z. 15 7 2 15 7
  63. 63. 62 An Autoencoder for Detecting Anomalies Anomaly Detection • Train on Normal Data • Learns a bottleneck vector suitable for reconstructing normal signals • High reconstruction error (MSE) for abnormal images. Flag high error as anomalies. 15 7 2 15 7
  64. 64. 63 An Autoencoder for Detecting Anomalies Example, ECG data 15 7 2 15 7 0.8 0.2 Error
  65. 65. 64 # Keras, Python # Encoder inputs = Input(shape=(num_feature s,)) # define hidden layers // Tensorflow JS // Encoder const input = tf.input({ shape: [numFeatures] }) # define hidden layers
  66. 66. 65 # Keras, Python # define hidden layers enc_hidden = Dense(15, activation='relu')(inpu ts) // Tensorflow JS # define hidden layers let encHidden = tf.layers.dense({ units: 15, activation: "relu" }).apply(input);
  67. 67. 66 # Keras, Python # define hidden layers enc_hidden = Dense(7, activation='relu') (enc_hidden) // Tensorflow JS # define hidden layers let encHidden = encHidden = tf.layers.dense({ units: 7, activation: "relu" }).apply(encHidden);
  68. 68. 67 # Keras, Python # Encoder z_ = Dense(latent_dim)(enc_hidden) encoder = Model(inputs, z_, name='encoder') // Tensorflow JS // Encoder const z_ = tf.layers.dense({ units: latentDim }).apply(encHidden); const encoder = tf.model({ inputs: input, outputs: z_, name:"encoder" })
  69. 69. 69 Train - Data
  70. 70. 70 Test - Data
  71. 71. 71 Test - Data
  72. 72. 72 Interactive Visualizations of Predictions
  73. 73. 73 Interactive Visualizations of Bottleneck Predictions
  74. 74. 74 Interactive Visualizations of Bottleneck Predictions 2600 data points
  75. 75. 75 Interactive Visualizations of Bottleneck Predictions
  76. 76. 76 Building Handtrack.js, https://github.com/victordibia/handtrack.js/ A library for prototyping hand gestures in the browser
  77. 77. 77 Handtrack.js • Given an image, predict the location (bounding box) of all hands. • Uses a Convolutional Neural Network (Object Detection) • Bundled into a JS library to enable developers prototype interactive. • Open source.
  78. 78. 78 // Load the model. handTrack.load().then(model => { // detect objects in the image. model.detect(img).then(predictions => { console.log('Predictions: ', predictions); }); }); <script src="https://cdn.jsdelivr.net/npm/handtrackjs/dist/handtrack.min.js"> </script>
  79. 79. 79 [{ bbox: [x, y, width, height], class: "hand", score: 0.8380282521247864 }, { bbox: [x, y, width, height], class: "hand", score: 0.74644153267145157 }]
  80. 80. 80 How was this built?
  81. 81. 81 How was this built?
  82. 82. 82 Data Assembly (Egohands Dataset)
  83. 83. 83 Model Training • Convert images to Tensorflow Records format – For each image, list of bbox coordinates, and labels. • Train using Tensorflow Object Detection API (python) – Framework and sample code for training various object detection models. – Mobilenet, SSD (fast, optimized for mobile). • Export Trained Model as savedModel from checkpoints. •
  84. 84. 84 Model Conversion • Use tensorflow.js converter • Identify model output node variable name • Specify model format • Specify input/output directories • Supports • savedModel (default for Tensorflow) • Keras Model • Tensorflow Hub $ pip install tensorflowjs $ tensorflowjs_converter --input_format tf_saved_model --output_node_names 'Predictions/Reshape_1' path/to/tf_saved_model path/to/tfjs_model Exported model can be imported in javascript.
  85. 85. 85 Exported Model • 18.5mb .. final model size • Sharded into 5 files with a maximum size of 4.2mb • • Bundled into an NPM library, hosted on NPM (with benefits!) • Model artifact versioning • CDN serving (jsdelivr) •
  86. 86. 86
  87. 87. 87
  88. 88. 88
  89. 89. 89 Demo! • So .. does it work live? • https://victordibia.github.io/handtrack.js/#/
  90. 90. 90 Some Challenges • Manual Memory Management • Tensorflow.js implements tensors using WebGLTextures. • Copied to GPU and cached to enable fast computation • Need to be explicitly deleted using the tensor.dispose(). • Use tf.memory() to see how many tensors your application is holding onto
  91. 91. 91 Some Challenges • General limitations of the Browser. • The browser is single threaded • Compute heavy operations will block the UI thread.
  92. 92. 92 Some Challenges • Device Fragmentation • Differences in devices can lead to wildly varying performance • CPU (Mobile, Desktop, Embedded) • GPU availability • Device power mode
  93. 93. 93 Some Good Practices • Optimize Your Model • Replace full Convolutional Operations with Depthwise separable convolutions (less parameters). • Remove optional post processing from model graph during model export. • Explore model compression methods (quantization, distillation, pruning). • Explore NAS/AutoML. .
  94. 94. 94 Some Good Practices • Design and Test Considerations • Asynchronous methods (async await, promises) to avoid blocking the UI thread. • Visually communicate model load/inference latency. • Communicate model uncertainty where possible. • Test and benchmark across multiple devices!! • Chrome Profiler is your friend! • See Google PAIR (People + AI Guidebook) document on how to design human-centered AI products.
  95. 95. 95 Conclusion • ML in the browser is valuable when your objectives are focused on privacy, ease of distribution, low latency and interactivity. • Tensorflow.js provides an API for ML in Javascript (online, offline, hybrid workflow). Fast, expressive, integrated with the rest of the TF ecosystem. • Research advances in ML (compression, distillation, pruning) + Advances in acceleration in the browser (WebAssembly, WebGPU), positions ML in the browser for faster and wider integration.
  96. 96. 96 Thank you! Research Engineer Cloudera Fast Forward Labs, Brooklyn. @vykthur | @victordibia Source Code: Handtrack.js https://github.com/victordibia/handtrack.js
  97. 97. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ ml-browser-tensorflow-js/

×