Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OSG(a)i: because AI needs a runtime - Tim Verbelen (imec)

8 views

Published on

OSGi Community Event 2018 Presentation by Tim Verbelen (imec)

Abstract: Nowadays AI is reaching new heights on the hype cycle, especially due to recent advances in deep learning techniques. A lot of deep learning frameworks exist for creating and training deep neural networks, the most popular ones being PyTorch and TensorFlow. However, how to integrate, deploy and manage these neural networks in complex software systems is often overlooked. In this talk we show how OSGi can be used as a modular runtime for deep learning models. We embed those models inside OSGi bundles, and use the extender pattern to make these available as OSGi services. You can then use your favorite OSGi specs such as DS and PushStreams to integrate these into your application.

Published in: Technology
  • Be the first to comment

OSG(a)i: because AI needs a runtime - Tim Verbelen (imec)

  1. 1. PUBLIC1 OSG(A)I: BECAUSE A.I. NEEDS A RUNTIME TIM VERBELEN Senior Researcher imec – Ghent University
  2. 2. PUBLIC WHAT IS A.I.? 2 THE NEXT BIG THING (AFTER BLOCKCHAIN OFC!)
  3. 3. PUBLIC WHAT IS MACHINE LEARNING? 3 LEARNING A FUNCTION APPROXIMATION MAPPING INPUTS TO OUTPUTS ModelModel outputsinputs
  4. 4. PUBLIC WHAT IS DEEP LEARNING? 4 USING DEEP NEURAL NETWORKS AS MACHINE LEARNING MODEL Neural NetworkNeural Network “cat”
  5. 5. PUBLIC DEEP LEARNING FRAMEWORKS AND TOOLS 5
  6. 6. PUBLIC THE AVERAGE DATA SCIENTIST WORKFLOW 1. Inspect and clean up the data 2. Select and encode features / outputs 3. Script together a model training procedure 4. Find some good hyperparameters 5. Dump the trained model 6
  7. 7. PUBLIC BUT WHAT ABOUT PRODUCTION? 7
  8. 8. PUBLIC TENSORFLOW SERVING 8 Client Servable Handler Dynamic Manager VersionPolicy models/ 1/ … 2/ … Server gRPC/REST request
  9. 9. PUBLIC CHECK THE BOXES I want to query my models with an RPC/REST call I train my models using TensorFlow I deploy my models on a containerized infrastructure I don’t need additional metadata/versioning besides a single incrementing integer 9
  10. 10. PUBLIC OSGI CAN HELP!  A unit of deployment  Package ML model as an OSGi bundle  A resolveable artifact with requirements and capabilities  Requirements: what do you need to run this ML model  Capabilities: what kind of inputs can it process and what kind of outputs does it give you?  A lightweight service model  Access your model via an OSGi service  Service selection at runtime  Use service properties and target filters to select the best model at runtime 10
  11. 11. PUBLIC A WAY TO SHARE ML MODELS A common format for describing computation graphs  defined as protocol buffers  specifies data types and operators  framework agnostic 11 OPEN NEURAL NETWORK EXCHANGE (ONNX)
  12. 12. PUBLIC A WAY TO EXECUTE MODELS IN OSGI  A modular deep learning framework developed in OSGi  Neural networks, operators, datasets, learners, … as an OSGi service  Low-level operations via JNI and blas/cublas/cudnn backends  Distributed deployments using OSGi remote services  Web UI to build, deploy, train models 12 DIANNE FRAMEWORK http://dianne.intec.ugent.be
  13. 13. PUBLIC DEMO TIME 13
  14. 14. PUBLIC NOW LET’S ADD A ROBOT 14
  15. 15. PUBLIC NOW LET’S ADD A ROBOT 15
  16. 16. PUBLIC PUTTING THINGS TOGETHER 16 @Reference Camera camera; @Reference NeuralNetwork nn; @Reference ArmController controller; public Promise<Boolean> grasp(){ return camera.stream() // returns PushStream<Frame> // timeout after 10 seconds .timeout(Duration.ofSeconds(10)) // process camera frame with neural network .map(frame -> nn.forward(toTensor(frame)) // use result to update controller .map(r -> controller.update(r)) // short circuit in case of success .anyMatch(s -> s); }
  17. 17. PUBLIC NOW LET’S ADD A ROBOT 17
  18. 18. PUBLIC18
  19. 19. PUBLIC19 THANK YOU! http://dianne.intec.ugent.be

×