Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache deep learning 202 Washington DC - DWS 2019

135 views

Published on

#DWS19
Apache Deep Learning - Apache MXNet, Apache NiFi
A quick integration for Big Data Engineers on how to use Apache MXNet with Apache NiFi in streams, at the edge, in a processor and on Linux and OSX.

Published in: Data & Analytics
  • Be the first to comment

Apache deep learning 202 Washington DC - DWS 2019

  1. 1. @PaaSDev Apache Deep Learning 202 v1.00 (For Data Engineers) Timothy Spann https://github.com/tspannhw/ApacheDeepLearning202/
  2. 2. @PaaSDev Disclaimer • This is my personal integration and use of Apache software, no companies vision. • This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. This is Tim’s ideas only. • Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. • This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. • Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Since this document contains an outline of general product development plans, customers should not rely upon it when making a purchase decision.
  3. 3. @PaaSDev There are some who call him... DZone Zone Leader and Big Data MVB; Princeton Future of Data Meetup https://github.com/tspannhw https://community.hortonworks.com/users/9304/tspann.html https://dzone.com/users/297029/bunkertor.html https://www.meetup.com/futureofdata-princeton/
  4. 4. @PaaSDev Deep Learning for Big Data Engineers Multiple users, frameworks, languages, devices, data sources & clusters BIG DATA ENGINEER • Experience in ETL • Coding skills in Scala, Python, Java • Experience with Apache Hadoop • Knowledge of database query languages such as SQL • Knowledge of Hadoop tools such as Hive, or Pig • Expert in ETL (Eating, Ties and Laziness) • Social Media Maven • Deep SME in Buzzwords • No Coding Skills • Interest in Pig and Falcon CAT AI • Will Drive your Car • Will Fix Your Code • Will Beat You At Q-Bert • Will Not Be Discussed Today • Will Not Finish This Talk For Me, This Time http://gluon.mxnet.io/chapter01_crashcourse/preface.html
  5. 5. @PaaSDev • Cloud ready • Python, C++, Scala, R, Julia, Matlab, MXNet.js and Perl Support • Experienced team (XGBoost) • AWS, Microsoft, NVIDIA, Baidu, Intel • Apache Incubator Project • Run distributed on YARN and Spark • In my early tests, faster than TensorFlow. (Try this yourself) • Runs on Raspberry PI, NVidia Jetson Nano and other constrained devices https://mxnet.incubator.apache.org/how_to/cloud.html https://github.com/apache/incubator-mxnet/tree/1.3.1/example https://gluon-cv.mxnet.io/api/model_zoo.html https://elinux.org/Jetson_Nano
  6. 6. @PaaSDev • Great documentation • Crash Course • Gluon (Open API), GluonCV, GluonNLP • Keras (One API Many Runtime Options) • Great Python Interaction. Java and Scala APIs! • Open Source Model Server Available • ONNX (Open Neural Network Exchange Format) Support for AI Models • Now in Version 1.4.0! • Rich Model Zoo! • Math Kernel Library and NVidia CUDA Optimizations • TensorBoard compatible http://mxnet.incubator.apache.org/ http://gluon.mxnet.io/https://onnx.ai/ pip3.6 install -U keras-mxnet https://gluon-nlp.mxnet.io/ pip3.6 install --upgrade mxnet pip3.6 install gluonnlp pip3.6 install gluoncv pip3.6 install mxnet-mkl>=1.3.0 --upgrade
  7. 7. @PaaSDev Apache MXNet GluonCV Zoo https://gluon-cv.mxnet.io/model_zoo/classification.html • ResNet152_v2 • MobileNetV2_0.25 • VGG19_bn • SqueezeNet1.1 • DenseNet201 • Darknet53 • InceptionV3 • CIFAR_ResNeXt29_16x64 • yolo3_darknet53_voc • ssd_512_mobilenet1.0_coco • faster_rcnn_resnet101_v1d_coco • yolo3_darknet53_coco • FCN model on PASCAL VOC
  8. 8. @PaaSDev Object Detection: GluonCV YOLO v3 and Apache NiFi https://community.hortonworks.com/articles/222367/using-apache-nifi-with-apache-mxnet-gluoncv-for-yo.html
  9. 9. @PaaSDev Object Detection: Faster RCNN with GluonCV net = gcv.model_zoo.get_model(faster_rcnn_resnet50_v1b_voc, pretrained=True) Faster RCNN model trained on Pascal VOC dataset with ResNet-50 backbone https://gluon-cv.mxnet.io/api/model_zoo.html
  10. 10. @PaaSDev Instance Segmentation: Mask RCNN with GluonCV net = model_zoo.get_model('mask_rcnn_resnet50_v1b_coco', pretrained=True) Mask RCNN model trained on COCO dataset with ResNet-50 backbone https://gluon-cv.mxnet.io/build/examples_instance/demo_mask_rcnn.html https://arxiv.org/abs/1703.06870 https://github.com/matterport/Mask_RCNN
  11. 11. @PaaSDev Semantic Segmentation: DeepLabV3 with GluonCV model = gluoncv.model_zoo.get_model('deeplab_resnet101_ade', pretrained=True) GluonCV DeepLabV3 model on ADE20K dataset https://gluon-cv.mxnet.io/build/examples_segmentation/demo_deeplab.html run1.sh demo_deeplab_webcam.py http://groups.csail.mit.edu/vision/datasets/ADE20K/ https://arxiv.org/abs/1706.05587 https://www.cityscapes-dataset.com/ This one is a bit slower.
  12. 12. @PaaSDev Semantic Segmentation: Fully Convolutional Networks model = gluoncv.model_zoo.get_model(‘fcn_resnet101_voc ', pretrained=True) GluonCV FCN model on PASCAL VOC dataset https://gluon-cv.mxnet.io/build/examples_segmentation/demo_fcn.html run1.sh demo_fcn_webcam.py https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf
  13. 13. @PaaSDev Simple Pose Estimation https://gluon-cv.mxnet.io/build/examples_pose/cam_demo.html pip3.6 install gluoncv --pre --upgrade https://github.com/dmlc/gluon-cv/tree/master/scripts/pose/simple_pose yolo3_mobilenet1.0_coco + simple_pose_resnet18_v1b
  14. 14. @PaaSDev Apache MXNet Native Processor for Apache NiFi This is a beta, community release by me using the new beta Java API for Apache MXNet. https://github.com/tspannhw/nifi-mxnetinference-processor https://community.hortonworks.com/articles/229215/apache-nifi-processor-for-apache-mxnet-ssd-single.html https://www.youtube.com/watch?v=Q4dSGPvqXSA
  15. 15. @PaaSDev Using Apache MXNet on The Edge with Sensors and Google Coral (MiNiFi) https://www.datainmotion.dev/2019/03/using-raspberry-pi-3b-with-apache-nifi.html

×