Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MiniFi and Apache NiFi : IoT in Berlin Germany 2018

984 views

Published on

Future of Data : Berlin
Apache NiFi and MiniFi with Apache MXNet and Tensorfor for IoT from edge devices like Raspberry Pis. Including Python and other tools.

Published in: Technology
  • Girls for sex in your area are there: tinyurl.com/areahotsex
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: www.bit.ly/sexinarea
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: www.bit.ly/2AJerkH
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area for one night is there tinyurl.com/hotsexinarea Copy and paste link in your browser to visit a site)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Girls for sex are waiting for you https://bit.ly/2TQ8UAY
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

MiniFi and Apache NiFi : IoT in Berlin Germany 2018

  1. 1. 1 © Hortonworks Inc. 2011–2018. All rights reserved. © Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information. IoT With MiniFi and Apache NiFi Berlin 2018 Timothy Spann, Solutions Engineer Hortonworks @PaaSDev
  2. 2. 2 © Hortonworks Inc. 2011–2018. All rights reserved. Disclaimer • This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. • Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. • This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. • Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Since this document contains an outline of general product development plans, customers should not rely upon it when making a purchase decision.
  3. 3. 3 © Hortonworks Inc. 2011–2018. All rights reserved. Agenda • Apache MiniFi • Apache NiFi • Devices • Demos • Questions
  4. 4. 4 © Hortonworks Inc. 2011–2018. All rights reserved. What do we want to do? • MiniFi ingests camera images and sensor data • MiniFi executes Apache MXNet at the edge • Run Apache MXNet Inception to recognize objects in image • Apache NiFi stores images, metadata and enriched data in Hadoop • Apache NiFi ingests social data and REST feeds • Apache OpenNLP and Apache Tika for textual data
  5. 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved. Aggregate all data from sensors, drones, logs, geo-location devices, images, machines and social feeds. Collect: Bring Together Mediate point-to-point and bi-directional data flows, delivering data reliably to Apache HBase, Apache Hive, HDFS, Slack and Email. Conduct: Mediate the Data Flow Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather, location, Apache OpenNLP and Apache MXNet. Curate: Gain Insights
  6. 6. 6 © Hortonworks Inc. 2011–2018. All rights reserved. Actionable Insights Architecture Ingestion Simple Event Processing Engine Stream Processing Destination Data Bus Build Predictive Model From Historical Data Deploy Predictive Model For Real-time Insights Perishable Insights Historical Insights
  7. 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved. IIoT Multiple devices, protocols, frameworks, languages, data types, sensors and networks Protocols • MQTT • HTTPS / SSL (REST/JSON) • OPC UA • CoAP • AMQP • JSON • XML • CSV • Raw Text • Images (JPEG, PNG) • Raw Data Streams Data Types Sensors • Cameras • Temperature/Humidity • IR • Proximity • Motion Sensors • GPS Protocols • NVidia Jetson TX1 • Raspberry Pi • Arduino • TS-7800 V2 • ESP8266 • DragonBoard 410c • BeagleBone Black
  8. 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved. HDF Data-In-Motion Platform – with HDF 3.1 GA Release
  9. 9. 9 © Hortonworks Inc. 2011–2018. All rights reserved. Apache MiNiFi • NiFi lives in the data center. Give it an enterprise server or a cluster of them. • MiNiFi lives as close to where data is born and is a guest on that device or system “Let me get the key parts of NiFi close to where data begins and provide bidirectional data transfer"
  10. 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved. Edge Intelligence with Apache MiNiFi à Guaranteed delivery à Data buffering ‒ Backpressure ‒ Pressure release à Prioritized queuing à Flow specific QoS ‒ Latency vs. throughput ‒ Loss tolerance à Data provenance à Recovery / recording a rolling log of fine-grained history à Designed for extension Different from Apache NiFi à Design and Deploy à Warm re-deploys Key Features
  11. 11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Java Agent C++ Agent StreamLine Builder Ops Insight Central HDP Cluster HBase Druid YARN HDFS Hive Weather Location Sensors
  12. 12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved http://mxnet.incubator.apache.org/ • Cloud ready • Experienced team (XGBoost) • AWS, Microsoft, NVIDIA, Baidu, Intel backing • Apache Incubator Project • Run distributed on YARN • In my early tests, faster than TensorFlow. • Runs on Raspberry PI, Nvidia Jetson TX1 and other constrained devices • Great documentation • Gluon • Great Python Interaction • Model Server Available • ONNX Support • Now in Version 1.1! • Great Model Zoo https://mxnet.incubator.apache.org/how_to/cloud.html https://github.com/apache/incubator-mxnet/tree/1.1.0/example
  13. 13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved • Apache MXNet via Execute Process (Python) • Apache MXNet Running on Edge Nodes (MiniFi) S2S • Apache MXNet Model Server Integration (REST API) Not Covered Today • *Dockerized Apache MXNet on Hadoop YARN 3 with NVidia GPU • *Apache MXNet on Spark Apache NiFi Integration with Apache MXNet Options
  14. 14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved • https://github.com/apache/incubator-mxnet/tree/master/tools/coreml • https://github.com/Leliana/WhatsThis • https://github.com/apache/incubator-mxnet/tree/master/amalgamation/jni • https://hub.docker.com/r/mxnet/ • https://github.com/apache/incubator-mxnet/tree/master/scala- package/spark Other Options
  15. 15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Pre-Built Models • CaffeNet • SqueezeNet v1.1 • Inception v3 • Single Shot Detection (SSD) • VGG19 • ResidualNet 152 • LSTM http://mxnet.incubator.apache.org/model_zoo/index.html https://github.com/dmlc/mxnet-model-gallery
  16. 16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved python3 -W ignore analyze.py {"uuid": "mxnet_uuid_img_20180208204131", "top1pct": "30.0999999046", "top1": "n02871525 bookshop, bookstore, bookstall", "top2pct": "23.7000003457", "top2": "n04200800 shoe shop, shoe-shop, shoe store", "top3pct": "4.80000004172", "top3": "n03141823 crutch", "top4pct": "2.89999991655", "top4": "n04370456 sweatshirt", "top5pct": "2.80000008643", "top5": "n02834397 bib", "imagefilename": "images/tx1_image_img_20180208204131.jpg", "runtime": "2"} Apache MXNet via Python (OSX Local with WebCam) https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an- apache-nifi-15-instance-w.html
  17. 17. 17 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet via Python (OSX Local with WebCam) {"uuid": "mxnet_uuid_img_20180208204131", "top1pct": "30.0999999046", "top1": "n02871525 bookshop, bookstore, bookstall", "top2pct": "23.7000003457","top2": "n04200800 shoe shop, shoe-shop, shoe store", "top3pct": "4.80000004172", "top3":"n03141823 crutch", "top4pct": "2.89999991655", "top4": "n04370456 sweatshirt", "top5pct":"2.80000008643", "top5": "n02834397 bib", "imagefilename": "images/tx1_image_img_20180208204131.jpg", "runtime": "2"}
  18. 18. 18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Installation on OSX git clone https://github.com/apache/incubator-mxnet.git cd incubator-mxnet mkdir images curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' - -header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep-alive' 'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L tar -xvzf inception-bn.tar.gz cp Inception-BN-0126.params Inception-BN-0000.params brew install graphviz pip install --upgrade pip pip install --upgrade setuptools pip install graphviz pip install mxnet http://mxnet.incubator.apache.org/install/index.html
  19. 19. 19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Running on with Apache NiFi Node
  20. 20. 20 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Installation on with Apache NiFi Node git clone https://github.com/apache/incubator-mxnet.git sudo yum groupinstall 'Development Tools' -y sudo yum install cmake git pkgconfig -y sudo yum install libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel -y sudo yum install libdc1394-devel libv4l-devel gstreamer-plugins-base-devel -y sudo yum install gtk2-devel -y sudo yum install tbb-devel eigen3-devel -y pip install numpy You will need a full Python development environment, C++ and I recommend building OpenCV2. https://community.hortonworks.com/articles/174227/apache-deep-learning-101-using-apache-mxnet-on-an.html
  21. 21. 21 © Hortonworks Inc. 2011–2018. All rights reserved. Data Ingestion: Optimize Log Analytics with Content Based Routing
  22. 22. 22 © Hortonworks Inc. 2011–2018. All rights reserved. Apache MXNet Running on Edge Nodes (MiniFi) https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html https://github.com/tspannhw/mxnet_rpi https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1- running-apac.html
  23. 23. 23 © Hortonworks Inc. 2011–2018. All rights reserved. Edge Analytics with NVidia Jetson TX1 Running Apache MXNet (MiniFi) https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-running- apac.html
  24. 24. 24 © Hortonworks Inc. 2011–2018. All rights reserved. Using Apache MXNet on The Edge with Sensors and Intel Movidius (MiniFi) https://community.hortonworks.com/articles/176932/apache-deep-learning-101-using-apache-mxnet-on- the.html
  25. 25. 25 © Hortonworks Inc. 2011–2018. All rights reserved. IIoT Processing Data Acquisition Edge Processing Real Time Stream Analytics Rapid Application Development IoT ANALYTICS CLOUD Acquire Move Routing & Filtering Deliver Parse Analysis Aggregation Modeling
  26. 26. 26 © Hortonworks Inc. 2011–2018. All rights reserved. Streaming Analytics Manager Part of MiniFi C++ Agent Detect metadata and data Extract metadata and data Content Analysis Deep Learning Framework Complex Event Processing Joining DataSets for Streaming Analytics Open Source Components Enabling Record Processing Schema Management
  27. 27. 27 © Hortonworks Inc. 2011–2018. All rights reserved. Streaming Analytics Manager Run everywhere Orchestration Queueing Simple Event Processing Deep Learning Framework Entity Resolution Natural Language Processing Open Source Components Edge Ingest Data Buffering Device Integration
  28. 28. 28 © Hortonworks Inc. 2011–2018. All rights reserved. Flow Apache NiFi 2 Apache NiFi 3 Apache Kafka 4 Apache Spark Streaming, SAM, Apache Storm 5 Apache MiniFi, Apache Mxnet1 Edge Devices Gateways & Routers Ingestion Platform Data Bus Stream Processing Engines
  29. 29. 29 © Hortonworks Inc. 2011–2018. All rights reserved. Flow Apache Spark on YARN7 Apache Zeppelin8 Apache Hadoop 6 Storage Platform Data Science Platform Dashboards
  30. 30. 30 © Hortonworks Inc. 2011–2018. All rights reserved. Questions?
  31. 31. 31 © Hortonworks Inc. 2011–2018. All rights reserved. Contact https://github.com/tspannhw/ApacheDeepLearning101 https://community.hortonworks.com/users/9304/tspann.html https://dzone.com/users/297029/bunkertor.html https://www.meetup.com/futureofdata-princeton/ https://twitter.com/PaaSDev https://community.hortonworks.com/articles/155435/using-the-new-mxnet-model-server.html @PaasDev
  32. 32. 32 © Hortonworks Inc. 2011–2018. All rights reserved. Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
  33. 33. 33 © Hortonworks Inc. 2011–2018. All rights reserved. Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved 4,000+ Registered Users 10,000+ Answers 15,000+ Technical Assets One Website!
  34. 34. 34 © Hortonworks Inc. 2011–2018. All rights reserved. https://community.hortonworks.com/articles/161761/new-features-in-apache-nifi- 15-apache-nifi-registr.html https://community.hortonworks.com/articles/171787/hdf-31-executing-apache- spark-via-executesparkinte.html https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an- apache-nifi-15-instance-w.html https://community.hortonworks.com/articles/171893/hdf-31-executing-apache- spark-via-executesparkinte-1.html Resources

×