MiniFi and Apache NiFi : IoT in Berlin Germany 2018

Timothy Spann
Timothy SpannDeveloper Advocate
1 © Hortonworks Inc. 2011–2018. All rights reserved.
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
IoT With MiniFi and Apache NiFi
Berlin 2018
Timothy Spann, Solutions Engineer
Hortonworks @PaaSDev
2 © Hortonworks Inc. 2011–2018. All rights reserved.
Disclaimer
• This document may contain product features and technology directions that are under
development, may be under development in the future or may ultimately not be
developed.
• Technical feasibility, market demand, user feedback, and the Apache Software
Foundation community development process can all effect timing and final delivery.
• This document’s description of these features and technology directions does not
represent a contractual commitment, promise or obligation from Hortonworks to deliver
these features in any generally available product.
• Product features and technology directions are subject to change, and must not be
included in contracts, purchase orders, or sales agreements of any kind.
• Since this document contains an outline of general product development plans,
customers should not rely upon it when making a purchase decision.
3 © Hortonworks Inc. 2011–2018. All rights reserved.
Agenda
• Apache MiniFi
• Apache NiFi
• Devices
• Demos
• Questions
4 © Hortonworks Inc. 2011–2018. All rights reserved.
What do we want to do?
• MiniFi ingests camera images and
sensor data
• MiniFi executes Apache MXNet at the
edge
• Run Apache MXNet Inception to
recognize objects in image
• Apache NiFi stores images, metadata
and enriched data in Hadoop
• Apache NiFi ingests social data and
REST feeds
• Apache OpenNLP and Apache Tika for
textual data
5 © Hortonworks Inc. 2011–2018. All rights reserved.
Aggregate all data from sensors, drones, logs, geo-location devices, images,
machines and social feeds.
Collect: Bring Together
Mediate point-to-point and bi-directional data flows, delivering data
reliably to Apache HBase, Apache Hive, HDFS, Slack and Email.
Conduct: Mediate the Data Flow
Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather,
location, Apache OpenNLP and Apache MXNet.
Curate: Gain Insights
6 © Hortonworks Inc. 2011–2018. All rights reserved.
Actionable Insights Architecture
Ingestion
Simple Event Processing
Engine
Stream Processing
Destination
Data Bus
Build
Predictive Model
From Historical Data
Deploy
Predictive Model
For Real-time Insights
Perishable Insights
Historical Insights
7 © Hortonworks Inc. 2011–2018. All rights reserved.
IIoT
Multiple devices, protocols, frameworks, languages, data types, sensors and networks
Protocols
• MQTT
• HTTPS / SSL (REST/JSON)
• OPC UA
• CoAP
• AMQP
• JSON
• XML
• CSV
• Raw Text
• Images (JPEG, PNG)
• Raw Data Streams
Data Types Sensors
• Cameras
• Temperature/Humidity
• IR
• Proximity
• Motion Sensors
• GPS
Protocols
• NVidia Jetson TX1
• Raspberry Pi
• Arduino
• TS-7800 V2
• ESP8266
• DragonBoard 410c
• BeagleBone Black
8 © Hortonworks Inc. 2011–2018. All rights reserved.
HDF Data-In-Motion Platform – with HDF 3.1 GA Release
9 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache MiNiFi
• NiFi lives in the data center. Give it an
enterprise server or a cluster of them.
• MiNiFi lives as close to where data is born
and is a guest on that device or system
“Let me get the key parts of NiFi close to where data begins and provide
bidirectional data transfer"
10 © Hortonworks Inc. 2011–2018. All rights reserved.
Edge Intelligence with Apache MiNiFi
à Guaranteed delivery
à Data buffering
‒ Backpressure
‒ Pressure release
à Prioritized queuing
à Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
à Data provenance
à Recovery / recording a rolling log
of fine-grained history
à Designed for extension
Different from Apache NiFi
à Design and Deploy
à Warm re-deploys
Key Features
11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Java Agent C++ Agent
StreamLine
Builder Ops Insight
Central HDP Cluster
HBase Druid
YARN
HDFS
Hive
Weather
Location
Sensors
12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
http://mxnet.incubator.apache.org/
• Cloud ready
• Experienced team (XGBoost)
• AWS, Microsoft, NVIDIA, Baidu, Intel backing
• Apache Incubator Project
• Run distributed on YARN
• In my early tests, faster than TensorFlow.
• Runs on Raspberry PI, Nvidia Jetson TX1
and other constrained devices
• Great documentation
• Gluon
• Great Python Interaction
• Model Server Available
• ONNX Support
• Now in Version 1.1!
• Great Model Zoo
https://mxnet.incubator.apache.org/how_to/cloud.html
https://github.com/apache/incubator-mxnet/tree/1.1.0/example
13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
• Apache MXNet via Execute Process (Python)
• Apache MXNet Running on Edge Nodes (MiniFi) S2S
• Apache MXNet Model Server Integration (REST API)
Not Covered Today
• *Dockerized Apache MXNet on Hadoop YARN 3 with NVidia GPU
• *Apache MXNet on Spark
Apache NiFi Integration with Apache MXNet Options
14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
• https://github.com/apache/incubator-mxnet/tree/master/tools/coreml
• https://github.com/Leliana/WhatsThis
• https://github.com/apache/incubator-mxnet/tree/master/amalgamation/jni
• https://hub.docker.com/r/mxnet/
• https://github.com/apache/incubator-mxnet/tree/master/scala-
package/spark
Other Options
15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Pre-Built Models
• CaffeNet
• SqueezeNet v1.1
• Inception v3
• Single Shot Detection (SSD)
• VGG19
• ResidualNet 152
• LSTM
http://mxnet.incubator.apache.org/model_zoo/index.html
https://github.com/dmlc/mxnet-model-gallery
16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
python3 -W ignore analyze.py
{"uuid": "mxnet_uuid_img_20180208204131", "top1pct": "30.0999999046", "top1":
"n02871525 bookshop, bookstore, bookstall", "top2pct": "23.7000003457", "top2":
"n04200800 shoe shop, shoe-shop, shoe store", "top3pct": "4.80000004172", "top3":
"n03141823 crutch", "top4pct": "2.89999991655", "top4": "n04370456 sweatshirt",
"top5pct": "2.80000008643", "top5": "n02834397 bib", "imagefilename":
"images/tx1_image_img_20180208204131.jpg", "runtime": "2"}
Apache MXNet via Python (OSX Local with WebCam)
https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an-
apache-nifi-15-instance-w.html
17 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet via Python (OSX Local with WebCam)
{"uuid":
"mxnet_uuid_img_20180208204131", "top1pct": "30.0999999046", "top1":
"n02871525 bookshop,
bookstore, bookstall", "top2pct": "23.7000003457","top2": "n04200800 shoe
shop, shoe-shop, shoe store",
"top3pct": "4.80000004172", "top3":"n03141823 crutch", "top4pct":
"2.89999991655",
"top4": "n04370456 sweatshirt", "top5pct":"2.80000008643", "top5": "n02834397
bib", "imagefilename":
"images/tx1_image_img_20180208204131.jpg", "runtime": "2"}
18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Installation on OSX
git clone https://github.com/apache/incubator-mxnet.git
cd incubator-mxnet
mkdir images
curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0)
Gecko/20100101 Firefox/45.0' --header 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' -
-header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep-alive'
'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L
tar -xvzf inception-bn.tar.gz
cp Inception-BN-0126.params Inception-BN-0000.params
brew install graphviz
pip install --upgrade pip
pip install --upgrade setuptools
pip install graphviz
pip install mxnet
http://mxnet.incubator.apache.org/install/index.html
19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Running on with Apache NiFi Node
20 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Installation on with Apache NiFi Node
git clone https://github.com/apache/incubator-mxnet.git
sudo yum groupinstall 'Development Tools' -y
sudo yum install cmake git pkgconfig -y
sudo yum install libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel
libtiff-devel libwebp-devel -y
sudo yum install libdc1394-devel libv4l-devel gstreamer-plugins-base-devel -y
sudo yum install gtk2-devel -y
sudo yum install tbb-devel eigen3-devel -y
pip install numpy
You will need a full Python development environment, C++ and I recommend
building OpenCV2.
https://community.hortonworks.com/articles/174227/apache-deep-learning-101-using-apache-mxnet-on-an.html
21 © Hortonworks Inc. 2011–2018. All rights reserved.
Data Ingestion: Optimize Log Analytics with Content Based Routing
22 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache MXNet Running on Edge Nodes (MiniFi)
https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html
https://github.com/tspannhw/mxnet_rpi
https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-
running-apac.html
23 © Hortonworks Inc. 2011–2018. All rights reserved.
Edge Analytics with NVidia Jetson TX1 Running Apache MXNet (MiniFi)
https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-running-
apac.html
24 © Hortonworks Inc. 2011–2018. All rights reserved.
Using Apache MXNet on The Edge with Sensors and Intel Movidius
(MiniFi)
https://community.hortonworks.com/articles/176932/apache-deep-learning-101-using-apache-mxnet-on-
the.html
25 © Hortonworks Inc. 2011–2018. All rights reserved.
IIoT Processing
Data Acquisition
Edge Processing
Real Time Stream Analytics
Rapid Application Development
IoT
ANALYTICS
CLOUD
Acquire Move
Routing
&
Filtering
Deliver Parse Analysis
Aggregation
Modeling
26 © Hortonworks Inc. 2011–2018. All rights reserved.
Streaming Analytics
Manager
Part of MiniFi C++ Agent
Detect metadata and data
Extract metadata and data
Content Analysis
Deep Learning Framework
Complex Event Processing
Joining DataSets for Streaming Analytics
Open Source Components
Enabling Record Processing
Schema Management
27 © Hortonworks Inc. 2011–2018. All rights reserved.
Streaming Analytics
Manager
Run everywhere
Orchestration
Queueing
Simple Event Processing
Deep Learning Framework
Entity Resolution
Natural Language Processing
Open Source Components
Edge Ingest
Data Buffering
Device Integration
28 © Hortonworks Inc. 2011–2018. All rights reserved.
Flow
Apache NiFi
2
Apache NiFi
3
Apache Kafka
4
Apache Spark Streaming, SAM, Apache Storm
5
Apache MiniFi, Apache Mxnet1 Edge Devices
Gateways &
Routers
Ingestion
Platform
Data Bus
Stream
Processing
Engines
29 © Hortonworks Inc. 2011–2018. All rights reserved.
Flow
Apache Spark on YARN7
Apache Zeppelin8
Apache Hadoop
6 Storage Platform
Data Science
Platform
Dashboards
30 © Hortonworks Inc. 2011–2018. All rights reserved.
Questions?
31 © Hortonworks Inc. 2011–2018. All rights reserved.
Contact
https://github.com/tspannhw/ApacheDeepLearning101
https://community.hortonworks.com/users/9304/tspann.html
https://dzone.com/users/297029/bunkertor.html
https://www.meetup.com/futureofdata-princeton/
https://twitter.com/PaaSDev
https://community.hortonworks.com/articles/155435/using-the-new-mxnet-model-server.html
@PaasDev
32 © Hortonworks Inc. 2011–2018. All rights reserved.
Hortonworks Community Connection
Read access for everyone, join to participate and be recognized
• Full Q&A Platform (like StackOverflow)
• Knowledge Base Articles
• Code Samples and Repositories
33 © Hortonworks Inc. 2011–2018. All rights reserved.
Community Engagement
Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved
4,000+
Registered Users
10,000+
Answers
15,000+
Technical Assets
One Website!
34 © Hortonworks Inc. 2011–2018. All rights reserved.
https://community.hortonworks.com/articles/161761/new-features-in-apache-nifi-
15-apache-nifi-registr.html
https://community.hortonworks.com/articles/171787/hdf-31-executing-apache-
spark-via-executesparkinte.html
https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an-
apache-nifi-15-instance-w.html
https://community.hortonworks.com/articles/171893/hdf-31-executing-apache-
spark-via-executesparkinte-1.html
Resources
1 of 34

Recommended

Open Source Predictive Analytics Pipeline with Apache NiFi and MiniFi Princeton by
Open Source Predictive Analytics Pipeline with Apache NiFi and MiniFi PrincetonOpen Source Predictive Analytics Pipeline with Apache NiFi and MiniFi Princeton
Open Source Predictive Analytics Pipeline with Apache NiFi and MiniFi PrincetonTimothy Spann
1.1K views34 slides
Apache MXNet for IoT with Apache NiFi by
Apache MXNet for IoT with Apache NiFiApache MXNet for IoT with Apache NiFi
Apache MXNet for IoT with Apache NiFiTimothy Spann
1.2K views23 slides
Deep learning on HDP 2018 Prague by
Deep learning on HDP 2018 PragueDeep learning on HDP 2018 Prague
Deep learning on HDP 2018 PragueTimothy Spann
1.4K views46 slides
Apache Deep Learning 101 - DWS Berlin 2018 by
Apache Deep Learning 101 - DWS Berlin 2018Apache Deep Learning 101 - DWS Berlin 2018
Apache Deep Learning 101 - DWS Berlin 2018Timothy Spann
1.2K views42 slides
Enterprise IIoT Edge Processing with Apache NiFi by
Enterprise IIoT Edge Processing with Apache NiFiEnterprise IIoT Edge Processing with Apache NiFi
Enterprise IIoT Edge Processing with Apache NiFiTimothy Spann
1.1K views38 slides
Apache Nifi Crash Course by
Apache Nifi Crash CourseApache Nifi Crash Course
Apache Nifi Crash CourseDataWorks Summit
3.1K views33 slides

More Related Content

What's hot

Introduction to Apache NiFi 1.11.4 by
Introduction to Apache NiFi 1.11.4Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4Timothy Spann
1K views32 slides
Apache NiFi + Tensorflow + Hadoop: Big Data AI サンドイッチの作り方 by
Apache NiFi + Tensorflow + Hadoop:Big Data AI サンドイッチの作り方Apache NiFi + Tensorflow + Hadoop:Big Data AI サンドイッチの作り方
Apache NiFi + Tensorflow + Hadoop: Big Data AI サンドイッチの作り方HortonworksJapan
1.9K views43 slides
The First Mile - Edge and IoT Data Collection With Apache Nifi and MiniFi by
The First Mile - Edge and IoT Data Collection With Apache Nifi and MiniFiThe First Mile - Edge and IoT Data Collection With Apache Nifi and MiniFi
The First Mile - Edge and IoT Data Collection With Apache Nifi and MiniFiDataWorks Summit
596 views49 slides
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi by
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFiData at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFiAldrin Piri
897 views31 slides
The Avant-garde of Apache NiFi by
The Avant-garde of Apache NiFiThe Avant-garde of Apache NiFi
The Avant-garde of Apache NiFiDataWorks Summit/Hadoop Summit
1.3K views45 slides
Apache NiFi Crash Course - San Jose Hadoop Summit by
Apache NiFi Crash Course - San Jose Hadoop SummitApache NiFi Crash Course - San Jose Hadoop Summit
Apache NiFi Crash Course - San Jose Hadoop SummitAldrin Piri
778 views35 slides

What's hot(20)

Introduction to Apache NiFi 1.11.4 by Timothy Spann
Introduction to Apache NiFi 1.11.4Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4
Timothy Spann1K views
Apache NiFi + Tensorflow + Hadoop: Big Data AI サンドイッチの作り方 by HortonworksJapan
Apache NiFi + Tensorflow + Hadoop:Big Data AI サンドイッチの作り方Apache NiFi + Tensorflow + Hadoop:Big Data AI サンドイッチの作り方
Apache NiFi + Tensorflow + Hadoop: Big Data AI サンドイッチの作り方
HortonworksJapan1.9K views
The First Mile - Edge and IoT Data Collection With Apache Nifi and MiniFi by DataWorks Summit
The First Mile - Edge and IoT Data Collection With Apache Nifi and MiniFiThe First Mile - Edge and IoT Data Collection With Apache Nifi and MiniFi
The First Mile - Edge and IoT Data Collection With Apache Nifi and MiniFi
DataWorks Summit596 views
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi by Aldrin Piri
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFiData at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Aldrin Piri897 views
Apache NiFi Crash Course - San Jose Hadoop Summit by Aldrin Piri
Apache NiFi Crash Course - San Jose Hadoop SummitApache NiFi Crash Course - San Jose Hadoop Summit
Apache NiFi Crash Course - San Jose Hadoop Summit
Aldrin Piri778 views
Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose by Aldrin Piri
Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San JoseDataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose
Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose
Aldrin Piri2.3K views
Dataflow Management From Edge to Core with Apache NiFi by DataWorks Summit
Dataflow Management From Edge to Core with Apache NiFiDataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFi
DataWorks Summit864 views
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi by DataWorks Summit
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFiIntelligently Collecting Data at the Edge - Intro to Apache MiNiFi
Intelligently Collecting Data at the Edge - Intro to Apache MiNiFi
DataWorks Summit1.4K views
Dataflow Management From Edge to Core with Apache NiFi by DataWorks Summit
Dataflow Management From Edge to Core with Apache NiFiDataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFi
DataWorks Summit708 views
Running Apache NiFi with Apache Spark : Integration Options by Timothy Spann
Running Apache NiFi with Apache Spark : Integration OptionsRunning Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration Options
Timothy Spann6.2K views
BigData Techcon - Beyond Messaging with Apache NiFi by Aldrin Piri
BigData Techcon - Beyond Messaging with Apache NiFiBigData Techcon - Beyond Messaging with Apache NiFi
BigData Techcon - Beyond Messaging with Apache NiFi
Aldrin Piri215 views
Intelligently collecting data at the edge—intro to Apache MiNiFi by DataWorks Summit
Intelligently collecting data at the edge—intro to Apache MiNiFiIntelligently collecting data at the edge—intro to Apache MiNiFi
Intelligently collecting data at the edge—intro to Apache MiNiFi
DataWorks Summit2.1K views
The First Mile -- Edge and IoT Data Collection with Apache NiFi and MiNiFi by DataWorks Summit
The First Mile -- Edge and IoT Data Collection with Apache NiFi and MiNiFiThe First Mile -- Edge and IoT Data Collection with Apache NiFi and MiNiFi
The First Mile -- Edge and IoT Data Collection with Apache NiFi and MiNiFi
DataWorks Summit77 views
Apache NiFi: Ingesting Enterprise Data At Scale by Timothy Spann
Apache NiFi:   Ingesting Enterprise Data At Scale Apache NiFi:   Ingesting Enterprise Data At Scale
Apache NiFi: Ingesting Enterprise Data At Scale
Timothy Spann1.6K views
Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi by Bryan Bende
Taking DataFlow Management to the Edge with Apache NiFi/MiNiFiTaking DataFlow Management to the Edge with Apache NiFi/MiNiFi
Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi
Bryan Bende3K views
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi by DataWorks Summit
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFiThe First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi
The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi
DataWorks Summit2.8K views
Hadoop Operations - Past, Present, and Future by DataWorks Summit
Hadoop Operations - Past, Present, and FutureHadoop Operations - Past, Present, and Future
Hadoop Operations - Past, Present, and Future
DataWorks Summit201 views

Similar to MiniFi and Apache NiFi : IoT in Berlin Germany 2018

IoT with Apache MXNet and Apache NiFi and MiniFi by
IoT with Apache MXNet and Apache NiFi and MiniFiIoT with Apache MXNet and Apache NiFi and MiniFi
IoT with Apache MXNet and Apache NiFi and MiniFiDataWorks Summit
1.8K views23 slides
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018 by
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018Timothy Spann
18.7K views32 slides
Hands-On Deep Dive with MiniFi and Apache MXNet by
Hands-On Deep Dive with MiniFi and Apache MXNetHands-On Deep Dive with MiniFi and Apache MXNet
Hands-On Deep Dive with MiniFi and Apache MXNetTimothy Spann
770 views37 slides
Apache deep learning 101 by
Apache deep learning 101Apache deep learning 101
Apache deep learning 101DataWorks Summit
743 views29 slides
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox by
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache KnoxFortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache KnoxDataWorks Summit
498 views27 slides
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using... by
Hortonworks Technical Workshop:   HDP everywhere - cloud considerations using...Hortonworks Technical Workshop:   HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...Hortonworks
9.9K views79 slides

Similar to MiniFi and Apache NiFi : IoT in Berlin Germany 2018(20)

IoT with Apache MXNet and Apache NiFi and MiniFi by DataWorks Summit
IoT with Apache MXNet and Apache NiFi and MiniFiIoT with Apache MXNet and Apache NiFi and MiniFi
IoT with Apache MXNet and Apache NiFi and MiniFi
DataWorks Summit1.8K views
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018 by Timothy Spann
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
Timothy Spann18.7K views
Hands-On Deep Dive with MiniFi and Apache MXNet by Timothy Spann
Hands-On Deep Dive with MiniFi and Apache MXNetHands-On Deep Dive with MiniFi and Apache MXNet
Hands-On Deep Dive with MiniFi and Apache MXNet
Timothy Spann770 views
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox by DataWorks Summit
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache KnoxFortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
DataWorks Summit498 views
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using... by Hortonworks
Hortonworks Technical Workshop:   HDP everywhere - cloud considerations using...Hortonworks Technical Workshop:   HDP everywhere - cloud considerations using...
Hortonworks Technical Workshop: HDP everywhere - cloud considerations using...
Hortonworks9.9K views
Hadoop Everywhere & Cloudbreak by Sean Roberts
Hadoop Everywhere & CloudbreakHadoop Everywhere & Cloudbreak
Hadoop Everywhere & Cloudbreak
Sean Roberts2K views
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open... by DataWorks Summit
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...
DataWorks Summit1.7K views
Social Media Monitoring with NiFi, Druid and Superset by Thiago Santiago
Social Media Monitoring with NiFi, Druid and SupersetSocial Media Monitoring with NiFi, Druid and Superset
Social Media Monitoring with NiFi, Druid and Superset
Thiago Santiago1.2K views
Saving the elephant—now, not later by DataWorks Summit
Saving the elephant—now, not laterSaving the elephant—now, not later
Saving the elephant—now, not later
DataWorks Summit337 views
Discover.hdp2.2.storm and kafka.final by Hortonworks
Discover.hdp2.2.storm and kafka.finalDiscover.hdp2.2.storm and kafka.final
Discover.hdp2.2.storm and kafka.final
Hortonworks6.1K views
HDF 3.1 : An Introduction to New Features by Timothy Spann
HDF 3.1 : An Introduction to New FeaturesHDF 3.1 : An Introduction to New Features
HDF 3.1 : An Introduction to New Features
Timothy Spann1.5K views
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se... by DataWorks Summit
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...
Bridle your Flying Islands and Castles in the Sky: Built-in Governance and Se...
DataWorks Summit207 views
From leading IoT Protocols to Python Dashboarding_final by Lukas Ott
From leading IoT Protocols to Python Dashboarding_finalFrom leading IoT Protocols to Python Dashboarding_final
From leading IoT Protocols to Python Dashboarding_final
Lukas Ott66 views
Introduction to the Hadoop EcoSystem by Shivaji Dutta
Introduction to the Hadoop EcoSystemIntroduction to the Hadoop EcoSystem
Introduction to the Hadoop EcoSystem
Shivaji Dutta1.8K views
Apache Deep Learning 101 - ApacheCon Montreal 2018 v0.31 by Timothy Spann
Apache Deep Learning 101 - ApacheCon Montreal 2018 v0.31Apache Deep Learning 101 - ApacheCon Montreal 2018 v0.31
Apache Deep Learning 101 - ApacheCon Montreal 2018 v0.31
Timothy Spann736 views
Data Con LA 2018 - Streaming and IoT by Pat Alwell by Data Con LA
Data Con LA 2018 - Streaming and IoT by Pat AlwellData Con LA 2018 - Streaming and IoT by Pat Alwell
Data Con LA 2018 - Streaming and IoT by Pat Alwell
Data Con LA96 views

More from Timothy Spann

Building Real-Time Travel Alerts by
Building Real-Time Travel AlertsBuilding Real-Time Travel Alerts
Building Real-Time Travel AlertsTimothy Spann
115 views48 slides
JConWorld_ Continuous SQL with Kafka and Flink by
JConWorld_ Continuous SQL with Kafka and FlinkJConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and FlinkTimothy Spann
106 views36 slides
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines by
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data PipelinesTimothy Spann
98 views25 slides
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo by
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoEvolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoTimothy Spann
158 views8 slides
CoC23_ Looking at the New Features of Apache NiFi by
CoC23_ Looking at the New Features of Apache NiFiCoC23_ Looking at the New Features of Apache NiFi
CoC23_ Looking at the New Features of Apache NiFiTimothy Spann
36 views24 slides
CoC23_ Let’s Monitor The Conditions at the Conference by
CoC23_ Let’s Monitor The Conditions at the ConferenceCoC23_ Let’s Monitor The Conditions at the Conference
CoC23_ Let’s Monitor The Conditions at the ConferenceTimothy Spann
17 views17 slides

More from Timothy Spann(20)

Building Real-Time Travel Alerts by Timothy Spann
Building Real-Time Travel AlertsBuilding Real-Time Travel Alerts
Building Real-Time Travel Alerts
Timothy Spann115 views
JConWorld_ Continuous SQL with Kafka and Flink by Timothy Spann
JConWorld_ Continuous SQL with Kafka and FlinkJConWorld_ Continuous SQL with Kafka and Flink
JConWorld_ Continuous SQL with Kafka and Flink
Timothy Spann106 views
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines by Timothy Spann
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
[EN]DSS23_tspann_Integrating LLM with Streaming Data Pipelines
Timothy Spann98 views
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo by Timothy Spann
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines DemoEvolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Evolve 2023 NYC - Integrating AI Into Realtime Data Pipelines Demo
Timothy Spann158 views
CoC23_ Looking at the New Features of Apache NiFi by Timothy Spann
CoC23_ Looking at the New Features of Apache NiFiCoC23_ Looking at the New Features of Apache NiFi
CoC23_ Looking at the New Features of Apache NiFi
Timothy Spann36 views
CoC23_ Let’s Monitor The Conditions at the Conference by Timothy Spann
CoC23_ Let’s Monitor The Conditions at the ConferenceCoC23_ Let’s Monitor The Conditions at the Conference
CoC23_ Let’s Monitor The Conditions at the Conference
Timothy Spann17 views
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdf by Timothy Spann
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdfOSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdf
OSSFinance_UnlockingFinancialDatawithReal-TimePipelines.pdf
Timothy Spann23 views
CoC23_Utilizing Real-Time Transit Data for Travel Optimization by Timothy Spann
CoC23_Utilizing Real-Time Transit Data for Travel OptimizationCoC23_Utilizing Real-Time Transit Data for Travel Optimization
CoC23_Utilizing Real-Time Transit Data for Travel Optimization
Timothy Spann31 views
The Never Landing Stream with HTAP and Streaming by Timothy Spann
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and Streaming
Timothy Spann254 views
Meetup - Brasil - Data In Motion - 2023 September 19 by Timothy Spann
Meetup - Brasil - Data In Motion - 2023 September 19Meetup - Brasil - Data In Motion - 2023 September 19
Meetup - Brasil - Data In Motion - 2023 September 19
Timothy Spann319 views
Implement a Universal Data Distribution Architecture to Manage All Streaming ... by Timothy Spann
Implement a Universal Data Distribution Architecture to Manage All Streaming ...Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Implement a Universal Data Distribution Architecture to Manage All Streaming ...
Timothy Spann28 views
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data by Timothy Spann
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataBuilding Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Timothy Spann193 views
big data fest building modern data streaming apps by Timothy Spann
big data fest building modern data streaming appsbig data fest building modern data streaming apps
big data fest building modern data streaming apps
Timothy Spann317 views
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp by Timothy Spann
Using Apache NiFi with Apache Pulsar for Fast Data On-RampUsing Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Timothy Spann163 views
OSSNA Building Modern Data Streaming Apps by Timothy Spann
OSSNA Building Modern Data Streaming AppsOSSNA Building Modern Data Streaming Apps
OSSNA Building Modern Data Streaming Apps
Timothy Spann155 views
GSJUG: Mastering Data Streaming Pipelines 09May2023 by Timothy Spann
GSJUG: Mastering Data Streaming Pipelines 09May2023GSJUG: Mastering Data Streaming Pipelines 09May2023
GSJUG: Mastering Data Streaming Pipelines 09May2023
Timothy Spann255 views
BestInFlowCompetitionTutorials03May2023 by Timothy Spann
BestInFlowCompetitionTutorials03May2023BestInFlowCompetitionTutorials03May2023
BestInFlowCompetitionTutorials03May2023
Timothy Spann11 views
Cloudera Sandbox Event Guidelines For Workflow by Timothy Spann
Cloudera Sandbox Event Guidelines For WorkflowCloudera Sandbox Event Guidelines For Workflow
Cloudera Sandbox Event Guidelines For Workflow
Timothy Spann32 views
Meet the Committers Webinar_ Lab Preparation by Timothy Spann
Meet the Committers Webinar_ Lab PreparationMeet the Committers Webinar_ Lab Preparation
Meet the Committers Webinar_ Lab Preparation
Timothy Spann32 views

Recently uploaded

ESPC 2023 - Protect and Govern your Sensitive Data with Microsoft Purview in ... by
ESPC 2023 - Protect and Govern your Sensitive Data with Microsoft Purview in ...ESPC 2023 - Protect and Govern your Sensitive Data with Microsoft Purview in ...
ESPC 2023 - Protect and Govern your Sensitive Data with Microsoft Purview in ...Jasper Oosterveld
19 views49 slides
20231123_Camunda Meetup Vienna.pdf by
20231123_Camunda Meetup Vienna.pdf20231123_Camunda Meetup Vienna.pdf
20231123_Camunda Meetup Vienna.pdfPhactum Softwareentwicklung GmbH
41 views73 slides
Evolving the Network Automation Journey from Python to Platforms by
Evolving the Network Automation Journey from Python to PlatformsEvolving the Network Automation Journey from Python to Platforms
Evolving the Network Automation Journey from Python to PlatformsNetwork Automation Forum
13 views21 slides
MVP and prioritization.pdf by
MVP and prioritization.pdfMVP and prioritization.pdf
MVP and prioritization.pdfrahuldharwal141
31 views8 slides
SAP Automation Using Bar Code and FIORI.pdf by
SAP Automation Using Bar Code and FIORI.pdfSAP Automation Using Bar Code and FIORI.pdf
SAP Automation Using Bar Code and FIORI.pdfVirendra Rai, PMP
23 views38 slides
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors by
TouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective SensorsTouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective Sensors
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensorssugiuralab
21 views15 slides

Recently uploaded(20)

ESPC 2023 - Protect and Govern your Sensitive Data with Microsoft Purview in ... by Jasper Oosterveld
ESPC 2023 - Protect and Govern your Sensitive Data with Microsoft Purview in ...ESPC 2023 - Protect and Govern your Sensitive Data with Microsoft Purview in ...
ESPC 2023 - Protect and Govern your Sensitive Data with Microsoft Purview in ...
SAP Automation Using Bar Code and FIORI.pdf by Virendra Rai, PMP
SAP Automation Using Bar Code and FIORI.pdfSAP Automation Using Bar Code and FIORI.pdf
SAP Automation Using Bar Code and FIORI.pdf
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors by sugiuralab
TouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective SensorsTouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective Sensors
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors
sugiuralab21 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson92 views
Business Analyst Series 2023 - Week 3 Session 5 by DianaGray10
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5
DianaGray10300 views
"Running students' code in isolation. The hard way", Yurii Holiuk by Fwdays
"Running students' code in isolation. The hard way", Yurii Holiuk "Running students' code in isolation. The hard way", Yurii Holiuk
"Running students' code in isolation. The hard way", Yurii Holiuk
Fwdays17 views
Piloting & Scaling Successfully With Microsoft Viva by Richard Harbridge
Piloting & Scaling Successfully With Microsoft VivaPiloting & Scaling Successfully With Microsoft Viva
Piloting & Scaling Successfully With Microsoft Viva

MiniFi and Apache NiFi : IoT in Berlin Germany 2018

  • 1. 1 © Hortonworks Inc. 2011–2018. All rights reserved. © Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information. IoT With MiniFi and Apache NiFi Berlin 2018 Timothy Spann, Solutions Engineer Hortonworks @PaaSDev
  • 2. 2 © Hortonworks Inc. 2011–2018. All rights reserved. Disclaimer • This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. • Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. • This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. • Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Since this document contains an outline of general product development plans, customers should not rely upon it when making a purchase decision.
  • 3. 3 © Hortonworks Inc. 2011–2018. All rights reserved. Agenda • Apache MiniFi • Apache NiFi • Devices • Demos • Questions
  • 4. 4 © Hortonworks Inc. 2011–2018. All rights reserved. What do we want to do? • MiniFi ingests camera images and sensor data • MiniFi executes Apache MXNet at the edge • Run Apache MXNet Inception to recognize objects in image • Apache NiFi stores images, metadata and enriched data in Hadoop • Apache NiFi ingests social data and REST feeds • Apache OpenNLP and Apache Tika for textual data
  • 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved. Aggregate all data from sensors, drones, logs, geo-location devices, images, machines and social feeds. Collect: Bring Together Mediate point-to-point and bi-directional data flows, delivering data reliably to Apache HBase, Apache Hive, HDFS, Slack and Email. Conduct: Mediate the Data Flow Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather, location, Apache OpenNLP and Apache MXNet. Curate: Gain Insights
  • 6. 6 © Hortonworks Inc. 2011–2018. All rights reserved. Actionable Insights Architecture Ingestion Simple Event Processing Engine Stream Processing Destination Data Bus Build Predictive Model From Historical Data Deploy Predictive Model For Real-time Insights Perishable Insights Historical Insights
  • 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved. IIoT Multiple devices, protocols, frameworks, languages, data types, sensors and networks Protocols • MQTT • HTTPS / SSL (REST/JSON) • OPC UA • CoAP • AMQP • JSON • XML • CSV • Raw Text • Images (JPEG, PNG) • Raw Data Streams Data Types Sensors • Cameras • Temperature/Humidity • IR • Proximity • Motion Sensors • GPS Protocols • NVidia Jetson TX1 • Raspberry Pi • Arduino • TS-7800 V2 • ESP8266 • DragonBoard 410c • BeagleBone Black
  • 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved. HDF Data-In-Motion Platform – with HDF 3.1 GA Release
  • 9. 9 © Hortonworks Inc. 2011–2018. All rights reserved. Apache MiNiFi • NiFi lives in the data center. Give it an enterprise server or a cluster of them. • MiNiFi lives as close to where data is born and is a guest on that device or system “Let me get the key parts of NiFi close to where data begins and provide bidirectional data transfer"
  • 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved. Edge Intelligence with Apache MiNiFi à Guaranteed delivery à Data buffering ‒ Backpressure ‒ Pressure release à Prioritized queuing à Flow specific QoS ‒ Latency vs. throughput ‒ Loss tolerance à Data provenance à Recovery / recording a rolling log of fine-grained history à Designed for extension Different from Apache NiFi à Design and Deploy à Warm re-deploys Key Features
  • 11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Java Agent C++ Agent StreamLine Builder Ops Insight Central HDP Cluster HBase Druid YARN HDFS Hive Weather Location Sensors
  • 12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved http://mxnet.incubator.apache.org/ • Cloud ready • Experienced team (XGBoost) • AWS, Microsoft, NVIDIA, Baidu, Intel backing • Apache Incubator Project • Run distributed on YARN • In my early tests, faster than TensorFlow. • Runs on Raspberry PI, Nvidia Jetson TX1 and other constrained devices • Great documentation • Gluon • Great Python Interaction • Model Server Available • ONNX Support • Now in Version 1.1! • Great Model Zoo https://mxnet.incubator.apache.org/how_to/cloud.html https://github.com/apache/incubator-mxnet/tree/1.1.0/example
  • 13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved • Apache MXNet via Execute Process (Python) • Apache MXNet Running on Edge Nodes (MiniFi) S2S • Apache MXNet Model Server Integration (REST API) Not Covered Today • *Dockerized Apache MXNet on Hadoop YARN 3 with NVidia GPU • *Apache MXNet on Spark Apache NiFi Integration with Apache MXNet Options
  • 14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved • https://github.com/apache/incubator-mxnet/tree/master/tools/coreml • https://github.com/Leliana/WhatsThis • https://github.com/apache/incubator-mxnet/tree/master/amalgamation/jni • https://hub.docker.com/r/mxnet/ • https://github.com/apache/incubator-mxnet/tree/master/scala- package/spark Other Options
  • 15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Pre-Built Models • CaffeNet • SqueezeNet v1.1 • Inception v3 • Single Shot Detection (SSD) • VGG19 • ResidualNet 152 • LSTM http://mxnet.incubator.apache.org/model_zoo/index.html https://github.com/dmlc/mxnet-model-gallery
  • 16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved python3 -W ignore analyze.py {"uuid": "mxnet_uuid_img_20180208204131", "top1pct": "30.0999999046", "top1": "n02871525 bookshop, bookstore, bookstall", "top2pct": "23.7000003457", "top2": "n04200800 shoe shop, shoe-shop, shoe store", "top3pct": "4.80000004172", "top3": "n03141823 crutch", "top4pct": "2.89999991655", "top4": "n04370456 sweatshirt", "top5pct": "2.80000008643", "top5": "n02834397 bib", "imagefilename": "images/tx1_image_img_20180208204131.jpg", "runtime": "2"} Apache MXNet via Python (OSX Local with WebCam) https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an- apache-nifi-15-instance-w.html
  • 17. 17 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet via Python (OSX Local with WebCam) {"uuid": "mxnet_uuid_img_20180208204131", "top1pct": "30.0999999046", "top1": "n02871525 bookshop, bookstore, bookstall", "top2pct": "23.7000003457","top2": "n04200800 shoe shop, shoe-shop, shoe store", "top3pct": "4.80000004172", "top3":"n03141823 crutch", "top4pct": "2.89999991655", "top4": "n04370456 sweatshirt", "top5pct":"2.80000008643", "top5": "n02834397 bib", "imagefilename": "images/tx1_image_img_20180208204131.jpg", "runtime": "2"}
  • 18. 18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Installation on OSX git clone https://github.com/apache/incubator-mxnet.git cd incubator-mxnet mkdir images curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' - -header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep-alive' 'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L tar -xvzf inception-bn.tar.gz cp Inception-BN-0126.params Inception-BN-0000.params brew install graphviz pip install --upgrade pip pip install --upgrade setuptools pip install graphviz pip install mxnet http://mxnet.incubator.apache.org/install/index.html
  • 19. 19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Running on with Apache NiFi Node
  • 20. 20 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Installation on with Apache NiFi Node git clone https://github.com/apache/incubator-mxnet.git sudo yum groupinstall 'Development Tools' -y sudo yum install cmake git pkgconfig -y sudo yum install libpng-devel libjpeg-turbo-devel jasper-devel openexr-devel libtiff-devel libwebp-devel -y sudo yum install libdc1394-devel libv4l-devel gstreamer-plugins-base-devel -y sudo yum install gtk2-devel -y sudo yum install tbb-devel eigen3-devel -y pip install numpy You will need a full Python development environment, C++ and I recommend building OpenCV2. https://community.hortonworks.com/articles/174227/apache-deep-learning-101-using-apache-mxnet-on-an.html
  • 21. 21 © Hortonworks Inc. 2011–2018. All rights reserved. Data Ingestion: Optimize Log Analytics with Content Based Routing
  • 22. 22 © Hortonworks Inc. 2011–2018. All rights reserved. Apache MXNet Running on Edge Nodes (MiniFi) https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html https://github.com/tspannhw/mxnet_rpi https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1- running-apac.html
  • 23. 23 © Hortonworks Inc. 2011–2018. All rights reserved. Edge Analytics with NVidia Jetson TX1 Running Apache MXNet (MiniFi) https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-running- apac.html
  • 24. 24 © Hortonworks Inc. 2011–2018. All rights reserved. Using Apache MXNet on The Edge with Sensors and Intel Movidius (MiniFi) https://community.hortonworks.com/articles/176932/apache-deep-learning-101-using-apache-mxnet-on- the.html
  • 25. 25 © Hortonworks Inc. 2011–2018. All rights reserved. IIoT Processing Data Acquisition Edge Processing Real Time Stream Analytics Rapid Application Development IoT ANALYTICS CLOUD Acquire Move Routing & Filtering Deliver Parse Analysis Aggregation Modeling
  • 26. 26 © Hortonworks Inc. 2011–2018. All rights reserved. Streaming Analytics Manager Part of MiniFi C++ Agent Detect metadata and data Extract metadata and data Content Analysis Deep Learning Framework Complex Event Processing Joining DataSets for Streaming Analytics Open Source Components Enabling Record Processing Schema Management
  • 27. 27 © Hortonworks Inc. 2011–2018. All rights reserved. Streaming Analytics Manager Run everywhere Orchestration Queueing Simple Event Processing Deep Learning Framework Entity Resolution Natural Language Processing Open Source Components Edge Ingest Data Buffering Device Integration
  • 28. 28 © Hortonworks Inc. 2011–2018. All rights reserved. Flow Apache NiFi 2 Apache NiFi 3 Apache Kafka 4 Apache Spark Streaming, SAM, Apache Storm 5 Apache MiniFi, Apache Mxnet1 Edge Devices Gateways & Routers Ingestion Platform Data Bus Stream Processing Engines
  • 29. 29 © Hortonworks Inc. 2011–2018. All rights reserved. Flow Apache Spark on YARN7 Apache Zeppelin8 Apache Hadoop 6 Storage Platform Data Science Platform Dashboards
  • 30. 30 © Hortonworks Inc. 2011–2018. All rights reserved. Questions?
  • 31. 31 © Hortonworks Inc. 2011–2018. All rights reserved. Contact https://github.com/tspannhw/ApacheDeepLearning101 https://community.hortonworks.com/users/9304/tspann.html https://dzone.com/users/297029/bunkertor.html https://www.meetup.com/futureofdata-princeton/ https://twitter.com/PaaSDev https://community.hortonworks.com/articles/155435/using-the-new-mxnet-model-server.html @PaasDev
  • 32. 32 © Hortonworks Inc. 2011–2018. All rights reserved. Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
  • 33. 33 © Hortonworks Inc. 2011–2018. All rights reserved. Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved 4,000+ Registered Users 10,000+ Answers 15,000+ Technical Assets One Website!
  • 34. 34 © Hortonworks Inc. 2011–2018. All rights reserved. https://community.hortonworks.com/articles/161761/new-features-in-apache-nifi- 15-apache-nifi-registr.html https://community.hortonworks.com/articles/171787/hdf-31-executing-apache- spark-via-executesparkinte.html https://community.hortonworks.com/articles/171960/using-apache-mxnet-on-an- apache-nifi-15-instance-w.html https://community.hortonworks.com/articles/171893/hdf-31-executing-apache- spark-via-executesparkinte-1.html Resources