SlideShare a Scribd company logo
1 © Hortonworks Inc. 2011–2018. All rights reserved.
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
IoT Edge Processing with
Apache NiFi and MiniFi and
Apache MXNet for IoT NY 2018
Timothy Spann, Senior Solutions Engineer
Hortonworks @PaaSDev
2 © Hortonworks Inc. 2011–2018. All rights reserved.
Disclaimer
• This document may contain product features and technology directions that are under
development, may be under development in the future or may ultimately not be
developed.
• Technical feasibility, market demand, user feedback, and the Apache Software
Foundation community development process can all effect timing and final delivery.
• This document’s description of these features and technology directions does not
represent a contractual commitment, promise or obligation from Hortonworks to deliver
these features in any generally available product.
• Product features and technology directions are subject to change, and must not be
included in contracts, purchase orders, or sales agreements of any kind.
• Since this document contains an outline of general product development plans,
customers should not rely upon it when making a purchase decision.
3 © Hortonworks Inc. 2011–2018. All rights reserved.
What do we want to do?
• MiniFi ingests camera images and
sensor data
• MiniFi executes Apache MXNet at
the edge
• Run Apache MXNet Inception to
recognize objects in image
• Apache NiFi stores images,
metadata and enriched data in
Hadoop
4 © Hortonworks Inc. 2011–2018. All rights reserved.
Aggregate all data from sensors, drones, cameras, logs, geo-location
devices, images, machines and social feeds.
Collect: Bring Together
Mediate point-to-point and bi-directional data flows, delivering data
reliably to Files, Apache Kafka, Apache HBase, Apache Hive, HDFS, Slack
and Email.
Conduct: Mediate the Data Flow
Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather,
location, image recognition and object detection with TensorFlow and
Apache MXNet.
Curate: Gain Insights
5 © Hortonworks Inc. 2011–2018. All rights reserved.
{
"imagefilename" : "/opt/demo/images/2018-04-
17_1127.jpg",
"yaw" : 100.0,
"host" : "sensehatmovidius",
"top3" : "n06874185 traffic light, traffic signal,
stoplight",
"top5" : "n03773504 missile",
"humidity" : 31.2,
"uuid" : "mxnet_uuid_json_20180417152727.json",
"ipaddress" : 192.168.1.104,
"top2" : "n04286575 spotlight, spot",
"top3pct" : "6.199999898672104",
"top2pct" : "10.199999809265137",
"cputemp2" : 56.92,
"z" : 1.0,
"diskfree" : "4152.5 MB",
"top1pct" : "13.79999965429306",
"currenttime" : "2018-04-17 15:27:37",
"label2" : "n04592741 wing",
"pitch" : 360.0,
"pressure" : 1026.2,
"roll" : 1.0,
"label1" : "n04286575 spotlight, spot",
"top5pct" : "4.30000014603138",
"label4" : "n06874185 traffic light, traffic signal, stoplight",
"y" : 0.0,
"label3" : "n04009552 projector",
"cputemp" : 58,
"top1" : "n02930766 cab, hack, taxi, taxicab",
"top4pct" : "5.000000074505806",
"tempf" : 75.81,
"memory" : 56.5,
"top4" : "n03345487 fire engine, fire truck",
"starttime" : "2018-04-17 15:27:25",
"runtime" : "12",
"label5" : "n09229709 bubble",
"temp" : 35.45,
"x" : 0.0
}
Example Data
6 © Hortonworks Inc. 2011–2018. All rights reserved.
IoT For Enterprise
Multiple devices, protocols, frameworks, languages, data types, sensors and networks
Protocols
• MQTT
• HTTPS / SSL (REST/JSON)
• OPC UA
• CoAP
• AMQP
• TCP/IP
• UDP
• Zigbee
• BLE
• LoRaWAN
• JSON
• XML
• CSV
• Raw Text
• Images (JPEG, PNG)
• Raw Data Streams
• Videos
• Log Files
• Geo Data
• Audio
Data Types Sensors
• Cameras
• Temperature/Humidity
• IR
• Proximity
• Motion Sensors
• GPS
• LIDAR
• Gas
• RFID
Protocols
• NVidia Jetson TX1
• Raspberry Pi
• Arduino
• TS-7800 V2
• ESP8266
• DragonBoard 410c
• BeagleBone Black
• NanoPi
• Google AIY
• Tinkerboard
Devices
7 © Hortonworks Inc. 2011–2018. All rights reserved.
IIoT Processing
Data Acquisition
Edge Processing
Real Time Stream Analytics
Rapid Application Development
IoT
ANALYTICS
CLOUD
Acquire Move
Route
&
Filter
Deliver Parse Enrich Analyze
Aggregate
Model
8 © Hortonworks Inc. 2011–2018. All rights reserved.
Run everywhere
Orchestration
Queueing
Simple Event Processing
Deep Learning Framework
Enabling Record Processing
Schema Management
Open Source Components
Edge Ingest
Data Buffering
Device Integration
9 © Hortonworks Inc. 2011–2018. All rights reserved.
HDF Data-In-Motion Platform – with HDF 3.1
10 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache MiNiFi
• NiFi lives in the data center. Give it an
enterprise server or a cluster of them.
• MiNiFi lives as close to where data is born
and is a guest on that device or system
“Let me get the key parts of NiFi close to where data begins and provide
bidirectional data transfer"
11 © Hortonworks Inc. 2011–2018. All rights reserved.
Edge Intelligence with Apache MiNiFi
à Guaranteed delivery
à Data buffering
‒ Backpressure
‒ Pressure release
à Prioritized queuing
à Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
à Data provenance
à Recovery / recording a rolling log
of fine-grained history
à Designed for extension
Different from Apache NiFi
à Design and Deploy
à Warm re-deploys
Key Features
12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Java Agent C++ Agent
Multi Cloud HDP Clusters
HBase Druid
YARN
HDFS
Hive
Weather
Location
Sensors
13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
http://mxnet.incubator.apache.org/
• Cloud ready
• Experienced team (XGBoost)
• AWS, Microsoft, NVIDIA, Baidu, Intel backing
• Apache Incubator Project
• Run distributed on YARN
• In my early tests, faster than TensorFlow.
• Runs on Raspberry PI, NVidia Jetson TX1
and other constrained devices
• Great documentation
• Gluon
• Great Python Interaction
• Model Server Available
• ONNX Support
• Now in Version 1.1!
• Great Model Zoo
https://mxnet.incubator.apache.org/how_to/cloud.html
https://github.com/apache/incubator-mxnet/tree/1.1.0/example
14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Pre-Built Models
• CaffeNet
• SqueezeNet v1.1
• Inception v3
• Single Shot Detection (SSD)
• VGG19
• ResidualNet 152
• LSTM
http://mxnet.incubator.apache.org/model_zoo/index.html
https://github.com/dmlc/mxnet-model-gallery
15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Raspberry Pi Setup
pip install --upgrade pip
pip install scikit-image
git clone https://github.com/tspannhw/mxnet_rpi.git
sudo apt-get update -y
sudo apt-get install python-pip python-opencv python-scipy python-picamera -y
sudo apt-get -y install git cmake build-essential g++-4.8 c++-4.8 liblapack* libblas* libopencv*
git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet --branch 1.1.0
cd incubator-mxnet
export USE_OPENCV = 0
make
pip install mxnet==1.1.0
https://mxnet.incubator.apache.org/tutorials/embedded/wine_detector.html
16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Apache MXNet Raspberry Pi Setup
curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11;
rv:45.0) Gecko/20100101 Firefox/45.0' --header 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-
US,en;q=0.5' --header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep-
alive' 'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L
tar -xvzf inception-bn.tar.gz
cp Inception-BN-0126.params Inception-BN-0000.params
17 © Hortonworks Inc. 2011–2018. All rights reserved.
Apache MXNet Running on IoT Devices
https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html
https://github.com/tspannhw/mxnet_rpi
https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-
running-apac.html
18 © Hortonworks Inc. 2011–2018. All rights reserved.
GPU Powered Nodes with NVidia Jetson TX1 Running Apache MXNet
https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-running-apac.html
19 © Hortonworks Inc. 2011–2018. All rights reserved.
Using Apache MXNet on Devices with Sensors and Intel Movidius
https://community.hortonworks.com/articles/176932/apache-deep-learning-101-using-apache-mxnet-on-the.html
https://community.hortonworks.com/articles/155475/powering-apache-minifi-flows-with-a-movidius-neura.html
https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html
20 © Hortonworks Inc. 2011–2018. All rights reserved.
TensorFlow with MiniFi
https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html
https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html
https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
21 © Hortonworks Inc. 2011–2018. All rights reserved.
Human Integration
https://community.hortonworks.com/articles/121916/controlling-big-data-flows-with-gestures-minifi-ni.html
• Voice
• Touch
• Gestures
https://community.hortonworks.com/articles/167196/ingesting-data-from-the-matrix-creator-with-minifi.html
https://community.hortonworks.com/articles/155519/voice-controlled-data-flows-with-google-aiy-voice.html
22 © Hortonworks Inc. 2011–2018. All rights reserved.
Geolocation
https://community.hortonworks.com/articles/101679/iot-ingesting-gps-data-from-raspberry-pi-zero-wire.html
https://community.hortonworks.com/articles/155606/iot-ingesting-gps-data-from-odroid-xu4-devices-wit.html
https://community.hortonworks.com/articles/177137/ingesting-flight-data-ads-b-usb-receiver-with-apac.html
https://community.hortonworks.com/articles/178196/integrating-lucene-geo-gazetteer-for-geo-parsing-w.html
23 © Hortonworks Inc. 2011–2018. All rights reserved.
Weather Integration
https://community.hortonworks.com/articles/101904/part-2-iot-augmenting-gps-data-with-weather.html
24 © Hortonworks Inc. 2011–2018. All rights reserved.
Image Analytics
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html
https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html
https://community.hortonworks.com/articles/182850/vision-thing.html
https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html
25 © Hortonworks Inc. 2011–2018. All rights reserved.
Bluetooth Low Energy (BLE)
https://community.hortonworks.com/articles/108947/minifi-for-ble-bluetooth-low-energy-beacon-data-in.html
Beacons
IBeacon
NFC
https://community.hortonworks.com/articles/99861/ingesting-ibeacon-data-via-ble-to-mqtt-wifi-gatewa.html
26 © Hortonworks Inc. 2011–2018. All rights reserved.
Sensors via Python API
{"tempf": 75.14, "temp": 35.08,
"pitch": 1.0,
"diskfree": "1211.8 MB", "yaw": 55.0,
"cputemp": 52.08,
"ts": "2017-06-16 17:39:08",
"humidity": 41.5, "pressure": 0.0,
"memory": 23.0, "y": 0.0, "x": -1.0, "z": 1.0,
"roll": 1.0}
https://community.hortonworks.com/articles/108966/minifi-for-sensor-data-ingest-from-devices.html
https://community.hortonworks.com/articles/110475/ingesting-sensor-data-from-raspberry-pis-running-r.html
https://community.hortonworks.com/articles/155326/monitoring-energy-usage-utilizing-apache-nifi-pyth.html
27 © Hortonworks Inc. 2011–2018. All rights reserved.
Text Processing
https://community.hortonworks.com/articles/178510/integration-apache-opennlp-184-into-apache-nifi-15.html
• Dates
• Locations
• Names
• Money
• Organizations
https://community.hortonworks.com/articles/52415/processing-social-media-feeds-in-stream-with-apach.html
Sentiment AnalysisNamed Entity Extraction
https://community.hortonworks.com/articles/76935/using-sentiment-analysis-and-nlp-tools-with-hdp-25.html
https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html
https://community.hortonworks.com/articles/81222/adding-stanford-corenlp-to-big-data-pipelines-apac.html
https://community.hortonworks.com/articles/81270/adding-stanford-corenlp-to-big-data-pipelines-apac-1.html
28 © Hortonworks Inc. 2011–2018. All rights reserved.
MQTT
https://community.hortonworks.com/articles/167199/sending-messages-and-displaying-them-on-an-oled-sc.html
https://community.hortonworks.com/articles/183474/iot-using-minifi-java-agent-to-send-mqtt-messages.html
https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html
https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html
https://community.hortonworks.com/articles/72420/ingesting
-remote-sensor-feeds-into-apache-phoenix.html
29 © Hortonworks Inc. 2011–2018. All rights reserved.
Questions?
30 © Hortonworks Inc. 2011–2018. All rights reserved.
Contact
https://github.com/tspannhw/ApacheDeepLearning101
https://community.hortonworks.com/users/9304/tspann.html
https://dzone.com/users/297029/bunkertor.html
https://www.meetup.com/futureofdata-princeton/
https://twitter.com/PaaSDev
@PaasDev
31 © Hortonworks Inc. 2011–2018. All rights reserved.
Hortonworks Community Connection
Read access for everyone, join to participate and be recognized
• Full Q&A Platform (like StackOverflow)
• Knowledge Base Articles
• Code Samples and Repositories
32 © Hortonworks Inc. 2011–2018. All rights reserved.
Community Engagement
Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved
4,000+
Registered Users
10,000+
Answers
15,000+
Technical Assets
One Website!

More Related Content

What's hot

Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafkaReal time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafka
Timothy Spann
 
Apache NiFi in the Hadoop Ecosystem
Apache NiFi in the Hadoop Ecosystem Apache NiFi in the Hadoop Ecosystem
Apache NiFi in the Hadoop Ecosystem
DataWorks Summit/Hadoop Summit
 
Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4
Timothy Spann
 
Nifi
NifiNifi
Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi
Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFiReal-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi
Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi
Timothy Spann
 
Hadoop Summit Tokyo Apache NiFi Crash Course
Hadoop Summit Tokyo Apache NiFi Crash CourseHadoop Summit Tokyo Apache NiFi Crash Course
Hadoop Summit Tokyo Apache NiFi Crash Course
DataWorks Summit/Hadoop Summit
 
Time-Series Apache HBase
Time-Series Apache HBaseTime-Series Apache HBase
Time-Series Apache HBase
HBaseCon
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Kai Wähner
 
Data ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFiData ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFi
Lev Brailovskiy
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache Kafka
Kai Wähner
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Kai Wähner
 
Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...
Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...
Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...
Hortonworks
 
NoSQL Databases: Why, what and when
NoSQL Databases: Why, what and whenNoSQL Databases: Why, what and when
NoSQL Databases: Why, what and when
Lorenzo Alberton
 
Introduction to Bigdata and HADOOP
Introduction to Bigdata and HADOOP Introduction to Bigdata and HADOOP
Introduction to Bigdata and HADOOP
vinoth kumar
 
Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)
Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)
Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)
Timothy Spann
 
Apache NiFi SDLC Improvements
Apache NiFi SDLC ImprovementsApache NiFi SDLC Improvements
Apache NiFi SDLC Improvements
Bryan Bende
 
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
DataWorks Summit
 
BYOP: Custom Processor Development with Apache NiFi
BYOP: Custom Processor Development with Apache NiFiBYOP: Custom Processor Development with Apache NiFi
BYOP: Custom Processor Development with Apache NiFi
DataWorks Summit
 
Spark streaming + kafka 0.10
Spark streaming + kafka 0.10Spark streaming + kafka 0.10
Spark streaming + kafka 0.10
Joan Viladrosa Riera
 
Apache Ranger
Apache RangerApache Ranger
Apache Ranger
Rommel Garcia
 

What's hot (20)

Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafkaReal time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafka
 
Apache NiFi in the Hadoop Ecosystem
Apache NiFi in the Hadoop Ecosystem Apache NiFi in the Hadoop Ecosystem
Apache NiFi in the Hadoop Ecosystem
 
Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4Introduction to Apache NiFi 1.11.4
Introduction to Apache NiFi 1.11.4
 
Nifi
NifiNifi
Nifi
 
Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi
Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFiReal-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi
Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi
 
Hadoop Summit Tokyo Apache NiFi Crash Course
Hadoop Summit Tokyo Apache NiFi Crash CourseHadoop Summit Tokyo Apache NiFi Crash Course
Hadoop Summit Tokyo Apache NiFi Crash Course
 
Time-Series Apache HBase
Time-Series Apache HBaseTime-Series Apache HBase
Time-Series Apache HBase
 
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaReal-Life Use Cases & Architectures for Event Streaming with Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka
 
Data ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFiData ingestion and distribution with apache NiFi
Data ingestion and distribution with apache NiFi
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache Kafka
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...
Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...
Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...
 
NoSQL Databases: Why, what and when
NoSQL Databases: Why, what and whenNoSQL Databases: Why, what and when
NoSQL Databases: Why, what and when
 
Introduction to Bigdata and HADOOP
Introduction to Bigdata and HADOOP Introduction to Bigdata and HADOOP
Introduction to Bigdata and HADOOP
 
Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)
Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)
Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)
 
Apache NiFi SDLC Improvements
Apache NiFi SDLC ImprovementsApache NiFi SDLC Improvements
Apache NiFi SDLC Improvements
 
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
 
BYOP: Custom Processor Development with Apache NiFi
BYOP: Custom Processor Development with Apache NiFiBYOP: Custom Processor Development with Apache NiFi
BYOP: Custom Processor Development with Apache NiFi
 
Spark streaming + kafka 0.10
Spark streaming + kafka 0.10Spark streaming + kafka 0.10
Spark streaming + kafka 0.10
 
Apache Ranger
Apache RangerApache Ranger
Apache Ranger
 

Similar to IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018

MiniFi and Apache NiFi : IoT in Berlin Germany 2018
MiniFi and Apache NiFi : IoT in Berlin Germany 2018MiniFi and Apache NiFi : IoT in Berlin Germany 2018
MiniFi and Apache NiFi : IoT in Berlin Germany 2018
Timothy Spann
 
IoT with Apache MXNet and Apache NiFi and MiniFi
IoT with Apache MXNet and Apache NiFi and MiniFiIoT with Apache MXNet and Apache NiFi and MiniFi
IoT with Apache MXNet and Apache NiFi and MiniFi
DataWorks Summit
 
Apache MXNet for IoT with Apache NiFi
Apache MXNet for IoT with Apache NiFiApache MXNet for IoT with Apache NiFi
Apache MXNet for IoT with Apache NiFi
Timothy Spann
 
Hands-On Deep Dive with MiniFi and Apache MXNet
Hands-On Deep Dive with MiniFi and Apache MXNetHands-On Deep Dive with MiniFi and Apache MXNet
Hands-On Deep Dive with MiniFi and Apache MXNet
Timothy Spann
 
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...
DataWorks Summit
 
Enterprise IIoT Edge Processing with Apache NiFi
Enterprise IIoT Edge Processing with Apache NiFiEnterprise IIoT Edge Processing with Apache NiFi
Enterprise IIoT Edge Processing with Apache NiFi
Timothy Spann
 
Apache deep learning 101
Apache deep learning 101Apache deep learning 101
Apache deep learning 101
DataWorks Summit
 
Apache Deep Learning 101 - DWS Berlin 2018
Apache Deep Learning 101 - DWS Berlin 2018Apache Deep Learning 101 - DWS Berlin 2018
Apache Deep Learning 101 - DWS Berlin 2018
Timothy Spann
 
Deep learning on HDP 2018 Prague
Deep learning on HDP 2018 PragueDeep learning on HDP 2018 Prague
Deep learning on HDP 2018 Prague
Timothy Spann
 
Apache NiFi + Tensorflow + Hadoop: Big Data AI サンドイッチの作り方
Apache NiFi + Tensorflow + Hadoop:Big Data AI サンドイッチの作り方Apache NiFi + Tensorflow + Hadoop:Big Data AI サンドイッチの作り方
Apache NiFi + Tensorflow + Hadoop: Big Data AI サンドイッチの作り方
HortonworksJapan
 
Social Media Monitoring with NiFi, Druid and Superset
Social Media Monitoring with NiFi, Druid and SupersetSocial Media Monitoring with NiFi, Druid and Superset
Social Media Monitoring with NiFi, Druid and Superset
Thiago Santiago
 
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFiData at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Aldrin Piri
 
State of the Apache NiFi Ecosystem & Community
State of the Apache NiFi Ecosystem & CommunityState of the Apache NiFi Ecosystem & Community
State of the Apache NiFi Ecosystem & Community
Accumulo Summit
 
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
DataWorks Summit
 
HDF 3.1 : An Introduction to New Features
HDF 3.1 : An Introduction to New FeaturesHDF 3.1 : An Introduction to New Features
HDF 3.1 : An Introduction to New Features
Timothy Spann
 
Data Con LA 2018 - Streaming and IoT by Pat Alwell
Data Con LA 2018 - Streaming and IoT by Pat AlwellData Con LA 2018 - Streaming and IoT by Pat Alwell
Data Con LA 2018 - Streaming and IoT by Pat Alwell
Data Con LA
 
Dataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFiDataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFi
DataWorks Summit
 
Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python
Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, PythonOpen Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python
Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python
Timothy Spann
 
Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...
Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...
Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...
Data Con LA
 
Dataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFiDataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFi
DataWorks Summit
 

Similar to IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018 (20)

MiniFi and Apache NiFi : IoT in Berlin Germany 2018
MiniFi and Apache NiFi : IoT in Berlin Germany 2018MiniFi and Apache NiFi : IoT in Berlin Germany 2018
MiniFi and Apache NiFi : IoT in Berlin Germany 2018
 
IoT with Apache MXNet and Apache NiFi and MiniFi
IoT with Apache MXNet and Apache NiFi and MiniFiIoT with Apache MXNet and Apache NiFi and MiniFi
IoT with Apache MXNet and Apache NiFi and MiniFi
 
Apache MXNet for IoT with Apache NiFi
Apache MXNet for IoT with Apache NiFiApache MXNet for IoT with Apache NiFi
Apache MXNet for IoT with Apache NiFi
 
Hands-On Deep Dive with MiniFi and Apache MXNet
Hands-On Deep Dive with MiniFi and Apache MXNetHands-On Deep Dive with MiniFi and Apache MXNet
Hands-On Deep Dive with MiniFi and Apache MXNet
 
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...
Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, Open...
 
Enterprise IIoT Edge Processing with Apache NiFi
Enterprise IIoT Edge Processing with Apache NiFiEnterprise IIoT Edge Processing with Apache NiFi
Enterprise IIoT Edge Processing with Apache NiFi
 
Apache deep learning 101
Apache deep learning 101Apache deep learning 101
Apache deep learning 101
 
Apache Deep Learning 101 - DWS Berlin 2018
Apache Deep Learning 101 - DWS Berlin 2018Apache Deep Learning 101 - DWS Berlin 2018
Apache Deep Learning 101 - DWS Berlin 2018
 
Deep learning on HDP 2018 Prague
Deep learning on HDP 2018 PragueDeep learning on HDP 2018 Prague
Deep learning on HDP 2018 Prague
 
Apache NiFi + Tensorflow + Hadoop: Big Data AI サンドイッチの作り方
Apache NiFi + Tensorflow + Hadoop:Big Data AI サンドイッチの作り方Apache NiFi + Tensorflow + Hadoop:Big Data AI サンドイッチの作り方
Apache NiFi + Tensorflow + Hadoop: Big Data AI サンドイッチの作り方
 
Social Media Monitoring with NiFi, Druid and Superset
Social Media Monitoring with NiFi, Druid and SupersetSocial Media Monitoring with NiFi, Druid and Superset
Social Media Monitoring with NiFi, Druid and Superset
 
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFiData at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
Data at Scales and the Values of Starting Small with Apache NiFi & MiNiFi
 
State of the Apache NiFi Ecosystem & Community
State of the Apache NiFi Ecosystem & CommunityState of the Apache NiFi Ecosystem & Community
State of the Apache NiFi Ecosystem & Community
 
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
Real-Time Ingesting and Transforming Sensor Data and Social Data with NiFi an...
 
HDF 3.1 : An Introduction to New Features
HDF 3.1 : An Introduction to New FeaturesHDF 3.1 : An Introduction to New Features
HDF 3.1 : An Introduction to New Features
 
Data Con LA 2018 - Streaming and IoT by Pat Alwell
Data Con LA 2018 - Streaming and IoT by Pat AlwellData Con LA 2018 - Streaming and IoT by Pat Alwell
Data Con LA 2018 - Streaming and IoT by Pat Alwell
 
Dataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFiDataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFi
 
Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python
Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, PythonOpen Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python
Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python
 
Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...
Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...
Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...
 
Dataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFiDataflow Management From Edge to Core with Apache NiFi
Dataflow Management From Edge to Core with Apache NiFi
 

More from Timothy Spann

06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
Timothy Spann
 
Startup Grind Princeton 18 June 2024 - AI Advancement
Startup Grind Princeton 18 June 2024 - AI AdvancementStartup Grind Princeton 18 June 2024 - AI Advancement
Startup Grind Princeton 18 June 2024 - AI Advancement
Timothy Spann
 
Startup Grind Princeton - Gen AI 240618 18 June 2024
Startup Grind Princeton - Gen AI 240618 18 June 2024Startup Grind Princeton - Gen AI 240618 18 June 2024
Startup Grind Princeton - Gen AI 240618 18 June 2024
Timothy Spann
 
06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus
Timothy Spann
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
Timothy Spann
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
Timothy Spann
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24  Building Real-Time Pipelines With FLaNKDATA SUMMIT 24  Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
Timothy Spann
 
Generative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and MilvusGenerative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and Milvus
Timothy Spann
 
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
Timothy Spann
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max Princeton
Timothy Spann
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Timothy Spann
 
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
Timothy Spann
 
28March2024-Codeless-Generative-AI-Pipelines
28March2024-Codeless-Generative-AI-Pipelines28March2024-Codeless-Generative-AI-Pipelines
28March2024-Codeless-Generative-AI-Pipelines
Timothy Spann
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
Timothy Spann
 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits
Timothy Spann
 
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Timothy Spann
 
Conf42-Python-Building Apache NiFi 2.0 Python Processors
Conf42-Python-Building Apache NiFi 2.0 Python ProcessorsConf42-Python-Building Apache NiFi 2.0 Python Processors
Conf42-Python-Building Apache NiFi 2.0 Python Processors
Timothy Spann
 
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Timothy Spann
 

More from Timothy Spann (20)

06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
06-20-2024-AI Camp Meetup-Unstructured Data and Vector Databases
 
Startup Grind Princeton 18 June 2024 - AI Advancement
Startup Grind Princeton 18 June 2024 - AI AdvancementStartup Grind Princeton 18 June 2024 - AI Advancement
Startup Grind Princeton 18 June 2024 - AI Advancement
 
Startup Grind Princeton - Gen AI 240618 18 June 2024
Startup Grind Princeton - Gen AI 240618 18 June 2024Startup Grind Princeton - Gen AI 240618 18 June 2024
Startup Grind Princeton - Gen AI 240618 18 June 2024
 
06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus06-18-2024-Princeton Meetup-Introduction to Milvus
06-18-2024-Princeton Meetup-Introduction to Milvus
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24  Building Real-Time Pipelines With FLaNKDATA SUMMIT 24  Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
 
Generative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and MilvusGenerative AI on Enterprise Cloud with NiFi and Milvus
Generative AI on Enterprise Cloud with NiFi and Milvus
 
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
April 2024 - NLIT Cloudera Real-Time LLM Streaming 2024
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max Princeton
 
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming PipelinesConf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
Conf42-LLM_Adding Generative AI to Real-Time Streaming Pipelines
 
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
 
28March2024-Codeless-Generative-AI-Pipelines
28March2024-Codeless-Generative-AI-Pipelines28March2024-Codeless-Generative-AI-Pipelines
28March2024-Codeless-Generative-AI-Pipelines
 
TCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI PipelinesTCFPro24 Building Real-Time Generative AI Pipelines
TCFPro24 Building Real-Time Generative AI Pipelines
 
2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits2024 Build Generative AI for Non-Profits
2024 Build Generative AI for Non-Profits
 
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
 
Conf42-Python-Building Apache NiFi 2.0 Python Processors
Conf42-Python-Building Apache NiFi 2.0 Python ProcessorsConf42-Python-Building Apache NiFi 2.0 Python Processors
Conf42-Python-Building Apache NiFi 2.0 Python Processors
 
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...
 

Recently uploaded

AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!
Tobias Schneck
 
Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!
Ortus Solutions, Corp
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
AWS Certified Solutions Architect Associate (SAA-C03)
AWS Certified Solutions Architect Associate (SAA-C03)AWS Certified Solutions Architect Associate (SAA-C03)
AWS Certified Solutions Architect Associate (SAA-C03)
HarpalGohil4
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
manji sharman06
 
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
Fwdays
 
AI in the Workplace Reskilling, Upskilling, and Future Work.pptx
AI in the Workplace Reskilling, Upskilling, and Future Work.pptxAI in the Workplace Reskilling, Upskilling, and Future Work.pptx
AI in the Workplace Reskilling, Upskilling, and Future Work.pptx
Sunil Jagani
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
christinelarrosa
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdfLee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
leebarnesutopia
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
zjhamm304
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
UiPathCommunity
 

Recently uploaded (20)

AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!
 
Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
AWS Certified Solutions Architect Associate (SAA-C03)
AWS Certified Solutions Architect Associate (SAA-C03)AWS Certified Solutions Architect Associate (SAA-C03)
AWS Certified Solutions Architect Associate (SAA-C03)
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
 
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba"NATO Hackathon Winner: AI-Powered Drug Search",  Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
 
AI in the Workplace Reskilling, Upskilling, and Future Work.pptx
AI in the Workplace Reskilling, Upskilling, and Future Work.pptxAI in the Workplace Reskilling, Upskilling, and Future Work.pptx
AI in the Workplace Reskilling, Upskilling, and Future Work.pptx
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdfLee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
 

IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018

  • 1. 1 © Hortonworks Inc. 2011–2018. All rights reserved. © Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information. IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018 Timothy Spann, Senior Solutions Engineer Hortonworks @PaaSDev
  • 2. 2 © Hortonworks Inc. 2011–2018. All rights reserved. Disclaimer • This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. • Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. • This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. • Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Since this document contains an outline of general product development plans, customers should not rely upon it when making a purchase decision.
  • 3. 3 © Hortonworks Inc. 2011–2018. All rights reserved. What do we want to do? • MiniFi ingests camera images and sensor data • MiniFi executes Apache MXNet at the edge • Run Apache MXNet Inception to recognize objects in image • Apache NiFi stores images, metadata and enriched data in Hadoop
  • 4. 4 © Hortonworks Inc. 2011–2018. All rights reserved. Aggregate all data from sensors, drones, cameras, logs, geo-location devices, images, machines and social feeds. Collect: Bring Together Mediate point-to-point and bi-directional data flows, delivering data reliably to Files, Apache Kafka, Apache HBase, Apache Hive, HDFS, Slack and Email. Conduct: Mediate the Data Flow Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather, location, image recognition and object detection with TensorFlow and Apache MXNet. Curate: Gain Insights
  • 5. 5 © Hortonworks Inc. 2011–2018. All rights reserved. { "imagefilename" : "/opt/demo/images/2018-04- 17_1127.jpg", "yaw" : 100.0, "host" : "sensehatmovidius", "top3" : "n06874185 traffic light, traffic signal, stoplight", "top5" : "n03773504 missile", "humidity" : 31.2, "uuid" : "mxnet_uuid_json_20180417152727.json", "ipaddress" : 192.168.1.104, "top2" : "n04286575 spotlight, spot", "top3pct" : "6.199999898672104", "top2pct" : "10.199999809265137", "cputemp2" : 56.92, "z" : 1.0, "diskfree" : "4152.5 MB", "top1pct" : "13.79999965429306", "currenttime" : "2018-04-17 15:27:37", "label2" : "n04592741 wing", "pitch" : 360.0, "pressure" : 1026.2, "roll" : 1.0, "label1" : "n04286575 spotlight, spot", "top5pct" : "4.30000014603138", "label4" : "n06874185 traffic light, traffic signal, stoplight", "y" : 0.0, "label3" : "n04009552 projector", "cputemp" : 58, "top1" : "n02930766 cab, hack, taxi, taxicab", "top4pct" : "5.000000074505806", "tempf" : 75.81, "memory" : 56.5, "top4" : "n03345487 fire engine, fire truck", "starttime" : "2018-04-17 15:27:25", "runtime" : "12", "label5" : "n09229709 bubble", "temp" : 35.45, "x" : 0.0 } Example Data
  • 6. 6 © Hortonworks Inc. 2011–2018. All rights reserved. IoT For Enterprise Multiple devices, protocols, frameworks, languages, data types, sensors and networks Protocols • MQTT • HTTPS / SSL (REST/JSON) • OPC UA • CoAP • AMQP • TCP/IP • UDP • Zigbee • BLE • LoRaWAN • JSON • XML • CSV • Raw Text • Images (JPEG, PNG) • Raw Data Streams • Videos • Log Files • Geo Data • Audio Data Types Sensors • Cameras • Temperature/Humidity • IR • Proximity • Motion Sensors • GPS • LIDAR • Gas • RFID Protocols • NVidia Jetson TX1 • Raspberry Pi • Arduino • TS-7800 V2 • ESP8266 • DragonBoard 410c • BeagleBone Black • NanoPi • Google AIY • Tinkerboard Devices
  • 7. 7 © Hortonworks Inc. 2011–2018. All rights reserved. IIoT Processing Data Acquisition Edge Processing Real Time Stream Analytics Rapid Application Development IoT ANALYTICS CLOUD Acquire Move Route & Filter Deliver Parse Enrich Analyze Aggregate Model
  • 8. 8 © Hortonworks Inc. 2011–2018. All rights reserved. Run everywhere Orchestration Queueing Simple Event Processing Deep Learning Framework Enabling Record Processing Schema Management Open Source Components Edge Ingest Data Buffering Device Integration
  • 9. 9 © Hortonworks Inc. 2011–2018. All rights reserved. HDF Data-In-Motion Platform – with HDF 3.1
  • 10. 10 © Hortonworks Inc. 2011–2018. All rights reserved. Apache MiNiFi • NiFi lives in the data center. Give it an enterprise server or a cluster of them. • MiNiFi lives as close to where data is born and is a guest on that device or system “Let me get the key parts of NiFi close to where data begins and provide bidirectional data transfer"
  • 11. 11 © Hortonworks Inc. 2011–2018. All rights reserved. Edge Intelligence with Apache MiNiFi à Guaranteed delivery à Data buffering ‒ Backpressure ‒ Pressure release à Prioritized queuing à Flow specific QoS ‒ Latency vs. throughput ‒ Loss tolerance à Data provenance à Recovery / recording a rolling log of fine-grained history à Designed for extension Different from Apache NiFi à Design and Deploy à Warm re-deploys Key Features
  • 12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Java Agent C++ Agent Multi Cloud HDP Clusters HBase Druid YARN HDFS Hive Weather Location Sensors
  • 13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved http://mxnet.incubator.apache.org/ • Cloud ready • Experienced team (XGBoost) • AWS, Microsoft, NVIDIA, Baidu, Intel backing • Apache Incubator Project • Run distributed on YARN • In my early tests, faster than TensorFlow. • Runs on Raspberry PI, NVidia Jetson TX1 and other constrained devices • Great documentation • Gluon • Great Python Interaction • Model Server Available • ONNX Support • Now in Version 1.1! • Great Model Zoo https://mxnet.incubator.apache.org/how_to/cloud.html https://github.com/apache/incubator-mxnet/tree/1.1.0/example
  • 14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Pre-Built Models • CaffeNet • SqueezeNet v1.1 • Inception v3 • Single Shot Detection (SSD) • VGG19 • ResidualNet 152 • LSTM http://mxnet.incubator.apache.org/model_zoo/index.html https://github.com/dmlc/mxnet-model-gallery
  • 15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Raspberry Pi Setup pip install --upgrade pip pip install scikit-image git clone https://github.com/tspannhw/mxnet_rpi.git sudo apt-get update -y sudo apt-get install python-pip python-opencv python-scipy python-picamera -y sudo apt-get -y install git cmake build-essential g++-4.8 c++-4.8 liblapack* libblas* libopencv* git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet --branch 1.1.0 cd incubator-mxnet export USE_OPENCV = 0 make pip install mxnet==1.1.0 https://mxnet.incubator.apache.org/tutorials/embedded/wine_detector.html
  • 16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache MXNet Raspberry Pi Setup curl --header 'Host: data.mxnet.io' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en- US,en;q=0.5' --header 'Referer: http://data.mxnet.io/models/imagenet/' --header 'Connection: keep- alive' 'http://data.mxnet.io/models/imagenet/inception-bn.tar.gz' -o 'inception-bn.tar.gz' -L tar -xvzf inception-bn.tar.gz cp Inception-BN-0126.params Inception-BN-0000.params
  • 17. 17 © Hortonworks Inc. 2011–2018. All rights reserved. Apache MXNet Running on IoT Devices https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html https://github.com/tspannhw/mxnet_rpi https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1- running-apac.html
  • 18. 18 © Hortonworks Inc. 2011–2018. All rights reserved. GPU Powered Nodes with NVidia Jetson TX1 Running Apache MXNet https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-running-apac.html
  • 19. 19 © Hortonworks Inc. 2011–2018. All rights reserved. Using Apache MXNet on Devices with Sensors and Intel Movidius https://community.hortonworks.com/articles/176932/apache-deep-learning-101-using-apache-mxnet-on-the.html https://community.hortonworks.com/articles/155475/powering-apache-minifi-flows-with-a-movidius-neura.html https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html
  • 20. 20 © Hortonworks Inc. 2011–2018. All rights reserved. TensorFlow with MiniFi https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
  • 21. 21 © Hortonworks Inc. 2011–2018. All rights reserved. Human Integration https://community.hortonworks.com/articles/121916/controlling-big-data-flows-with-gestures-minifi-ni.html • Voice • Touch • Gestures https://community.hortonworks.com/articles/167196/ingesting-data-from-the-matrix-creator-with-minifi.html https://community.hortonworks.com/articles/155519/voice-controlled-data-flows-with-google-aiy-voice.html
  • 22. 22 © Hortonworks Inc. 2011–2018. All rights reserved. Geolocation https://community.hortonworks.com/articles/101679/iot-ingesting-gps-data-from-raspberry-pi-zero-wire.html https://community.hortonworks.com/articles/155606/iot-ingesting-gps-data-from-odroid-xu4-devices-wit.html https://community.hortonworks.com/articles/177137/ingesting-flight-data-ads-b-usb-receiver-with-apac.html https://community.hortonworks.com/articles/178196/integrating-lucene-geo-gazetteer-for-geo-parsing-w.html
  • 23. 23 © Hortonworks Inc. 2011–2018. All rights reserved. Weather Integration https://community.hortonworks.com/articles/101904/part-2-iot-augmenting-gps-data-with-weather.html
  • 24. 24 © Hortonworks Inc. 2011–2018. All rights reserved. Image Analytics https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html https://community.hortonworks.com/articles/182850/vision-thing.html https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html
  • 25. 25 © Hortonworks Inc. 2011–2018. All rights reserved. Bluetooth Low Energy (BLE) https://community.hortonworks.com/articles/108947/minifi-for-ble-bluetooth-low-energy-beacon-data-in.html Beacons IBeacon NFC https://community.hortonworks.com/articles/99861/ingesting-ibeacon-data-via-ble-to-mqtt-wifi-gatewa.html
  • 26. 26 © Hortonworks Inc. 2011–2018. All rights reserved. Sensors via Python API {"tempf": 75.14, "temp": 35.08, "pitch": 1.0, "diskfree": "1211.8 MB", "yaw": 55.0, "cputemp": 52.08, "ts": "2017-06-16 17:39:08", "humidity": 41.5, "pressure": 0.0, "memory": 23.0, "y": 0.0, "x": -1.0, "z": 1.0, "roll": 1.0} https://community.hortonworks.com/articles/108966/minifi-for-sensor-data-ingest-from-devices.html https://community.hortonworks.com/articles/110475/ingesting-sensor-data-from-raspberry-pis-running-r.html https://community.hortonworks.com/articles/155326/monitoring-energy-usage-utilizing-apache-nifi-pyth.html
  • 27. 27 © Hortonworks Inc. 2011–2018. All rights reserved. Text Processing https://community.hortonworks.com/articles/178510/integration-apache-opennlp-184-into-apache-nifi-15.html • Dates • Locations • Names • Money • Organizations https://community.hortonworks.com/articles/52415/processing-social-media-feeds-in-stream-with-apach.html Sentiment AnalysisNamed Entity Extraction https://community.hortonworks.com/articles/76935/using-sentiment-analysis-and-nlp-tools-with-hdp-25.html https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html https://community.hortonworks.com/articles/81222/adding-stanford-corenlp-to-big-data-pipelines-apac.html https://community.hortonworks.com/articles/81270/adding-stanford-corenlp-to-big-data-pipelines-apac-1.html
  • 28. 28 © Hortonworks Inc. 2011–2018. All rights reserved. MQTT https://community.hortonworks.com/articles/167199/sending-messages-and-displaying-them-on-an-oled-sc.html https://community.hortonworks.com/articles/183474/iot-using-minifi-java-agent-to-send-mqtt-messages.html https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html https://community.hortonworks.com/articles/72420/ingesting -remote-sensor-feeds-into-apache-phoenix.html
  • 29. 29 © Hortonworks Inc. 2011–2018. All rights reserved. Questions?
  • 30. 30 © Hortonworks Inc. 2011–2018. All rights reserved. Contact https://github.com/tspannhw/ApacheDeepLearning101 https://community.hortonworks.com/users/9304/tspann.html https://dzone.com/users/297029/bunkertor.html https://www.meetup.com/futureofdata-princeton/ https://twitter.com/PaaSDev @PaasDev
  • 31. 31 © Hortonworks Inc. 2011–2018. All rights reserved. Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
  • 32. 32 © Hortonworks Inc. 2011–2018. All rights reserved. Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved 4,000+ Registered Users 10,000+ Answers 15,000+ Technical Assets One Website!