A hands-on deep dive on using Apachee MiniFi with Apache MXNet on the edge device including Raspberry Pi with Movidius and NVidia Jetson TX1. We run deep learning models on the edge device and send images, sensor data and deep learning results if values exceed norms. Using S2S data is sent to NiFi for further processing, additional deep learning processing, data augmentation. A stream of data is landed as ORC files in HDFS with Hive tables on-top.
Processed data in AVRO format with a schema stored in Schema Registry. Visualization is shown in Zeppelin.
Use Cases: Security Camera Monitoring, Utility Asset Anomaly Detection, Temperature and Humdiity filtering for devices.
This talk builds on several existing articles I have written:
Flow Management – the key is edge to anywhere with intelligence. This means the crux of it is being able to connect anything with anything else, from anywhere. This is a guiding principle for the roadmap on this aspect of data in motion.
Stream Processing – the key is time to insight. This means the crux of this is to be able to extract actionable information as quickly and easily as possible. This is the guiding principle for this aspect of data in motion.
Enterprise services – goes without saying, this has to all work together, efficiently, reliable, effectively.
TALK TRACK Apache MiNiFI is a sub project of Apache NiFi. It is designed to solve the difficulties of managing and transmitting data feeds to and from the source of origin, enabling edge intelligence to adjust dataflow behavior with bi-directional communication, out to the last mile of digital signal. It has a very small and lightweight footprint*, and generate the same level of data provenance as NiFi that is vital to edge analytics and IoAT (Internet of Any Thing) It’s a little bit diferent from NiF in that is is not a real-time command and control interface – in fact – the agent, unlike NiFi doesn’t have a built in UI at all. MiNiFi is designed for design and deploy situations and for “warm re-deploys”.
HDF 2.0 supports the java version of the MiNiFi agent, and a C++ version is coming soon as well.
Adam Gibson DL4J/Skymind has spoken at my meetup Deep Learning A Practitioner’s Approach – I consulted with them on the Spark/Hadoop chapter.
Data Science Cheat Sheet - https://hortonworks.app.box.com/file/234426455072
White Paper - https://hortonworks.app.box.com/file/151460926459