This document discusses using open source computer vision tools like OpenCV, TensorFlow, Apache Tika, Apache NiFi and MiniFi for processing images from IoT devices. Python is used to interface with OpenCV and capture images via PiCamera on edge devices like Raspberry Pis. TensorFlow runs on the edge devices to analyze images and determine attributes. MiniFi coordinates running the Python scripts and sends the results to an Apache NiFi server for additional processing like storing in HDFS. The NiFi server routes images and metadata to different flows, enriching the data before storing in Hive tables.
Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined.
Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined.
Kafka
Reads events in memory and write to distributed log
Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined.
Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined.
Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined.
Monitor Time
Follow—ups
Q/A at end
Defer additional questions to later, we are short on time
Ingest – multiple options, different types of data (rdbms, streams, files)
HDF, Sqoop, Flume, Kafka
Streaming
Script vs UI + Mgmt. Data Movement tool. Streamlined.
Flow Management – the key is edge to anywhere with intelligence. This means the crux of it is being able to connect anything with anything else, from anywhere. This is a guiding principle for the roadmap on this aspect of data in motion.
Stream Processing – the key is time to insight. This means the crux of this is to be able to extract actionable information as quickly and easily as possible. This is the guiding principle for this aspect of data in motion.
Enterprise services – goes without saying, this has to all work together, efficiently, reliable, effectively.
TALK TRACK
Apache MiNiFI is a sub project of Apache NiFi.
It is designed to solve the difficulties of managing and transmitting data feeds to and from the source of origin, enabling edge intelligence to adjust dataflow behavior with bi-directional communication, out to the last mile of digital signal.
It has a very small and lightweight footprint*, and generate the same level of data provenance as NiFi that is vital to edge analytics and IoAT (Internet of Any Thing)
It’s a little bit diferent from NiF in that is is not a real-time command and control interface – in fact – the agent, unlike NiFi doesn’t have a built in UI at all.
MiNiFi is designed for design and deploy situations and for “warm re-deploys”.
HDF 2.0 supports the java version of the MiNiFi agent, and a C++ version is coming soon as well.
Apache Ambari support for Kafka 1.0 – Users can now install, configure, manage, upgrade, monitor, and secure Kafka 1.0 clusters with Ambari.
Apache Ranger support for Kafka 1.0 – To enhance data governance and lineage, users can now manage access control policies using resource or tag-based security for Kafka 1.0 clusters.
New NiFi and SAM processors for Kafka 1.0 – New processors in NiFi and Streaming Analytics Manager support Kafka 1.0 features including message headers and transactions.
Data Science Cheat Sheet - https://hortonworks.app.box.com/file/234426455072
White Paper - https://hortonworks.app.box.com/file/151460926459
Videos - https://www.youtube.com/user/Hortonworks/feed?activity_view=1