Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved. © Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information. Open Source Computer Vision with TensorFlow, MiniFi, Apache NiFi, OpenCV, Apache Tika and Python Timothy Spann, Solutions Engineer Hortonworks @PaaSDev Vision Thing

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved. Disclaimer • This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. • Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. • This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. • Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Since this document contains an outline of general product development plans, customers should not rely upon it when making a purchase decision.

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved. What? • For processing images from IoT devices like Raspberry Pis and NVidia Jetson TX1 and more that are equipped with attached cameras or external USB webcams, we use Python to interface via OpenCV and PiCamera. From there we run image processing at the edge on these IoT device using OpenCV and TensorFlow to determine attributes and image analytics. Apache MiniFi coordinates running these Python scripts and decides when and what to send from that analysis and the image to a remote Apache NiFi server for additional processing. • At the Apache NiFi cluster, in the cluster it routes the images to one processing path and the JSON encoded metadata to another flow. The JSON data (with its schema referenced from a central Schema Registry) is routed using Record Processing and SQL. This data in enriched and augmented before conversion to Apache ORC to be stored in HDFS with Apache Hive tables.

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved. So Why Am I Ingesting Images From Edge Devices? Use Cases Object Recognition • WebCam Security • Anomaly Detection • Logging • Metadata about images • Customer Analysis Image Classification Motion Estimation • Movement tracking • Security • Occupied Room Active Archive • Store all images • Training datasets • Joining With Other Data • Cameras Everywhere

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved. Ingestion Simple Event Processing Engine Stream Processing Destination Data Bus Build Predictive Model From Historical Data Deploy Predictive Model For Real-time Insights Perishable Insights Historical Insights Open Computer Vision Flow

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved. Open Source Image Analytical Components Streaming Analytics Manager Image Ingest Distributed queue Buffering Process decoupling Routing and Pre-Processing Orchestration Queueing Simple Event Processing Image Capture Image Processing

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved. Streaming Analytics Manager Part of MiniFi C++ Agent Detect metadata and data Extract metadata and data Content Analysis Deep Learning Framework Open Source Image Analytical Components Enabling Record Processing Schema Management

10. 11 © Hortonworks Inc. 2011–2018. All rights reserved. Aggregate all data from sensors, geo-location devices, machines and social feeds Collect: Bring Together Mediate point-to-point and bi-directional data flows, delivering data reliably to Apache HBase, Apache Hive, Slack and Email. Conduct: Mediate the Data Flow Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather, location, and TensorFlow. Curate: Gain Insights

11. 12 © Hortonworks Inc. 2011–2018. All rights reserved. { "imagefilename" : "/opt/demo/images/2018-04- 17_1127.jpg", "yaw" : 100.0, "host" : "sensehatmovidius", "top3" : "n06874185 traffic light, traffic signal, stoplight", "top5" : "n03773504 missile", "humidity" : 31.2, "uuid" : "uuid_json_20180417152727.json", "ipaddress" : 192.168.1.104, "top2" : "n04286575 spotlight, spot", "top3pct" : "6.199999898672104", "top2pct" : "10.199999809265137", "cputemp2" : 56.92, "z" : 1.0, "diskfree" : "4152.5 MB", "top1pct" : "13.79999965429306", "currenttime" : "2018-04-17 15:27:37", "label2" : "n04592741 wing", "pitch" : 360.0, "pressure" : 1026.2, "roll" : 1.0, "label1" : "n04286575 spotlight, spot", "top5pct" : "4.30000014603138", "label4" : "n06874185 traffic light, traffic signal, stoplight", "y" : 0.0, "label3" : "n04009552 projector", "cputemp" : 58, "top1" : "n02930766 cab, hack, taxi, taxicab", "top4pct" : "5.000000074505806", "tempf" : 75.81, "memory" : 56.5, "top4" : "n03345487 fire engine, fire truck", "starttime" : "2018-04-17 15:27:25", "runtime" : "12", "label5" : "n09229709 bubble", "temp" : 35.45, "x" : 0.0 } Example Data

13. 14 © Hortonworks Inc. 2011–2018. All rights reserved. What Is OpenCV? • OpenCV is a an open source computer vision library • Nearly 20 years old • Started by Intel • Current Version 3.4.1 • C++, Python and Java Interfaces • Runs on Windows, Linux, Mac, BSD, iOS, and Android. • Can be built from source with Make • Runs on Raspberry PIs https://en.wikipedia.org/wiki/OpenCV

14. 15 © Hortonworks Inc. 2011–2018. All rights reserved. What Can I Do with OpenCV? • Facial Recognition • Image Capture From Cameras • Object Identification • Motion Tracking • Pixel Manipulation • Image Properties • Image Data Manipulation • Image Processing including filtering, color conversion and histograms • Image Labelling https://github.com/jdye64/nifi-opencv https://www.learnopencv.com/ https://docs.opencv.org/3.4.0/d9/df8/tutorial_root.html

15. 16 © Hortonworks Inc. 2011–2018. All rights reserved. What Does OpenCV Python Code Look Like? https://community.hortonworks.com/articles/182850/vision-thing.html https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html https://github.com/aruizga7/Self-Driving-Car-in-DSX/tree/master/1.%20Line%20Lane%20Detection import cv2 cap = cv2.VideoCapture(0) ret, frame = cap.read() filename = ‘images/ilovedataworkssummit.jpg’ cv2.imwrite(filename, frame) img = cv2.cvtColor(cv2.imread(filename),cv2.COLOR_BGR2RGB) img = cv2.resize(img, (224, 224)) cv2.rectangle(image, (x, y), (x + w, y + h), (255, 255, 0), 2)

17. 18 © Hortonworks Inc. 2011–2018. All rights reserved. Apache NiFi Integration with TensorFlow Options • TensorFlow (C++, Python, Java) via ExecuteStreamCommand • TensorFlow NiFi Java Custom Processor • TensorFlow Running on Edge Nodes (MiniFi)

18. 19 © Hortonworks Inc. 2011–2018. All rights reserved. python classify_image.py --image_file /opt/demo/dronedata/Bebop2_20160920083655-0400.jpg solar dish, solar collector, solar furnace (score = 0.98316) window screen (score = 0.00196) manhole cover (score = 0.00070) radiator (score = 0.00041) doormat, welcome mat (score = 0.00041) bazel-bin/tensorflow/examples/label_image/label_image -- image=/opt/demo/dronedata/Bebop2_20160920083655-0400.jpg tensorflow/examples/label_image/main.cc:204] solar dish (577): 0.983162I tensorflow/examples/label_image/main.cc:204] window screen (912): 0.00196204I tensorflow/examples/label_image/main.cc:204] manhole cover (763): 0.000704005I tensorflow/examples/label_image/main.cc:204] radiator (571): 0.000408321I tensorflow/examples/label_image/main.cc:204] doormat (972): 0.000406186 TensorFlow via Python or C++ Binary (Java Library Is New!)

21. 22 © Hortonworks Inc. 2011–2018. All rights reserved. Why TensorFlow? Also Apache MXNet, PyTorch and DL4J. • Google • Multiple platform support • Hadoop integration • Spark integration • Keras • Large Community • Python and Java APIs • GPU Support • Mobile Support • Inception v3 • Clustering • Fully functional demos • Open Source • Apache Licensed • Large Model Library • Buzz • Extensive Documentation • Raspberry Pi Support

22. 23 © Hortonworks Inc. 2011–2018. All rights reserved. TensorFlow Java Processor in NiFi https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html https://github.com/tspannhw/nifi-tensorflow-processor

24. 25 © Hortonworks Inc. 2011–2018. All rights reserved. Apache Tika with Apache NiFi • Detection • Parsing • Output Formats including Text and HTML • Translation • Language Identification • Apache NiFi - Bundled ExtractMediaMetadata Processor • Apache NiFi - Extract the content metadata from flowfiles https://tika.apache.org/1.18/gettingstarted.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-media-nar/1.6.0/org.apache.nifi.processors.media.ExtractMediaMetadata/

25. 26 © Hortonworks Inc. 2011–2018. All rights reserved. HORTONWORKS DATA FLOW NIFI 1.2.0HDF 3.0 Jul 2017 1.0.0 HDF 2.0 Mar 2016 1.1.0 NiFiRegistry Ranger 0.7.0 0.5.0 0.6.0 Ambari 2.5.1 2.4.0 2.4.2 Kafka 0.10.1.0 0.9.0 0.10.0 Zookeeper 3.4.6 3.4.6 3.4.6 Storm 1.1.0 1.0.1 1.0.2SAM 0.5.0 SchemaRegistry 0.3.0 HDF 2.1 Aug 2016 Ongoing Innovation in Apache HDF 1.0 Dec 2014 0.3.0 0.6.1 HDF 1.2 Oct 2015 MiNiFiC++andJava0.2.0 Ongoing Innovation in OpenSource 1.0.0 0.0.1 0.10.0 HDF 3.1.1 Jan 2018 1.5.0 0.1.0 0.7.02.6.11.0.0 3.4.61.1.10.6.0 0.5.00.4.0 SECURITYSTREAMING & INTEGRATION OPERATIONS Hortonworks Data Flow 3.1.1 https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_release-notes/content/ch_hdf_relnotes.html

26. 27 © Hortonworks Inc. 2011–2018. All rights reserved. Apache Tika Supported File Formats • HTML, XML • Microsoft Word, Excel, PowerPoint, Outlook • OpenOffice • RSS • RTF • Zip, Tar, 7zip, Gzip, RAR • PDF • MP3, WAV, MIDI • MP4, FLV • TIFF, JPEG, PNG, BMP, GIF • And more!

27. 28 © Hortonworks Inc. 2011–2018. All rights reserved. Apache Tika with Apache NiFi https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html https://community.hortonworks.com/articles/81694/extracttext-nifi-custom-processor-powered-by-apach.html https://community.hortonworks.com/articles/76924/data-processing-pipeline-parsing-pdfs-and-identify.html https://github.com/tspannhw/nifi-extracttext-processor https://community.hortonworks.com/content/kbentry/177370/extracting-html-from-pdf-excel-and-word- documents.html

30. 31 © Hortonworks Inc. 2011–2018. All rights reserved. Why Apache NiFi? • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Hundreds of processors • Visual command and control • Over a fifty sources • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering • Version Control

31. 32 © Hortonworks Inc. 2011–2018. All rights reserved. Apache MiNiFi • NiFi lives in the data center. Give it an enterprise server or a cluster of them. • MiNiFi lives as close to where data is born and is a guest on that device or system “Let me get the key parts of NiFi close to where data begins and provide bidirectional data transfer"

32. 33 © Hortonworks Inc. 2011–2018. All rights reserved. Edge Intelligence with Apache MiNiFi  Guaranteed delivery  Data buffering ‒ Backpressure ‒ Pressure release  Prioritized queuing  Flow specific QoS ‒ Latency vs. throughput ‒ Loss tolerance  Data provenance  Recovery / recording a rolling log of fine-grained history  Designed for extension Different from Apache NiFi  Design and Deploy  Warm re-deploys Key Features

34. 35 © Hortonworks Inc. 2011–2018. All rights reserved. TensorFlow with MiniFi https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html

35. 36 © Hortonworks Inc. 2011–2018. All rights reserved. https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html https://community.hortonworks.com/articles/182850/vision-thing.html https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html Image Analytics

36. 37 © Hortonworks Inc. 2011–2018. All rights reserved. NiFi and Kafka Are Complementary NiFi Provide dataflow solution • Centralized management, from edge to core • Great traceability, event level data provenance starting when data is born • Interactive command and control – real time operational visibility • Dataflow management, including prioritization, back pressure, and edge intelligence • Visual representation of global dataflow Kafka Provide durable stream store • Low latency • Distributed data durability • Decentralized management of producers & consumers +

37. 38 © Hortonworks Inc. 2011–2018. All rights reserved. Integrated Provisioning and Security Kafka 1.0 Support To enhance data governance and lineage, users can now manage access control policies using resource or tag-based security in Ranger for Kafka 1.0 clusters. Users can now install, configure, manage, upgrade, monitor, and secure Kafka 1.0 clusters with Ambari. New processors in NiFi and Streaming Analytics Manager support Kafka 1.0 features including message headers and transactions.

38. 39 © Hortonworks Inc. 2011–2018. All rights reserved. What Is Apache Kafka? • Distributed streaming platform that allows publishing and subscribing to streams of records • Streams of records are organized into categories called topics • Topics can be partitioned and/or replicated • Records consist of a key, value, and timestamp http://kafka.apache.org/intro Kafka Cluster producer producer producer consumer consumer consumer APACHE KAFKA

43. 44 © Hortonworks Inc. 2011–2018. All rights reserved. SAM and Schema Registry Integration • Streaming Apps Require a Schema • Unlike NiFi, SAM requires a schema to build streaming analytics applications. • Every SAM builder component requires a schema to function. • SAM’s primary mechanism for connecting to a stream of data is Kafka, but Kafka does not have a schema. • This is where HDF’s Schema Registry component becomes incredibly valuable. • SAM’s Kafka Source Component integrated with Schema Registry • When you configure a Kafka source and supply kafka topic, SAM calls the Schema Registry. • Using the Kafka topic as the key, SAM will retrieve the schema. • This schema is then displayed on the tile component, and is passed to downstream components.

46. 47 © Hortonworks Inc. 2011–2018. All rights reserved. Contact https://community.hortonworks.com/users/9304/tspann.html https://dzone.com/users/297029/bunkertor.html https://www.meetup.com/futureofdata-princeton/ https://twitter.com/PaaSDev https://dzone.com/refcardz/introduction-to-tensorflow

47. 48 © Hortonworks Inc. 2011–2018. All rights reserved. Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories

48. 49 © Hortonworks Inc. 2011–2018. All rights reserved. Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2018. All Rights Reserved 20,000+ Registered Users 55,000+ Answers 87,000+ Technical Assets One Website!

Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python

DataWorks Summit

Open source computer vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika, and Python