Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python

2,196 views

Published on

Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python

For processing images from IoT devices like Raspberry Pis, NVidia Jetson TX1, NanoPi Duos, and more that are equipped with attached cameras or external USB webcams, we use Python to interface via OpenCV and PiCamera. From there we run image processing at the edge on these IoT device using OpenCV and TensorFlow to determine attributes and image analytics. Apache MiniFi coordinates running these Python scripts and decides when and what to send from that analysis and the image to a remote Apache NiFi server for additional processing.
At the Apache NiFi cluster, in the cluster it routes the images to one processing path and the JSON encoded metadata to another flow. The JSON data (with its schema referenced from a central Schema Registry) is routed using Record Processing and SQL. This data in enriched and augmented before conversion to AVRO to be sent via Apache Kafka to SAM. Streaming Analytics Manager then does deeper processing on this stream and others including weather and twitter to determine what should be done on this data. https://github.com/tspannhw/OpenSourceComputerVision http://dataworkssummit.com/ San Jose 2018 DWS18

Published in: Software

Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python

  1. 1. 1 ©HortonworksInc. 2011–2018. All rightsreserved. © Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information. Open Source Computer Vision with TensorFlow, MiniFi, Apache NiFi, OpenCV, Apache Tika and Python Timothy Spann, Solutions Engineer Hortonworks @PaaSDev Vision Thing
  2. 2. 2 ©HortonworksInc. 2011–2018. All rightsreserved. Disclaimer • This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. • Technical feasibility, market demand, user feedback, and the Apache Software Foundation community development process can all effect timing and final delivery. • This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. • Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. • Since this document contains an outline of general product development plans, customers should not rely upon it when making a purchase decision.
  3. 3. 4 ©HortonworksInc. 2011–2018. All rightsreserved. Agenda • Architecture • OpenCV • TensorFlow • Apache Tika • Apache NiFi and MiniFi • Apache Kafka • Schema Registry • Streaming Analytics Manager • Demos • Questions
  4. 4. 5 ©HortonworksInc. 2011–2018. All rightsreserved. Use Cases So Why Am I Ingesting Images From Edge Devices? Object Recognition • WebCam Security • Anomaly Detection • Logging • Metadata about images • Customer Analysis Image Classification Motion Estimation • Movement tracking • Security • Occupied Room Active Archive • Store all images • Training datasets • Joining With Other Data • Cameras Everywhere
  5. 5. 6 ©HortonworksInc. 2011–2018. All rightsreserved. Architecture
  6. 6. 7 ©HortonworksInc. 2011–2018. All rightsreserved. Open Computer Vision Flow Ingestion Simple Event Processing Engine Stream Processing Destination Data Bus Build Predictive Model From Historical Data Deploy Predictive Model For Real-time Insights Perishable Insights Historical Insights
  7. 7. 8 ©HortonworksInc. 2011–2018. All rightsreserved. Open Source Image Analytical Components Streaming Analytics Manager Image Ingest Distributed queue Buffering Process decoupling Routing and Pre-Processing Orchestration Queueing Simple Event Processing Image Capture Image Processing
  8. 8. 9 ©HortonworksInc. 2011–2018. All rightsreserved. Streaming Analytics Manager Part of MiniFi C++ Agent Detect metadata and data Extract metadata and data Content Analysis Deep Learning Framework Open Source Image Analytical Components Enabling Record Processing Schema Management
  9. 9. 10 ©HortonworksInc. 2011–2018. All rightsreserved. Aggregate all data from sensors, geo-location devices, machines and social feeds Collect: Bring Together Mediate point-to-point and bi-directional data flows, delivering data reliably to Apache HBase, Apache Hive, Slack and Email. Conduct: Mediate the Data Flow Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather, location, and TensorFlow. Curate: Gain Insights
  10. 10. 11 ©HortonworksInc. 2011–2018. All rightsreserved. { "imagefilename" : "/opt/demo/images/2018-04- 17_1127.jpg", "yaw" : 100.0, "host" : "sensehatmovidius", "top3" : "n06874185 traffic light, traffic signal, stoplight", "top5" : "n03773504 missile", "humidity" : 31.2, "uuid" : "uuid_json_20180417152727.json", "ipaddress" : 192.168.1.104, "top2" : "n04286575 spotlight, spot", "top3pct" : "6.199999898672104", "top2pct" : "10.199999809265137", "cputemp2" : 56.92, "z" : 1.0, "diskfree" : "4152.5 MB", "top1pct" : "13.79999965429306", "currenttime" : "2018-04-17 15:27:37", "label2" : "n04592741 wing", "pitch" : 360.0, "pressure" : 1026.2, "roll" : 1.0, "label1" : "n04286575 spotlight, spot", "top5pct" : "4.30000014603138", "label4" : "n06874185 traffic light, traffic signal, stoplight", "y" : 0.0, "label3" : "n04009552 projector", "cputemp" : 58, "top1" : "n02930766 cab, hack, taxi, taxicab", "top4pct" : "5.000000074505806", "tempf" : 75.81, "memory" : 56.5, "top4" : "n03345487 fire engine, fire truck", "starttime" : "2018-04-17 15:27:25", "runtime" : "12", "label5" : "n09229709 bubble", "temp" : 35.45, "x" : 0.0 } Example Data
  11. 11. 12 ©HortonworksInc. 2011–2018. All rightsreserved. OpenCV
  12. 12. 13 ©HortonworksInc. 2011–2018. All rightsreserved. https://en.wikipedia.org/wiki/OpenCV • OpenCV is a an open source computer vision library • Nearly 20 years old • Started by Intel • Current Version 3.4.1 • C++, Python and Java Interfaces • Runs on Windows, Linux, Mac, BSD, iOS, and Android. • Can be built from source with Make • Runs on Raspberry Pis and other devices What is OpenCV?
  13. 13. 14 ©HortonworksInc. 2011–2018. All rightsreserved. https://github.com/jdye64/nifi-opencv • Facial Recognition • Image Capture From Cameras • Object Identification • Motion Tracking • Pixel Manipulation • Image Properties • Image Data Manipulation • Image Processing including filtering, color conversion and histograms • Image Labelling https://www.learnopencv.com/ https://docs.opencv.org/3.4.0/d9/df8/tutorial_root.html What can I do with OpenCV?
  14. 14. 15 ©HortonworksInc. 2011–2018. All rightsreserved. • https://community.hortonworks.com/articles/182850/vision-thing.html • https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html • https://github.com/aruizga7/Self-Driving-Car-in-DSX/tree/master/1.%20Line%20Lane%20Detection import cv2 cap = cv2.VideoCapture(0) ret, frame = cap.read() filename = ‘images/ilovedataworkssummit.jpg’ cv2.imwrite(filename, frame) img = cv2.cvtColor(cv2.imread(filename),cv2.COLOR_BGR2RGB) img = cv2.resize(img, (224, 224)) cv2.rectangle(image, (x, y), (x + w, y + h), (255, 255, 0), 2) What does OpenCV Python Code Look like?
  15. 15. 16 ©HortonworksInc. 2011–2018. All rightsreserved. TensorFlow
  16. 16. 17 ©HortonworksInc. 2011–2018. All rightsreserved. • TensorFlow (C++, Python, Java) via ExecuteStreamCommand • TensorFlow NiFi Java Custom Processor • TensorFlow Running on Edge Nodes (MiniFi) Apache NiFi Integration with TensorFlow Options
  17. 17. 18 ©HortonworksInc. 2011–2018. All rightsreserved. python classify_image.py --image_file /opt/demo/dronedata/Bebop2_20160920083655-0400.jpg solar dish, solar collector, solar furnace (score = 0.98316) window screen (score = 0.00196) manhole cover (score = 0.00070) radiator (score = 0.00041) doormat, welcome mat (score = 0.00041) bazel-bin/tensorflow/examples/label_image/label_image -- image=/opt/demo/dronedata/Bebop2_20160920083655-0400.jpg tensorflow/examples/label_image/main.cc:204] solar dish (577): 0.983162I tensorflow/examples/label_image/main.cc:204] window screen (912): 0.00196204I tensorflow/examples/label_image/main.cc:204] manhole cover (763): 0.000704005I tensorflow/examples/label_image/main.cc:204] radiator (571): 0.000408321I tensorflow/examples/label_image/main.cc:204] doormat (972): 0.000406186 TensorFlow via Python or C++ Binary (Java Library Is New!)
  18. 18. 19 ©HortonworksInc. 2011–2018. All rightsreserved. TensorFlow Python ExecuteStreamCommand NiFi https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html
  19. 19. 20 ©HortonworksInc. 2011–2018. All rightsreserved. Run TensorFlow on YARN 3.1 https://community.hortonworks.com/articles/83872/data-lake-30-containerization-erasure-coding-gpu-p.html
  20. 20. 21 ©HortonworksInc. 2011–2018. All rightsreserved. Why TensorFlow? Also Apache MXNet, PyTorch and DL4J. • Google • Multiple platform support • Hadoop integration • Spark integration • Keras • Large Community • Python and Java APIs • GPU Support • Mobile Support • Inception v3 • Clustering • Fully functional demos • Open Source • Apache Licensed • Large Model Library • Buzz • Extensive Documentation • Raspberry Pi Support
  21. 21. 22 ©HortonworksInc. 2011–2018. All rightsreserved. TensorFlow Java Processor in NiFi https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html https://github.com/tspannhw/nifi-tensorflow-processor
  22. 22. 23 ©HortonworksInc. 2011–2018. All rightsreserved. TensorFlow Running on Edge Nodes (MiniFi)
  23. 23. 24 ©HortonworksInc. 2011–2018. All rightsreserved. Apache Tika with Apache NiFi https://tika.apache.org/1.18/gettingstarted.html • Detection • Parsing • Output Formats including Text and HTML • Translation • Language Identification https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-media-nar/1.6.0/org.apache.nifi.processors.media.ExtractMediaMetadata/ • Apache NiFi - Bundled ExtractMediaMetadata Processor • Apache NiFi - Extract the content metadata from flowfiles
  24. 24. 25 ©HortonworksInc. 2011–2018. All rightsreserved. Apache Tika Supported File Formats • HTML, XML • Microsoft Word, Excel, PowerPoint, Outlook • OpenOffice • RSS • RTF • Zip, Tar, 7zip, Gzip, RAR • PDF • MP3, WAV, MIDI • MP4, FLV • TIFF, JPEG, PNG, BMP, GIF • And more!
  25. 25. 26 ©HortonworksInc. 2011–2018. All rightsreserved. Apache Tika with Apache NiFi https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html https://community.hortonworks.com/articles/81694/extracttext-nifi-custom-processor-powered-by-apach.html https://community.hortonworks.com/articles/76924/data-processing-pipeline-parsing-pdfs-and-identify.html https://github.com/tspannhw/nifi-extracttext-processor https://community.hortonworks.com/content/kbentry/177370/extracting-html-from-pdf-excel-and-word- documents.html
  26. 26. 27 ©HortonworksInc. 2011–2018. All rightsreserved. Apache Tika with Apache NiFi
  27. 27. 28 ©HortonworksInc. 2011–2018. All rightsreserved. HORTONWORKS DATA FLOW NIFI 1.2.0HDF3.0 Jul 2017 1.0.0HDF2.0 Mar 2016 1.1.0 NiFiRegistry Ranger 0.7.0 0.5.0 0.6.0 Ambari 2.5.1 2.4.0 2.4.2 Kafka 0.10.1.0 0.9.0 0.10.0 Zookeeper 3.4.6 3.4.6 3.4.6 Storm 1.1.0 1.0.1 1.0.2SAM 0.5.0 SchemaRegistry 0.3.0 HDF2.1 Aug2016 Ongoing Innovation in Apache HDF1.0 Dec2014 0.3.0 0.6.1HDF1.2 Oct 2015 MiNiFiC++andJava 0.2.0 Ongoing Innovation in OpenSource 1.0.0 0.0.1 0.10.0 HDF 3.1.2 June 2018 1.5.0 0.1.0 0.7.02.6.11.0.0 3.4.61.1.10.6.0 0.5.10.4.0 SECURITYSTREAM ING & INTEGRATION OPERATIONS Hortonworks Data Flow 3.1.2 https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_release-notes/content/ch_hdf_relnotes.html
  28. 28. 29 ©HortonworksInc. 2011–2018. All rightsreserved. HDF Data-In-Motion Platform – with HDF 3.1
  29. 29. 30 ©HortonworksInc. 2011–2018. All rightsreserved. Why Apache NiFi? • Guaranteed delivery • Data buffering - Backpressure - Pressure release • Prioritized queuing • Flow specific QoS - Latency vs. throughput - Loss tolerance • Data provenance • Supports push and pull models • Hundreds of processors • Visual command and control • Over a fifty sources • Flow templates • Pluggable/multi-role security • Designed for extension • Clustering • Version Control
  30. 30. 31 ©HortonworksInc. 2011–2018. All rightsreserved. Apache MiNiFi • NiFi lives in the data center. Give it an enterprise server or a cluster of them. • MiNiFi lives as close to where data is born and is a guest on that device or system “Let me get the key parts of NiFi close to where data begins and provide bidirectional data transfer"
  31. 31. 32 ©HortonworksInc. 2011–2018. All rightsreserved. Edge Intelligence with Apache MiNiFi à Guaranteed delivery à Data buffering ‒ Backpressure ‒ Pressure release à Prioritized queuing à Flow specific QoS ‒ Latency vs. throughput ‒ Loss tolerance à Data provenance à Recovery / recording a rolling log of fine-grained history à Designed for extension Different from Apache NiFi à Design and Deploy à Warm re-deploys Key Features
  32. 32. 33 ©HortonworksInc. 2011–2018. All rightsreserved. Custom Apache NiFi Processors for Open Source Computer Vision
  33. 33. 34 ©HortonworksInc. 2011–2018. All rightsreserved. TensorFlow with MiniFi https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
  34. 34. 35 ©HortonworksInc. 2011–2018. All rightsreserved. Image Analytics https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html https://community.hortonworks.com/articles/182850/vision-thing.html https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html
  35. 35. 36 ©HortonworksInc. 2011–2018. All rightsreserved. NiFi and Kafka Are Complementary NiFi Provide dataflow solution • Centralized management, from edge to core • Great traceability, event level data provenance starting when data is born • Interactive command and control – real time operational visibility • Dataflow management, including prioritization, back pressure, and edge intelligence • Visual representation of global dataflow Kafka Provide durable stream store • Low latency • Distributed data durability • Decentralized management of producers & consumers +
  36. 36. 37 ©HortonworksInc. 2011–2018. All rightsreserved. Integrated Provisioning and Security Kafka 1.0 Support To enhance data governance and lineage, users can now manage access control policies using resource or tag-based security in Ranger for Kafka 1.0 clusters. Users can now install, configure, manage, upgrade, monitor, and secure Kafka 1.0 clusters with Ambari. New processors in NiFi and Streaming Analytics Manager support Kafka 1.0 features including message headers and transactions.
  37. 37. 38 ©HortonworksInc. 2011–2018. All rightsreserved. What is Apache Kafka? • Distributed streaming platformthat allows publishing and subscribing to streams of records • Streams of records are organized into categories called topics • Topics can be partitioned and/or replicated • Records consist of a key, value, and timestamp http://kafka.apache.org/intro Kafka Cluster producer producer producer consumer consumer consumer APACHE KAFKA
  38. 38. 39 ©HortonworksInc. 2011–2018. All rightsreserved.
  39. 39. 40 ©HortonworksInc. 2011–2018. All rightsreserved. https://community.hortonworks.com/articles/177349/big-data-devops-apache-nifi- hwx-schema-registry-sc.html
  40. 40. 41 ©HortonworksInc. 2011–2018. All rightsreserved. Completion of Schema Lifecycle: Merged Schema from Dev Branch to Master
  41. 41. 42 ©HortonworksInc. 2011–2018. All rightsreserved. Schema Registry Support for Different “States”: Enable, Disable, Archive
  42. 42. 43 ©HortonworksInc. 2011–2018. All rightsreserved. SAM and Schema Registry Integration • Streaming Apps Require a Schema • Unlike NiFi, SAM requires a schema to build streaming analytics applications. • Every SAM builder component requires a schema to function. • SAM’s primary mechanism for connecting to a stream of data is Kafka, but Kafka does not have a schema. • This is where HDF’s Schema Registry component becomes incredibly valuable. • SAM’s Kafka Source Component integrated with Schema Registry • When you configure a Kafka source and supply kafka topic, SAM calls the Schema Registry. • Using the Kafka topic as the key, SAM will retrieve the schema. • This schema is then displayed on the tile component, and is passed to downstream components.
  43. 43. 44 ©HortonworksInc. 2011–2018. All rightsreserved. Streaming Analytics Manager
  44. 44. 45 ©HortonworksInc. 2011–2018. All rightsreserved. Streaming Analytics Manager
  45. 45. 46 ©HortonworksInc. 2011–2018. All rightsreserved. Thank you
  46. 46. 47 ©HortonworksInc. 2011–2018. All rightsreserved. Contact https://community.hortonworks.com/users/9304/tspann.html https://dzone.com/users/297029/bunkertor.html https://www.meetup.com/futureofdata-princeton/ https://twitter.com/PaaSDev https://dzone.com/articles/integrating-keras-tensorflow-yolov3-into- apache-ni
  47. 47. 48 ©HortonworksInc. 2011–2018. All rightsreserved. Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
  48. 48. 49 ©HortonworksInc. 2011–2018. All rightsreserved. Community Engagement Participate now at: community.hortonworks.com ©HortonworksInc. 2011–2015. All RightsReserved 4,000+ Registered Users 10,000+ Answers 15,000+ Technical Assets One Website!

×