Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python

1 ©HortonworksInc. 2011–2018. All rightsreserved.
© Hortonworks, Inc. 2011-2018. All rights reserved. | Hortonworks confidential and proprietary information.
Open Source Computer Vision with TensorFlow, MiniFi,
Apache NiFi, OpenCV, Apache Tika and Python
Timothy Spann, Solutions Engineer
Hortonworks @PaaSDev
Vision Thing

Disclaimer
• This document may contain product features and technology directions that are under
development, may be under development in the future or may ultimately not be
developed.
• Technical feasibility, market demand, user feedback, and the Apache Software
Foundation community development process can all effect timing and final delivery.
• This document’s description of these features and technology directions does not
represent a contractual commitment, promise or obligation from Hortonworks to deliver
these features in any generally available product.
• Product features and technology directions are subject to change, and must not be
included in contracts, purchase orders, or sales agreements of any kind.
• Since this document contains an outline of general product development plans,
customers should not rely upon it when making a purchase decision.

Agenda
• Architecture
• OpenCV
• TensorFlow
• Apache Tika
• Apache NiFi and MiniFi
• Apache Kafka
• Schema Registry
• Streaming Analytics Manager
• Demos
• Questions

Use Cases
So Why Am I Ingesting Images From Edge Devices?
Object Recognition
• WebCam Security
• Anomaly Detection
• Logging
• Metadata about images
• Customer Analysis
Image Classification Motion Estimation
• Movement tracking
• Security
• Occupied Room
Active Archive
• Store all images
• Training datasets
• Joining With Other Data
• Cameras Everywhere

Architecture

Open Computer Vision Flow
Ingestion
Simple Event Processing
Engine
Stream Processing
Destination
Data Bus
Build
Predictive Model
From Historical Data
Deploy
Predictive Model
For Real-time Insights
Perishable Insights
Historical Insights

Open Source Image Analytical Components
Streaming Analytics
Manager
Image Ingest
Distributed queue
Buffering
Process decoupling
Routing and Pre-Processing
Orchestration
Queueing
Simple Event Processing
Image Capture
Image Processing

Streaming Analytics
Manager
Part of MiniFi C++ Agent
Detect metadata and data
Extract metadata and data
Content Analysis
Deep Learning Framework
Open Source Image Analytical Components
Enabling Record Processing
Schema Management

Aggregate all data from sensors, geo-location devices, machines and social
feeds
Collect: Bring Together
Mediate point-to-point and bi-directional data flows, delivering data
reliably to Apache HBase, Apache Hive, Slack and Email.
Conduct: Mediate the Data Flow
Parse, filter, join, transform, fork, query, sort, dissect; enrich with weather,
location, and TensorFlow.
Curate: Gain Insights

{
"imagefilename" : "/opt/demo/images/2018-04-
17_1127.jpg",
"yaw" : 100.0,
"host" : "sensehatmovidius",
"top3" : "n06874185 traffic light, traffic signal,
stoplight",
"top5" : "n03773504 missile",
"humidity" : 31.2,
"uuid" : "uuid_json_20180417152727.json",
"ipaddress" : 192.168.1.104,
"top2" : "n04286575 spotlight, spot",
"top3pct" : "6.199999898672104",
"top2pct" : "10.199999809265137",
"cputemp2" : 56.92,
"z" : 1.0,
"diskfree" : "4152.5 MB",
"top1pct" : "13.79999965429306",
"currenttime" : "2018-04-17 15:27:37",
"label2" : "n04592741 wing",
"pitch" : 360.0,
"pressure" : 1026.2,
"roll" : 1.0,
"label1" : "n04286575 spotlight, spot",
"top5pct" : "4.30000014603138",
"label4" : "n06874185 traffic light, traffic signal, stoplight",
"y" : 0.0,
"label3" : "n04009552 projector",
"cputemp" : 58,
"top1" : "n02930766 cab, hack, taxi, taxicab",
"top4pct" : "5.000000074505806",
"tempf" : 75.81,
"memory" : 56.5,
"top4" : "n03345487 fire engine, fire truck",
"starttime" : "2018-04-17 15:27:25",
"runtime" : "12",
"label5" : "n09229709 bubble",
"temp" : 35.45,
"x" : 0.0
}
Example Data

OpenCV

https://en.wikipedia.org/wiki/OpenCV
• OpenCV is a an open source computer vision library
• Nearly 20 years old
• Started by Intel
• Current Version 3.4.1
• C++, Python and Java Interfaces
• Runs on Windows, Linux, Mac, BSD, iOS, and Android.
• Can be built from source with Make
• Runs on Raspberry Pis and other devices
What is OpenCV?

https://github.com/jdye64/nifi-opencv
• Facial Recognition
• Image Capture From Cameras
• Object Identification
• Motion Tracking
• Pixel Manipulation
• Image Properties
• Image Data Manipulation
• Image Processing including filtering, color conversion and histograms
• Image Labelling
https://www.learnopencv.com/
https://docs.opencv.org/3.4.0/d9/df8/tutorial_root.html
What can I do with OpenCV?

• https://community.hortonworks.com/articles/182850/vision-thing.html
• https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html
• https://github.com/aruizga7/Self-Driving-Car-in-DSX/tree/master/1.%20Line%20Lane%20Detection
import cv2
cap = cv2.VideoCapture(0)
ret, frame = cap.read()
filename = ‘images/ilovedataworkssummit.jpg’
cv2.imwrite(filename, frame)
img = cv2.cvtColor(cv2.imread(filename),cv2.COLOR_BGR2RGB)
img = cv2.resize(img, (224, 224))
cv2.rectangle(image, (x, y), (x + w, y + h), (255, 255, 0), 2)
What does OpenCV Python Code Look like?

TensorFlow

• TensorFlow (C++, Python, Java)
via ExecuteStreamCommand
• TensorFlow NiFi Java Custom Processor
• TensorFlow Running on Edge Nodes (MiniFi)
Apache NiFi Integration with TensorFlow Options

python classify_image.py --image_file /opt/demo/dronedata/Bebop2_20160920083655-0400.jpg
solar dish, solar collector, solar furnace (score = 0.98316)
window screen (score = 0.00196)
manhole cover (score = 0.00070)
radiator (score = 0.00041)
doormat, welcome mat (score = 0.00041)
bazel-bin/tensorflow/examples/label_image/label_image --
image=/opt/demo/dronedata/Bebop2_20160920083655-0400.jpg
tensorflow/examples/label_image/main.cc:204] solar dish (577): 0.983162I
tensorflow/examples/label_image/main.cc:204] window screen (912): 0.00196204I
tensorflow/examples/label_image/main.cc:204] manhole cover (763): 0.000704005I
tensorflow/examples/label_image/main.cc:204] radiator (571): 0.000408321I
tensorflow/examples/label_image/main.cc:204] doormat (972): 0.000406186
TensorFlow via Python or C++ Binary (Java Library Is New!)

TensorFlow Python ExecuteStreamCommand NiFi
https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html

Run TensorFlow on YARN 3.1
https://community.hortonworks.com/articles/83872/data-lake-30-containerization-erasure-coding-gpu-p.html

Why TensorFlow? Also Apache MXNet, PyTorch and DL4J.
• Google
• Multiple platform
support
• Hadoop integration
• Spark integration
• Keras
• Large Community
• Python and Java APIs
• GPU Support
• Mobile Support
• Inception v3
• Clustering
• Fully functional demos
• Open Source
• Apache Licensed
• Large Model Library
• Buzz
• Extensive Documentation
• Raspberry Pi Support

TensorFlow Java Processor in NiFi
https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html
https://github.com/tspannhw/nifi-tensorflow-processor

TensorFlow Running on Edge Nodes (MiniFi)

Apache Tika with Apache NiFi
https://tika.apache.org/1.18/gettingstarted.html
• Detection
• Parsing
• Output Formats including Text and HTML
• Translation
• Language Identification
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-media-nar/1.6.0/org.apache.nifi.processors.media.ExtractMediaMetadata/
• Apache NiFi - Bundled ExtractMediaMetadata Processor
• Apache NiFi - Extract the content metadata from flowfiles

Apache Tika Supported File Formats
• HTML, XML
• Microsoft Word, Excel, PowerPoint, Outlook
• OpenOffice
• RSS
• RTF
• Zip, Tar, 7zip, Gzip, RAR
• PDF
• MP3, WAV, MIDI
• MP4, FLV
• TIFF, JPEG, PNG, BMP, GIF
• And more!

https://community.hortonworks.com/articles/163776/parsing-any-document-with-apache-nifi-15-with-apac.html
https://community.hortonworks.com/articles/81694/extracttext-nifi-custom-processor-powered-by-apach.html
https://community.hortonworks.com/articles/76924/data-processing-pipeline-parsing-pdfs-and-identify.html
https://github.com/tspannhw/nifi-extracttext-processor
https://community.hortonworks.com/content/kbentry/177370/extracting-html-from-pdf-excel-and-word-
documents.html

HORTONWORKS DATA FLOW
NIFI
1.2.0HDF3.0
Jul 2017
1.0.0HDF2.0
Mar 2016
1.1.0
NiFiRegistry
Ranger
0.7.0
0.5.0
0.6.0
Ambari
2.5.1
2.4.0
2.4.2
Kafka
0.10.1.0
0.9.0
0.10.0
Zookeeper
3.4.6
3.4.6
3.4.6
Storm
1.1.0
1.0.1
1.0.2SAM
0.5.0
SchemaRegistry
0.3.0
HDF2.1
Aug2016
Ongoing Innovation in Apache
HDF1.0
Dec2014 0.3.0
0.6.1HDF1.2
Oct 2015
MiNiFiC++andJava
0.2.0
Ongoing Innovation in OpenSource
1.0.0
0.0.1
0.10.0
HDF 3.1.2
June 2018 1.5.0 0.1.0 0.7.02.6.11.0.0 3.4.61.1.10.6.0 0.5.10.4.0
SECURITYSTREAM ING & INTEGRATION OPERATIONS
Hortonworks Data Flow 3.1.2
https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_release-notes/content/ch_hdf_relnotes.html

HDF Data-In-Motion Platform – with HDF 3.1

Why Apache NiFi?
• Guaranteed delivery
• Data buffering
- Backpressure
- Pressure release
• Prioritized queuing
• Flow specific QoS
- Latency vs. throughput
- Loss tolerance
• Data provenance
• Supports push and pull
models
• Hundreds of processors
• Visual command and
control
• Over a fifty sources
• Flow templates
• Pluggable/multi-role
security
• Designed for extension
• Clustering
• Version Control

Apache MiNiFi
• NiFi lives in the data center. Give it an
enterprise server or a cluster of them.
• MiNiFi lives as close to where data is born
and is a guest on that device or system
“Let me get the key parts of NiFi close to where data begins and provide
bidirectional data transfer"

Edge Intelligence with Apache MiNiFi
Ã Guaranteed delivery
Ã Data buffering
‒ Backpressure
‒ Pressure release
Ã Prioritized queuing
Ã Flow specific QoS
‒ Latency vs. throughput
‒ Loss tolerance
Ã Data provenance
Ã Recovery / recording a rolling log
of fine-grained history
Ã Designed for extension
Different from Apache NiFi
Ã Design and Deploy
Ã Warm re-deploys
Key Features

Custom Apache NiFi Processors for Open Source Computer Vision

TensorFlow with MiniFi
https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html
https://community.hortonworks.com/articles/183151/enterprise-iiot-edge-processing-with-apache-nifi-m.html
https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html

Image Analytics
https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html
https://community.hortonworks.com/articles/155604/iot-ingesting-camera-data-from-nanopi-duo-devices.html
https://community.hortonworks.com/articles/182984/vision-thing-part-2-processing-capturing-and-displ.html
https://community.hortonworks.com/articles/182850/vision-thing.html
https://community.hortonworks.com/articles/77988/ingest-remote-camera-images-from-raspberry-pi-via.html

NiFi and Kafka Are Complementary
NiFi
Provide dataflow solution
• Centralized management, from edge to core
• Great traceability, event level data provenance
starting when data is born
• Interactive command and control – real time
operational visibility
• Dataflow management, including prioritization,
back pressure, and edge intelligence
• Visual representation of global dataflow
Kafka
Provide durable stream store
• Low latency
• Distributed data durability
• Decentralized management of producers &
consumers
+

Integrated Provisioning and Security
Kafka 1.0 Support
To enhance data governance and lineage, users can
now manage access control policies using resource or
tag-based security in Ranger for Kafka 1.0 clusters.
Users can now install, configure, manage, upgrade,
monitor, and secure Kafka 1.0 clusters with Ambari.
New processors in NiFi and Streaming Analytics
Manager support Kafka 1.0 features including message
headers and transactions.

What is Apache Kafka?
• Distributed streaming platformthat allows
publishing and subscribing to streams of
records
• Streams of records are organized into
categories called topics
• Topics can be partitioned and/or replicated
• Records consist of a key, value, and
timestamp
http://kafka.apache.org/intro
Kafka
Cluster
producer
producer
producer
consumer
consumer
consumer
APACHE KAFKA

https://community.hortonworks.com/articles/177349/big-data-devops-apache-nifi-
hwx-schema-registry-sc.html

Completion of Schema Lifecycle: Merged Schema from Dev Branch to
Master

Schema Registry Support for Different “States”: Enable, Disable,
Archive

SAM and Schema Registry Integration
• Streaming Apps Require a Schema
• Unlike NiFi, SAM requires a schema to build streaming
analytics applications.
• Every SAM builder component requires a schema to
function.
• SAM’s primary mechanism for connecting to a stream of
data is Kafka, but Kafka does not have a schema.
• This is where HDF’s Schema Registry component becomes
incredibly valuable.
• SAM’s Kafka Source Component integrated with
Schema Registry
• When you configure a Kafka source and supply kafka topic,
SAM calls the Schema Registry.
• Using the Kafka topic as the key, SAM will retrieve the
schema.
• This schema is then displayed on the tile component, and
is passed to downstream components.

Streaming Analytics Manager

Thank you

Contact
https://community.hortonworks.com/users/9304/tspann.html
https://dzone.com/users/297029/bunkertor.html
https://www.meetup.com/futureofdata-princeton/
https://twitter.com/PaaSDev
https://dzone.com/articles/integrating-keras-tensorflow-yolov3-into-
apache-ni

Hortonworks Community Connection
Read access for everyone, join to participate and be recognized
• Full Q&A Platform (like StackOverflow)
• Knowledge Base Articles
• Code Samples and Repositories

Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python

Similar to Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python (20)

More from Timothy Spann

More from Timothy Spann (20)

Recently uploaded

Recently uploaded (20)

Open Computer Vision with OpenCV, Apache NiFi, TensorFlow, Python