Drone Data Flowing Through Apache NiFi

351 views

Published on

Wednesday 14-Dec-2016 Future of Data - Princeton Meetup
@TigerLabs in Princeton, NJ A talk on Apache NiFi for processing Drone Data. Apache NiFi reads the images from a directory or MQTT, extracts metadata including geolocation, runs TensorFlow for image recognition and stores all the metadata in Phoenix as well as raw JSON in HDFS. Images are also stored in HDFS.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
351
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Drone Data Flowing Through Apache NiFi

  1. 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Drone Data Flowing Through Apache NiFi Timothy Spann 2016 Future of Data – Princeton Meetup
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved hdfs dfs -get /drone/meta/Bebop2_20160920085308-0400.json {"date":"2016-09-20T08:53:08","Compression":"JPEG","Exif Version":"2.10","Components Configuration":"YCbCr","file.group":"root","Compression Type":"Baseline","Image Description":"{ "product_id": "090C", "uuid": "CE112B4276E13339AFBAE10E9ED794E3", "run_date": "2016-09-20T083537- 0400", "filename": "Bebop_2_2016-09-20T085308-0400_CE112B.jpg", "media_date": "2016-09-20T085308-0400" }","Number of Components":"3","Component 2":"Cb component: Quantization table 1, Sampling factors 1 horiz/1 vert","Focal Length":"1.83 mm","Component 1":"Y component: Quantization table 0, Sampling factors 2 horiz/2 vert","YCbCr Positioning":"Center of pixel array","tiff:ResolutionUnit":"Inch","uuid":"9576a956-ec32-4314-bc8b- bd5bb43af4f8","Date/Time Original":"2016:09:20 08:53:08","Shutter Speed Value":"1/249 sec","X Resolution":"72 dots per inch","tiff:Make":"PARROT","path":"/","Photometric Interpretation":"YCbCr","Component 3":"Cr component: Quantization table 1, Sampling factors 1 horiz/1 vert","Unique Image ID":"[32 bytes]","F-Number":"F2.3","modified":"2016-09-20T08:53:08","Focal Length 35":"6mm","tiff:BitsPerSample":"8","Exposure Program":"Program action (high-speed program)","GPS Version ID":"2.200","GPS Latitude Ref":"N","meta:creation-date":"2016-09- 20T08:53:08","exif:FNumber":"2.2999999166745724","GPS Altitude Ref":"Sea level","Exposure Time":"8597231/2147483647 sec","GPS Longitude":"-73° 4' 50.41"","Creation-Date":"2016-09-20T08:53:08","ISO Speed Ratings":"426","Make":"PARROT","Orientation":"Top, left side (Horizontal / normal)","Metering Mode":"(Other)","tiff:Orientation":"1","GPS Longitude Ref":"W","tiff:Software":"Dragon 3.9.0","exif:FocalLength":"1.8300001180412324","filename":"Bebop2_20160920085308-0400.jpg","XMP Value Count":"9","geo:long":"- 73.080669","file.owner":"root","Software":"Dragon 3.9.0","Exif Image Height":"1088 pixels","tiff:YResolution":"72.0","Y Resolution":"72 dots per inch","GPS Latitude":"40° 45' 16.51"","dc:description":"{ "product_id": "090C", "uuid": "CE112B4276E13339AFBAE10E9ED794E3", "run_date": "2016-09-20T083537- 0400", "filename": "Bebop_2_2016-09-20T085308-0400_CE112B.jpg", "media_date": "2016-09-20T085308-0400" }","geo:lat":"40.754585","FlashPix Version":"1.00","Data Precision":"8 bits","White Balance":"Flash","tiff:ImageLength":"1088","description":"{ "product_id": "090C", "uuid": "CE112B4276E13339AFBAE10E9ED794E3", "run_date": "2016-09-20T083537-0400", "filename": "Bebop_2_2016-09-20T085308-0400_CE112B.jpg", "media_date": "2016-09-20T085308-0400" }","dcterms:created":"2016-09-20T08:53:08","dcterms:modified":"2016-09-20T08:53:08","Last-Modified":"2016-09- 20T08:53:08","file.permissions":"rwxrwxrwx","exif:ExposureTime":"0.00400339765660623","Last-Save-Date":"2016-09-20T08:53:08","GPS Altitude":"21 metres","absolute.path":"/opt/demo/dronedata/","Color Space":"Undefined","File Size":"853159 bytes","meta:save-date":"2016-09- 20T08:53:08","file.creationTime":"2016-09-20T12:53:10+0000","Date/Time Digitized":"2016:09:20 08:53:08","File Name":"apache-tika- 941370357006559178.tmp","Content-Type":"image/jpeg","Aperture Value":"F2.3","X-Parsed-By":"org.apache.tika.parser.DefaultParser, org.apache.tika.parser.jpeg.JpegParser","File Modified Date":"Tue Sep 20 23:33:24 UTC 2016","tiff:XResolution":"72.0","file.lastModifiedTime":"2016-09- 20T12:53:10+0000","exif:DateTimeOriginal":"2016-09-20T08:53:08","Date/Time":"2016:09:20 08:53:08","Exif Image Width":"1920 pixels","Image Height":"1088 pixels","Image Width":"1920 pixels","Unknown tag (0xc62f)":"[19 bytes]","Resolution Unit":"Inch","tiff:Model":"Bebop 2","exif:IsoSpeedRatings":"426","Max Aperture Value":"F2.3","Exposure Mode":"Auto exposure","Model":"Bebop 2","file.lastAccessTime":"2016-09-20T23:33:24+0000","tiff:ImageWidth":"1920","White Balance Mode":"Auto white balance"}
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved python classify_image.py --image_file /opt/demo/dronedataold/Bebop2_20160920083655-0400.jpg solar dish, solar collector, solar furnace (score = 0.98316) window screen (score = 0.00196) manhole cover (score = 0.00070) radiator (score = 0.00041) doormat, welcome mat (score = 0.00041) bazel-bin/tensorflow/examples/label_image/label_image -- image=/opt/demo/dronedataold/Bebop2_20160920083655-0400.jpg tensorflow/examples/label_image/main.cc:204] solar dish (577): 0.983162I tensorflow/examples/label_image/main.cc:204] window screen (912): 0.00196204I tensorflow/examples/label_image/main.cc:204] manhole cover (763): 0.000704005I tensorflow/examples/label_image/main.cc:204] radiator (571): 0.000408321I tensorflow/examples/label_image/main.cc:204] doormat (972): 0.000406186 Local TensorFlow via Python or C++ Binary
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Google Vision
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Google Vision
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Google VisionGoogle Vision
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved IBM
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Microsoft Computer Vision API
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Microsoft Computer Vision API
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NIFI Timothy Spann
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Installation Download the binary from here: http://hortonworks.com/downloads/#dataflow Or here: https://nifi.apache.org/download.html Or on Mac: brew install nifi https://nifi.apache.org/docs/nifi-docs/html/getting- started.html#starting-nifi bin/nifi.sh start
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved The Future of Data: Actionable Intelligence D A T A I N M O T I O N STORAGE STORAGE GROUP 2GROUP 1 GROUP 4GROUP 3 D A T A A T R E S T INTERNET OF ANYTHING Hortonworks’ unique approach to data-in-motion and data-at-rest powers Actionable Intelligence
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DATA AT REST DATA IN MOTION ACTIONABLE INTELLIGENCE MODERN DATA APPLICATIONS Actionable Intelligence from Connected Data Platforms Capturing perishable insights from data in motion Ensuring rich, historical insights on data at rest Necessary for modern data applications Hortonworks DataFlow Hortonworks Data Platform
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDF Provides “Data Plan of Control” by Managing IoT Dataflows Constrained High-Latency Localized Context Hybrid – Cloud/On-Premise Low-Latency Global Context Data source agnostic collection of data across heterogeneous environments
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Rapid Ecosystem Adoption: 170+ Processors HTTP Syslog Email HTML Image Hash Encrypt Extract TailMerge Evaluate Duplicate Execute Scan GeoEnrich Replace ConvertSplit Translate HL7 FTP UDP XML SFTP Route Content Route Context Route Text Control Rate Distribute Load AMQP
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data-in-motion: Hortonworks DataFlow Powered by Apache NiFi Collect, conduct and curate real-time data End-to-end security with encryption and rules Traceability and real-time provenance Delivers Instant, Perishable Insights Hortonworks DataFlow Data-in- motion
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Accumulate, Analyze, Act on All Data Centralized Architecture for Multi-tenancy Enterprise Operations, Governance and Security Delivers Rich Historical Insights Data-at-rest: Hortonworks Data Platform Powered by Apache Hadoop Hortonworks Data Platform Data-at-rest
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved à https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/ à https://github.com/jfrazee/awesome-nifi à https://dzone.com/articles/getting-started-with-apache-nifi-and-hdf à https://nifi.apache.org/docs.html à https://community.hortonworks.com/articles/4356/getting-started-with-nifi- expression-language-and.html Learning More
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Contact: Timothy Spann @PaaSDeV www.meetup.com/futureofdata-princeton community.hortonworks.com/users/9304/tspann.html
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved 4,000+ Registered Users 10,000+ Answers 15,000+ Technical Assets One Website!

×