Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Building a Smarter
Home with Apache
NiFi and Spark
Joseph Niemiec, Systems Architect
Christopher Gambino, Solutions Engine...
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Our Background
 Systems Architect
– Hadooping for 4+ Years
– War-Dr...
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
 Use Case Explanation
 Hardware & Software Summary
 Archit...
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Use Case Explanation
Hortonworks Data Flow (NiFi) enables easy acces...
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Software & Hardware Used
 Brand Name Smarthome Sensors + Hub
– Door...
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Architecture High Level
Hub & Spoke Approach
– Hub
• Hadoop Cluster ...
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Architecture Diagram
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Bidirectional Communication
 Edge to Cloud
– Collection (Push) Site...
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Home Collector High Level
Collectio
n
Monitoring
Redact + Encrypt +
...
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
1st Gen - Blocking, Sequential Writes
Decrypt->Local Disk -> S3 -> ...
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
1st Generation Nifi - Separate Channels
Parallel Decryption = Addit...
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
1st Gen - Separate Channels
Parallel Writes = Additional Management...
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
2nd Gen - Asynchronous Writes – Still Parallel
Local Disk, S3, and ...
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
3rd Generation Nifi - Common Architecture
Add Tag
Decrypt
Add Tag
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
3rd Generation Nifi - Merge on Variable
Extremely Low Overhead
Merg...
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
3rd Generation Nifi
Routing on Attributes
Split Flow
To Disk
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What We Discovered
 Joe’s wife absolutely hates being spied on
 P...
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Challenges // Other Lessons Learned
 Oracle vs. Open JDK for ARM
...
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Q&A
Come hang out with us @ The Dev Café
Upcoming SlideShare
Loading in …5
×

Building a Smarter Home with Apache NiFi and Spark

2,316 views

Published on

Building a Smarter Home with Apache NiFi and Spark

Published in: Technology
  • Be the first to comment

Building a Smarter Home with Apache NiFi and Spark

  1. 1. Building a Smarter Home with Apache NiFi and Spark Joseph Niemiec, Systems Architect Christopher Gambino, Solutions Engineer
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Our Background  Systems Architect – Hadooping for 4+ Years – War-Driving - Mad max style – Telematics Patent in Parking Floor Detection – Co-Author Apache Hadoop YARN Book  Smart Production Lines – Medical Device Quality Control – FDA Compliant Reporting – Custom IOT devices and framework
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda  Use Case Explanation  Hardware & Software Summary  Architecture  NiFi Development Patterns – Multiple Generations  Discoveries  Challenges //Other Lessons Learned
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Use Case Explanation Hortonworks Data Flow (NiFi) enables easy access and filtering to data that resides on the edge. This data can be transmitted site to site and accessed via data exploration frameworks such as Spark.  Joe was Paranoid Living In Detroit  Wanted to investigate the unseen daily activities  Understand problems with edge collection and consumption The Solution? More Technology
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Software & Hardware Used  Brand Name Smarthome Sensors + Hub – Door, Motion, Window  NiFi on a Raspberry Pi 2 Model B – Raspbian OS – Python Push based collector  NiFi on EC2  EC2 HDP Cluster  BareMetal HDP + HDF  Pig & Spark
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architecture High Level Hub & Spoke Approach – Hub • Hadoop Cluster in the DC • Stores All Historical Data • Aimed at making it usable for analysis's • Consumes from Spoke Brokers to land data in DC – Spoke • Per Region/Geo • Real Time Stream Processing for Customer Apps • Cleanse data to meet legal requirements for off-geo Hub
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architecture Diagram
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Bidirectional Communication  Edge to Cloud – Collection (Push) Site-2-Site – Updates (Pull) HTTP Calls  Cloud – Distributed Cache of Rules for Edges – HTTP Call Edge Updates – Site-2-Site or HTTP Call Rule Updates  Data Center to Cloud – Collection with Site-2-Site pull from the Cloud to HDFS – HTTP Call Rule Updates • Easily as GET or POST Cache Updates • Bulk Site-2-Site Push
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Home Collector High Level Collectio n Monitoring Redact + Encrypt + Transfer
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 1st Gen - Blocking, Sequential Writes Decrypt->Local Disk -> S3 -> Database Decrypt To Disk To S3To RDS
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 1st Generation Nifi - Separate Channels Parallel Decryption = Additional Management Decrypt Decrypt
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 1st Gen - Separate Channels Parallel Writes = Additional Management To DiskTo Disk Merge
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 2nd Gen - Asynchronous Writes – Still Parallel Local Disk, S3, and RDS all isolated Decrypt To Disk To S3 To RDS
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 3rd Generation Nifi - Common Architecture Add Tag Decrypt Add Tag
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 3rd Generation Nifi - Merge on Variable Extremely Low Overhead Merge To Disk
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved 3rd Generation Nifi Routing on Attributes Split Flow To Disk
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What We Discovered  Joe’s wife absolutely hates being spied on  Pre-empt Arrival with MAC Addresses  Validate Occupancy based on connected MAC Addresses  Hackers Attacking the Network  High Level of Control vs. Hub Software  Motion almost useless as standalone factor.
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Challenges // Other Lessons Learned  Oracle vs. Open JDK for ARM  Improperly Configured Backpressure  FlowFile Names  Site to Site Configuration Issues  Execute Process has ambiguous failure modes  Merging Files Larger  SmartHome Hub interface
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Q&A Come hang out with us @ The Dev Café

×