HDF 3.0 IoT Platform for Everyone
•
1 like•914 views
This document discusses Hortonworks DataFlow (HDF) 3.0 for building IoT platforms. It introduces HDF 3.0 and its key components for data ingestion, management, security, and real-time analysis. These include NiFi for data movement, Streaming Analytics Manager (SAM) for building streaming analytics apps visually, and Schema Registry for managing schemas. The document also presents example IoT use cases and demonstrates building a real-time analytics app in SAM to analyze vehicle event data.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to HDF 3.0 IoT Platform for Everyone
Similar to HDF 3.0 IoT Platform for Everyone (20)
More from Yifeng Jiang
More from Yifeng Jiang (18)
Recently uploaded
Recently uploaded (20)
HDF 3.0 IoT Platform for Everyone
- 1. 1 © Hortonworks Inc. 2011 – 2017 All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved IoT Platform for Everyone ~Hortonworks DataFlow (HDF) 3.0~ Yifeng Jiang Lead Solutions Engineer July 1, 2017
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About Me à 蒋 逸峰 (しょう いつほう / Yifeng Jiang) à Lead Solutions Engineer, NAPAC, Hortonworks – Hadooper since 2009 – HBase book author – Software engineer, cloud, PaaS, DevOps à Jogger, hiker à Twitter: @uprush
- 4. 4 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Drive IoT Data into Data Lake Constrained High-Latency Localized Context Hybrid – Cloud/On-Premise Low-Latency Global Context
- 5. 5 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Management Constrained High-Latency Localized Context Hybrid – Cloud/On-Premise Low-Latency Global Context Management Distributed Agent Central Control ß à Multi-Tenant Interactive C&C Push or Pull
- 6. 6 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Security Constrained High-Latency Localized Context Hybrid – Cloud/On-Premise Low-Latency Global Context Security Sign or Encrypt (data and control) ß à TLS, Obfuscation, Dynamic Entitlements, Kerberos, PKI, AD/DS, etc. ‘ACK’
- 7. 7 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Beyond Management: Real-time Analysis Constrained High-Latency Localized Context Hybrid – Cloud/On-Premise Low-Latency Global Context
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DATA AT REST DATA IN MOTION ACTIONABLE INTELLIGENCE Modern IoT Data Applications PERISHABLE INSIGHTS HISTORICAL INSIGHTS INTERNET OF ANYTHING Hortonworks DataFlow Hortonworks Data Platform Hortonworks Delivers Connected Data Platforms
- 11. 1 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Deeper Ecosystem Integration: 200+ Processors Hash Extract Merge Duplicate Scan GeoEnrich Replace ConvertSplit Translate Route Content Route Context Route Text Control Rate Distribute Load Generate Table Fetch Jolt Transform JSON Prioritized Delivery Encrypt Tail Evaluate Execute All Apache project logos are trademarks of the ASF and the respective projects. Fetch HTTP Syslog Email HTML Image HL7 FTP UDP XML SFTP MQTT WebSock et
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Movement Management with Nifi & Minifi § Constrained § High-latency § Localized context § Hybrid – cloud/on-premises § Low-latency § Global context SOURCES REGIONAL INFRASTRUCTURE CORE INFRASTRUCTURE
- 14. 1 4 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Introducing Hortonworks Streaming Analytics Manager (SAM) Streaming Analytics Manager Design, develop, deploy and manage streaming analytics app with a drag-and-drop paradigm “Make it a delightful experience to build streaming analytics apps. Provide the same type of awesome experience for streaming analytics that customers have today when building flow management apps with Nifi”
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved SAM’s 3 Components Builder – Ops - Insight BUILD OPERATEANALYZE
- 17. 1 7 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Ambari – HDF Cluster Manager
- 18. 1 8 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Stream Builder ⬢ Builder components, shown on the canvas palette, are the building blocks used by the app developer to build streaming apps. ⬢ Drag and drop to build a working streaming application without writing a single line of code. ⬢ 4 Types of Components: Sources, Processors, Sinks and Custom
- 19. 1 9 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Schema Registry ⬢ Data Governance – Centralized registry to provide reusable schema – Version management to define relationship between schemas – Validation to enable generic format conversion and generic routing ⬢ Operational Efficiency – Centralized registry to avoid attaching schema to every piece of data – Version management to enable consumers and producers can evolve at different rates – Validation to ensure data quality
- 20. 2 0 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Prescriptive Analytics ⬢ Routing ⬢ Filtering ⬢ UDF-based projections ⬢ Data delivery ⬢ Alerting
- 21. 2 1 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Predictive Analytics ⬢ Model Registry – Sam has repository to store and manage PMML based predictive models – First class features like version, evolution policies, etc, will be added in future release ⬢ PMML Processor – Processor that can use model from the registry and score the models based on the input stream of events coming in
- 22. 2 2 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Descriptive Analytics ⬢ Create time-series and real- time analytics dashboards, charts and graphs ⬢ 30+ visualization charts out of the box with customization capability ⬢ Powered by Druid and the Stream Insight
- 23. IoT Case Studies power by Connected Data Platform
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Build Your First Streaming Analytics App in Under 10 Minutes
- 28. 2 8 © Hortonworks Inc. 2011 – 2017 All Rights Reserved Trucking company w/ large fleet of international trucks A truck generates millions of events for a given route; an event could be: § 'Normal' events: starting / stopping of the vehicle § ‘Violation’ events: speeding, excessive acceleration and breaking, unsafe tail distance § ‘Speed’ Events: The speed of a driver that comes in every minute. Company uses an application that monitors truck locations and violations from the truck/driver in real- time Route? Truck? Driver? Analysts query a broad history to understand if today’s violations are part of a larger problem with specific routes, trucks, or drivers
- 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworksを始める為の５ステップ 1. Hortonworksサンドボックスを試す hortonworks.com/sandbox 2. チュートリアルを試す hortonworks.com/tutorials 3. Future of Data 勉強会に参加する futureofdata.connpass.com 4. Hortonworks Community Connect (HCC)に参加する hortonworks.com/community 5. サポートサービスを検討する http://hortonworks.com/support
- 35. 3 5 © Hortonworks Inc. 2011 – 2017 All Rights Reserved THANK YOU Yifeng Jiang @uprush