Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT
•Download as PPTX, PDF•
4 likes•1,089 views
The proliferation of connected devices and sensors is leading the Digital Transformation. By 2020 there will be over 20 billion connected devices. Data from these devices need to be ingested at extreme speeds in order to be analyzed before the data decays. The life cycle of the data is critical in revealing what insight can be revealed and how quickly they can be acted upon. In this session we will look at the past, present and future architecture trends streaming analytics. We will look at how to turn all the data from devices into actionable insights and dive into recommendations for streaming architecture depending on the data streams and time factor of the data. We will also discuss how to manage all the sensor data, understand the life cycle cost of the data, and how to scale capacity and capability easily with a modern infrastructure strategy.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT
Similar to Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT
- 1. Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT)
- 2. © Copyright 2017 Dell Inc.2 Thomas Henson • Dell EMC Hadoop Blackbelt • Rocket City Engineer • Organizer Huntsville Meetup • Pluralsight Data Analytics Author • Pig Latin, HDFS, Splunk on Hadoop • www.thomashenson.com
- 3. Overview • Analytics of Things • Streaming: Past, Present & Future • Recommended Architectures for Streams • Driving the Digital Transformation
- 4. Dell - Internal Use - Confidential4 of Y Tracking Farm to Table Equipment Maintenance Retail Experience Smart Agriculture
- 5. Dell - Internal Use - Confidential5 of Y Efficient Homes Preventive Maintenance Safety & Emergency Sensors Smart Homes
- 6. Gartner, Inc. forecasts that 8.4 billion connected things will be in use worldwide in 2017, up 31 percent from 2016, and will reach 20.4 billion by 2020.
- 7. Dell - Internal Use - Confidential7 of Y 44 Zettabytes by 2020
- 8. Past, Present, Future Architecture Streaming Analytics
- 9. © Copyright 2017 Dell Inc.9 Past Streaming Architecture Ingest No Real Streaming Solution MapReduce
- 10. © Copyright 2017 Dell Inc.10 Present Streaming Architecture Batch MapReduce DR Site Real-Time
- 11. © Copyright 2017 Dell Inc.11 Future Real-Time Solutions Processing Auto Tier Pravega
- 12. © Copyright 2017 Dell Inc.12 Pravega Architecture • Kappa Architecture Unleashed • New Storage Abstraction for Streams • Coordinated Scaling of Processing and Storage • Exactly Once System-Wide Processing
- 14. © Copyright 2017 Dell Inc.14 Timing is Everything…. Recommendation # 1 Know the time to value for the insights of the data • What is the critical time to value of the data? • Are there systems in place to ensure data is analyzed before decay? • In applications and data analytics timing is everything?
- 15. © Copyright 2017 Dell Inc.15 How Much Data is a lot of Data…. Recommendation # 2 Ensure Systems Can Scale Independently • How do you predict data growth and ingest? • Are there architectures in place to scale an extra 10 TBs? 10 PBs? • Does your architecture prevent bottlenecks and duplicated copies?
- 16. © Copyright 2017 Dell Inc.16 Life of Data…. Recommendation # 3 Plan for Lifecycle of Data • How long should data be kept? • What is the compliance or regulation for data retention? • How do you factor in lifecycle cost of data?
- 17. Digital Transformation Data Analytics and IoT
- 18. © Copyright 2017 Dell Inc.18 Nature Not Data Center
- 19. © Copyright 2017 Dell Inc.19 Today’s challenges with streaming analytics Deriving actionable real- time business insights The industry needs a new storage paradigm that solves for extreme stream velocity and unified real-time analytics Complex app ecosystem >300 Siloed real-time and batch Extreme stream and storage scale
- 20. ECS & Isilon Data Tier Nautilus Platform Secure | Integrated | Efficient | Elastic | Scalable Project Nautilus High level architecture for real-time, interactive, and batch analytics Pravega Streaming Storage Flink Stateful Stream Processor Flink API Zeppelin Notebook Experience Notebook UI Mesos + DC/OS Security Serviceability Streaming Storage API Real-Time/Batch Analytics Frameworks and Apps Interactive Exploration
- 21. © Copyright 2017 Dell Inc.21 Data Lake Traditional and Emerging Sources DATA LAKE ISILON >80% Storage Efficiency Primarily File: SMB, NFS, HDFS Scale-Out NAS Leader ECS Scale-Out Cloud Storage Leader Primarily Object: S3, Swift and Atmos 65% Lower TCO Than Public Cloud
- 22. © Copyright 2017 Dell Inc.22 Servers Storage Networking Automation Management Dell EMC Hadoop Data Analytics Solutions Dell EMC Ready Bundle for Cloudera Hadoop 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID Stack No. 1 2 25 26SFP+ 3 5 7 9 11 4 6 8 10 12 13 15 17 19 21 14 16 18 20 22 24 LNK ACT1 2 23 LNK ACT COMBO PORTS 23 24 6 720 1 14 1512 1310 118 9 22 2320 2118 1916 17 6 720 1 14 1512 1310 118 9 22 2320 2118 1916 17 6 720 1 14 1512 1310 118 9 22 2320 2118 1916 17 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID Stack No. 1 2 25 26SFP+ 3 5 7 9 11 4 6 8 10 12 13 15 17 19 21 14 16 18 20 22 24 LNK ACT1 2 23 LNK ACT COMBO PORTS 23 24 KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM Dell EMC Ready Bundle for Cloudera Hadoop with Isilon Shared Storage 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID Stack No. 1 2 25 26SFP+ 3 5 7 9 11 4 6 8 10 12 13 15 17 19 21 14 16 18 20 22 24 LNK ACT1 2 23 LNK ACT COMBO PORTS 23 24 KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM Dell EMC Ready Bundle for Hortonworks Hadoop 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID Stack No. 1 2 25 26SFP+ 3 5 7 9 11 4 6 8 10 12 13 15 17 19 21 14 16 18 20 22 24 LNK ACT1 2 23 LNK ACT COMBO PORTS 23 24 KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM Services and Financing
- 23. © Copyright 2017 Dell Inc.23 Platform manager Analytic Insights Module engineered for speed Coming Soon!
- 24. © Copyright 2017 Dell Inc.24 Analytics Integration Infrastructure SERVICES of organizations still struggle with how to approach Big Data Range of solutions © Copyright 2017 Dell Inc.24
- 25. © Copyright 2017 Dell Inc.25 Business Technology DEPLOYASSESS PROVE Big Data Proof of Value Big Data Proof of Technology Big Data Applied Analytics Implementation Big Data Technology Implementation Big Data Vision Workshop Big Data Technology Advisory Global services Data Analytics Portfolio © Copyright 2017 Dell Inc.
- 26. © Copyright 2017 Dell Inc.26 Customer success stories © Copyright 2017 Dell Inc.
- 27. Find out more at Booth #100
- 28. Closing Thoughts • Covered AoT for Future • Better Understanding of Future Streaming • Discussed Architecture Recommendations for Streams • Learned How to Drive Digital Transformation
Editor's Notes
- Agriculture is ecosystem centric meaning farmers rely on other members to supply pieces for their farm.
- Houston we have a problem. How much data is going into the 20 billion devices? Right now our estimates are at 44 zettabytes of data. Not only is the data going to be tough to take in and scale but how fast is that data going to decay? What is our value to insight of the data? Tell story about Batch vs. Streaming from the perspective of a crowd sourced navigation app.
- Pravega zoom: low latency append only, high throughput, auto-scaling, durable exactly-once semantics, tiering Flink / Spark zoom: Multi-tenancy, stateful stream processor, optimized by pravega Platform zoom: unified infrastructure, security, serviceability
- Transition: Isilon and ECS deliver on the data lake vision of the modern data center Isilon is the Scale-out NAS leader as recognized by leading industry analysts like IDC, Gartner, and ESG with over 7000 customers It’s primary workloads include multi-protocol support for unstructured file data access and analytics And, ECS is the leader is scale-out object based cloud storage that supports the emerging and new applications. it has much lower overhead and better TCO than public cloud options Isilon and ECS can handle all your unstructured data for your workloads of today and tomorrow.
- The Analytic Insights Module is built on converged infrastructure and Pivotal Cloud Foundry which provides the PaaS layer. Data analysis teams can select which Hadoop Distribution and tools they prefer to work with and with skills they can support. The flexibility of using a broad ecosystem supports a wide range of advanced analytics and a business’ digital transformation.
- We’ve shown you how critical the foundations of an analytics ecosystem can be, and we’ve mentioned just a few of our products and solutions and shown you how real world organizations have used them to attain significant results for their unique missions. <click> Did you know thought that 44% of IT organizations are unsure of how to begin their analytics journey – we’ve also made it clear that each organization may have a different starting point - which is why we have a massive portfolio that can address the critical foundations of data analytics.. Establishing an analytics foundation is critical to your future success. Whether your unique needs require assistance with industry leading integration, or world class infrastructure or a combination of these – From Boomi, to PowerEdge, to Isilon, to converged big data systems. DELL EMC has you covered. <click> At DELL EMC we also span beyond the core ecosystem to tailor our offerings to your unique needs, with data security solutions and partnerships with other industry leading providers, like Microsoft, Oracle, SAP, Tableau, Mongo DB, Hortonworks, Cloudera and many others. In addition, DELL EMC can provide any or all of the services you may need to determine use cases and successfully implement the solutions you need to accomplish your goals. Whether you need a complete custom-built solution, a pre-engineered plug-and-play appliance or simply a validated reference architecture to build your own, DELL EMC can help you execute at the speed of your business. <click>
- FOR EVENT SPEAKER DELL EMC Services for Big Data and analytics help both business and IT stakeholders integrate data and analytics into their business strategies. We have two distinct pathways of professional services, the business and the technology requirements for analytics. For business leaders, our consulting services help identify and prioritize optimal analytics use cases, prove their potential ROI, and Implement them into production. For technology leaders, our consulting services help determine and validate the right technology strategy and architectures for Big Data and analytics capabilities, and deploy those technologies and analytics platforms into production. These tracks – business-centric, technology-centric, or a mixture of both - will help you determine the best offering within the DELL EMC Big Data portfolio. They will also assist with software installation and overall implementation. We also have great education offerings like a prerecorded MOOC, our Big Data massive open online course as well as Data Science and Analytics courses focused on applying your knowledge and skills as a member of a data analytics team, managing your Hadoop, HDFS, and MapReduce environments, as well as advancing your knowledge and skills as a data scientist. <click> _________ FOR RECORDING Business Track The first is a business track that assumes we have a relationship with a business stakeholder who is looking to drive business value against a business initiative over the next 9-12 months. We recommend the Big Data Vision Workshop which aligns IT and the business around a strategic initiative, applies data science, and models a solutions to recommend a qualified analytic use case and action plan. The Big Data Proof of Value proves the analytical lift of that use case in an on-site test environment using a sampling of real customer data. The Big Data Applied Analytics Implementation then operationalizes that single analytical use case by embedding that actionable insight into an existing business process or a new application. Technology Track: The second track is focused on technology and capabilities (rather than implementing a specific analytics use case). It assumes we have a relationship within IT that has challenges supporting their Big Data technology initiative(s). The Client needs to develop technical capabilities to deliver data and analytics to its business users. We recommend the Big Data Technology Advisory. which assesses their current environment and then recommends an architecture that supports their technology initiative(s) and a roadmap to implement the new Big Data capabilities. The Big Data Proof of Technology pilots the recommended architecture to prove it works for the identified technical capability. The Big Data Technology Implementation then operationalizes that capability into production. For customers that are ready to deploy a robust converged platform for Big Data and Analytics, we offer services to deliver and install the software components of Big Data solutions so that it is fully operable, and we offer an Implementation service which deploys a proven analytics use case, as a production business solution, on top of the solution. We also have great education offerings like a prerecorded MOOC, our Big Data massive open online course as well as Data Science and Analytics courses focused on applying your knowledge and skills as a member of a data analytics team, managing your Hadoop, HDFS, and MapReduce environments, as well as advancing your knowledge and skills as a data scientist.
- ***EDIT TALK TRACK*** Although we have many customers that we have worked with to achieve amazing business outcomes. Some we have already spoken about and some that we will have public information on if you are interested in more detail, because today we want to talk about the success we have achieved internally. <click> For years, Dell had been using an enterprise data warehouse (EDW) to create reports, but as the EDW grew in size, reports took longer to create. EDW limitations became even more apparent when staff needed to analyze unstructured big data from new sources such as social media to gain a 360-degree view of customers. Dell began by improving operational performance by deploying a Cloudera Hadoop data reservoir to offload extract, transform and load (ETL) data jobs from the existing enterprise data warehouse. This provided exponentially faster reporting and queries, and the same infrastructure allowed the addition of new data sources to generate 360-degree views of customer experience and requirements, including in-house social media analysis. The increased insights as well as the ability to add new data sources for analysis boosts business efficiency and agility, so Dell is better positioned to reach its goals today, and in the future. <click> At EMC we have built our own data lake for data consolidation. The EMC data lake is focused on mulitple analytics use cases. IT to quickly identify application behaviour anomalies, marketing which gives us a better understanding of customer behaviours, needs, and sentiments. Security for threat analytsis using real-time analytics. Services account managment in order to give our customers proactive and predictive view of their environment. And we can not forget our newly announced “MyService360”, a new service-centric online dashboard that gives you deep visibility into the health and wellness of your EMC environment. <