Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Future Architecture of Streaming
Analytics: Capitalizing on the Analytics
of Things (AoT)
© Copyright 2017 Dell Inc.2
Thomas Henson
• Dell EMC Hadoop Blackbelt
• Rocket City Engineer
• Organizer Huntsville Meetup...
Overview
• Analytics of Things
• Streaming: Past, Present & Future
• Recommended Architectures for Streams
• Driving the D...
Dell - Internal Use - Confidential4 of Y
Tracking Farm to Table
Equipment Maintenance
Retail Experience
Smart
Agriculture
Dell - Internal Use - Confidential5 of Y
Efficient Homes
Preventive Maintenance
Safety & Emergency Sensors
Smart
Homes
Gartner, Inc. forecasts that 8.4 billion
connected things will be in use worldwide
in 2017, up 31 percent from 2016, and
w...
Dell - Internal Use - Confidential7 of Y
44 Zettabytes
by 2020
Past, Present, Future
Architecture
Streaming Analytics
© Copyright 2017 Dell Inc.9
Past Streaming Architecture
Ingest
No Real Streaming Solution
MapReduce
© Copyright 2017 Dell Inc.10
Present Streaming Architecture
Batch MapReduce
DR Site
Real-Time
© Copyright 2017 Dell Inc.11
Future Real-Time Solutions
Processing
Auto Tier
Pravega
© Copyright 2017 Dell Inc.12
Pravega Architecture
• Kappa Architecture Unleashed
• New Storage Abstraction for Streams
• C...
Architecture
Recommendations
Streaming Analytics
© Copyright 2017 Dell Inc.14
Timing is Everything….
Recommendation # 1
Know the time to value for the
insights of the data...
© Copyright 2017 Dell Inc.15
How Much Data is a lot of Data….
Recommendation # 2
Ensure Systems Can Scale
Independently
• ...
© Copyright 2017 Dell Inc.16
Life of Data….
Recommendation # 3
Plan for Lifecycle of Data
• How long should data be kept?
...
Digital Transformation
Data Analytics and IoT
© Copyright 2017 Dell Inc.18
Nature
Not Data Center
© Copyright 2017 Dell Inc.19
Today’s challenges with streaming analytics
Deriving
actionable real-
time business
insights
...
ECS & Isilon Data Tier
Nautilus Platform
Secure | Integrated | Efficient | Elastic | Scalable
Project Nautilus
High level ...
© Copyright 2017 Dell Inc.21
Data Lake
Traditional and Emerging Sources
DATA LAKE
ISILON
>80% Storage Efficiency
Primarily...
© Copyright 2017 Dell Inc.22
Servers Storage Networking Automation Management
Dell EMC Hadoop Data Analytics Solutions
Del...
© Copyright 2017 Dell Inc.23
Platform manager
Analytic Insights Module engineered for speed
Coming
Soon!
© Copyright 2017 Dell Inc.24
Analytics
Integration
Infrastructure
SERVICES
of organizations
still struggle with how
to app...
© Copyright 2017 Dell Inc.25
Business
Technology
DEPLOYASSESS PROVE
Big Data
Proof of Value
Big Data
Proof of
Technology
B...
© Copyright 2017 Dell Inc.26
Customer success stories
© Copyright 2017 Dell Inc.
Find out more at
Booth #100
Closing Thoughts
• Covered AoT for Future
• Better Understanding of Future Streaming
• Discussed Architecture Recommendati...
Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT)
Upcoming SlideShare
Loading in …5
×

Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT)

526 views

Published on

The proliferation of connected devices and sensors is leading the Digital Transformation. By 2020 there will be over 20 billion connected devices. Data from these devices need to be ingested at extreme speeds in order to be analyzed before the data decays. The life cycle of the data is critical in revealing what insight can be revealed and how quickly they can be acted upon.

In this session we will look at the past, present and future architecture trends streaming analytics. We will look at how to turn all the data from devices into actionable insights and dive into recommendations for streaming architecture depending on the data streams and time factor of the data. We will also discuss how to manage all the sensor data, understand the life cycle cost of the data, and how to scale capacity and capability easily with a modern infrastructure strategy.

Published in: Technology
  • Be the first to comment

Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT)

  1. 1. Future Architecture of Streaming Analytics: Capitalizing on the Analytics of Things (AoT)
  2. 2. © Copyright 2017 Dell Inc.2 Thomas Henson • Dell EMC Hadoop Blackbelt • Rocket City Engineer • Organizer Huntsville Meetup • Pluralsight Data Analytics Author • Pig Latin, HDFS, Splunk on Hadoop • www.thomashenson.com
  3. 3. Overview • Analytics of Things • Streaming: Past, Present & Future • Recommended Architectures for Streams • Driving the Digital Transformation
  4. 4. Dell - Internal Use - Confidential4 of Y Tracking Farm to Table Equipment Maintenance Retail Experience Smart Agriculture
  5. 5. Dell - Internal Use - Confidential5 of Y Efficient Homes Preventive Maintenance Safety & Emergency Sensors Smart Homes
  6. 6. Gartner, Inc. forecasts that 8.4 billion connected things will be in use worldwide in 2017, up 31 percent from 2016, and will reach 20.4 billion by 2020.
  7. 7. Dell - Internal Use - Confidential7 of Y 44 Zettabytes by 2020
  8. 8. Past, Present, Future Architecture Streaming Analytics
  9. 9. © Copyright 2017 Dell Inc.9 Past Streaming Architecture Ingest No Real Streaming Solution MapReduce
  10. 10. © Copyright 2017 Dell Inc.10 Present Streaming Architecture Batch MapReduce DR Site Real-Time
  11. 11. © Copyright 2017 Dell Inc.11 Future Real-Time Solutions Processing Auto Tier Pravega
  12. 12. © Copyright 2017 Dell Inc.12 Pravega Architecture • Kappa Architecture Unleashed • New Storage Abstraction for Streams • Coordinated Scaling of Processing and Storage • Exactly Once System-Wide Processing
  13. 13. Architecture Recommendations Streaming Analytics
  14. 14. © Copyright 2017 Dell Inc.14 Timing is Everything…. Recommendation # 1 Know the time to value for the insights of the data • What is the critical time to value of the data? • Are there systems in place to ensure data is analyzed before decay? • In applications and data analytics timing is everything?
  15. 15. © Copyright 2017 Dell Inc.15 How Much Data is a lot of Data…. Recommendation # 2 Ensure Systems Can Scale Independently • How do you predict data growth and ingest? • Are there architectures in place to scale an extra 10 TBs? 10 PBs? • Does your architecture prevent bottlenecks and duplicated copies?
  16. 16. © Copyright 2017 Dell Inc.16 Life of Data…. Recommendation # 3 Plan for Lifecycle of Data • How long should data be kept? • What is the compliance or regulation for data retention? • How do you factor in lifecycle cost of data?
  17. 17. Digital Transformation Data Analytics and IoT
  18. 18. © Copyright 2017 Dell Inc.18 Nature Not Data Center
  19. 19. © Copyright 2017 Dell Inc.19 Today’s challenges with streaming analytics Deriving actionable real- time business insights The industry needs a new storage paradigm that solves for extreme stream velocity and unified real-time analytics Complex app ecosystem >300 Siloed real-time and batch Extreme stream and storage scale
  20. 20. ECS & Isilon Data Tier Nautilus Platform Secure | Integrated | Efficient | Elastic | Scalable Project Nautilus High level architecture for real-time, interactive, and batch analytics Pravega Streaming Storage Flink Stateful Stream Processor Flink API Zeppelin Notebook Experience Notebook UI Mesos + DC/OS Security Serviceability Streaming Storage API Real-Time/Batch Analytics Frameworks and Apps Interactive Exploration
  21. 21. © Copyright 2017 Dell Inc.21 Data Lake Traditional and Emerging Sources DATA LAKE ISILON >80% Storage Efficiency Primarily File: SMB, NFS, HDFS Scale-Out NAS Leader ECS Scale-Out Cloud Storage Leader Primarily Object: S3, Swift and Atmos 65% Lower TCO Than Public Cloud
  22. 22. © Copyright 2017 Dell Inc.22 Servers Storage Networking Automation Management Dell EMC Hadoop Data Analytics Solutions Dell EMC Ready Bundle for Cloudera Hadoop 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID Stack No. 1 2 25 26SFP+ 3 5 7 9 11 4 6 8 10 12 13 15 17 19 21 14 16 18 20 22 24 LNK ACT1 2 23 LNK ACT COMBO PORTS 23 24 6 720 1 14 1512 1310 118 9 22 2320 2118 1916 17 6 720 1 14 1512 1310 118 9 22 2320 2118 1916 17 6 720 1 14 1512 1310 118 9 22 2320 2118 1916 17 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID Stack No. 1 2 25 26SFP+ 3 5 7 9 11 4 6 8 10 12 13 15 17 19 21 14 16 18 20 22 24 LNK ACT1 2 23 LNK ACT COMBO PORTS 23 24 KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM Dell EMC Ready Bundle for Cloudera Hadoop with Isilon Shared Storage 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID Stack No. 1 2 25 26SFP+ 3 5 7 9 11 4 6 8 10 12 13 15 17 19 21 14 16 18 20 22 24 LNK ACT1 2 23 LNK ACT COMBO PORTS 23 24 KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM Dell EMC Ready Bundle for Hortonworks Hadoop 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 Stack-ID LNK 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ACT 50 52 5433 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 51 53 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID 120124 112116 104108 96100 8892 8084 7276 6468 5660 4852 4044 3236 2428 1620 812 04 Stack ID Stack No. 1 2 25 26SFP+ 3 5 7 9 11 4 6 8 10 12 13 15 17 19 21 14 16 18 20 22 24 LNK ACT1 2 23 LNK ACT COMBO PORTS 23 24 KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM KVM Services and Financing
  23. 23. © Copyright 2017 Dell Inc.23 Platform manager Analytic Insights Module engineered for speed Coming Soon!
  24. 24. © Copyright 2017 Dell Inc.24 Analytics Integration Infrastructure SERVICES of organizations still struggle with how to approach Big Data Range of solutions © Copyright 2017 Dell Inc.24
  25. 25. © Copyright 2017 Dell Inc.25 Business Technology DEPLOYASSESS PROVE Big Data Proof of Value Big Data Proof of Technology Big Data Applied Analytics Implementation Big Data Technology Implementation Big Data Vision Workshop Big Data Technology Advisory Global services Data Analytics Portfolio © Copyright 2017 Dell Inc.
  26. 26. © Copyright 2017 Dell Inc.26 Customer success stories © Copyright 2017 Dell Inc.
  27. 27. Find out more at Booth #100
  28. 28. Closing Thoughts • Covered AoT for Future • Better Understanding of Future Streaming • Discussed Architecture Recommendations for Streams • Learned How to Drive Digital Transformation

×