Abstract:- Headaches dealing with big-data storage? A Pure Engineering use case: See how we use Spark and other big data technologies to improve our software engineering development cycle through automated testing and automated failure triaging. Learn how we gain flexibility, agility, and simplicity by leveraging the FlashBlade as a shared storage system across multiple silos of infrastructure instead of distributed DAS deployments.

1. JOSHUA ROBINSON
FLASHBLADE ENGINEERING, PURE STORAGE
From Big Data To Big Intelligence:
Spark Meets Flashblade
A Pure Engineering Use Case

2. © 2017 PURE STORAGE INC.
2
ALL-FLASH STORAGE FOR
DATA-INTENSIVE COMPUTING

3. © 2017 PURE STORAGE INC.
3
FLASHBLADE FOR BIG DATA ANALYTICS
FAST
DATA
BIG
DATA
X
AGILE
DATA
X
=
DATA
ADVANTAGE

4. © 2017 PURE STORAGE INC.
4
TRADITIONAL DATA ANALYTICS
EXAMPLE
SALES FEED/
PIPELINE
CRM
ENGINEERING
TICKETS
EXTRACT
AGGREGATE
ANALYTICS
PARAMETERS
PRODUCT LOGS
RAW LOG STORAGE
GREP, AWK, ETC
STORAGE
COMPUTE

5. © 2017 PURE STORAGE INC.
5
MODERN BIG DATA ANALYTICS
EXAMPLE
SALES FEED/
PIPELINE
CRM
ENGINEERING
TICKETS
PRODUCT LOGS
EXTRACT
AGGREGATE

6. © 2017 PURE STORAGE INC.
6
INFRASTRUCTURE OF BIG
DATA WAREHOUSES
>6PBs ACROSS 100s OF
HETEROGENEOUS DATA SILOS

7. © 2017 PURE STORAGE INC.
7
BIG DATA WAREHOUSES
INFRASTRUCTURE
Compute
Storage
BIG FAST SIMPLE

8. © 2017 PURE STORAGE INC.
8
INTRODUCING FLASHBLADE™
ALL-FLASH FILE AND OBJECT STORAGE
BIG
Up to 8 PBs
FAST
75 GBps / 8M IOPS
SIMPLE
Seamlessly scalable
BLADE
PURITY
FABRIC

9. Automating triage of test
failures in SW development
A Pure Engineering Use Case

10. © 2017 PURE STORAGE INC.
10
THE PROBLEM
Handful
1 Test
coordinator
(Jenkins)
Handful
Handful
100s of tests

11. © 2017 PURE STORAGE INC.
11
THE PROBLEM
1,000
test
failures
20,000+
tests / day
20 Engineers
2x in the next 12 months
1000+
VMs
120+
FBs
20+
Jenkins
400+
clients
100+
Engineers

12. © 2017 PURE STORAGE INC.
12
THE DREAM
1. Automate triaging of failures as much as possible
2. Extract performance metrics from the logs
3. Save our logs for future use
4. Do all of this in a scalable system
5. Real-time results!

13. © 2017 PURE STORAGE INC.
13
OUR DATA ANALYTICS PIPELINE
10 FB
20
clients
100+ tests
12
12
12
12
rsyslog
12
12
12
12
12
12
12
12

14. © 2017 PURE STORAGE INC.
14
OUR DATA ANALYTICS PIPELINE
100 FB
200
clients
1,000+ tests
12
12
12
12
rsyslog
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12

15. © 2017 PURE STORAGE INC.
15
OUR DATA ANALYTICS PIPELINE
100 FB
200
clients
1,000+ tests
12
12
12
12
rsyslog
12
12
12
12
12
12
12
12
12
12
12
12
12
12

16. © 2017 PURE STORAGE INC.
16
OUR DATA ANALYTICS PIPELINE
120+
FB
400+
clients
4,000+ tests
12
12
12
12
rsyslog
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12
12

17. © 2017 PURE STORAGE INC.
17
OUR DATA ANALYTICS PIPELINE
1,000+
VMs
120+
FBs
20+
Jenkins
400+
clients
16
16
16
16
rsyslog
12
12
12
12
12
12
12
12
12
12
6G
40
40
40
40
18T
18T
6T
6G
12
Custom code
ü Duplicate bug
ü Infrastructure failure
ü Performance regression
20,000+
tests

18. © 2017 PURE STORAGE INC.
18

19. © 2017 PURE STORAGE INC.
19
Processed: 18 TB 30 Billion events
per day
Extracted: 6 GB 8 Million events
per day

20. © 2017 PURE STORAGE INC.
20
THE POWER OF DATA ANALYTICS
20,000+
tests
1,000
test
failures
20,000+
tests
~30
distinct
test
failures
Data Analytics Pipeline

21. Shared Storage Benefits

22. © 2017 PURE STORAGE INC.
22
SCALING
STORAGE

23. © 2017 PURE STORAGE INC.
23
FLASHBLADE GUI

24. © 2017 PURE STORAGE INC.
24
SCALING
COMPUTE

25. © 2017 PURE STORAGE INC.
25
OUR DATA ANALYTICS PIPELINE
1,000+
VMs
120+
FBs
20+
Jenkins
400+
clients
16
16
16
16
rsyslog
12
12
12
12
4G
40
40
40
40
18T
18T
6T
4G
12
Custom code
20,000+
tests
12
12
12
12

26. © 2017 PURE STORAGE INC.
26
ANALYTICS PIPELINE
SCALING COMPUTE
1. Download docker image
2. Mount FlashBlade on container
3. Hot-add to Spark cluster
ALL OF THIS CAN BE DONE IN A SINGLE COMMAND
WITHOUT DISRUPTING YOUR SPARK JOBS!

27. © 2017 PURE STORAGE INC.
27
OUR DATA ANALYTICS PIPELINE
1000+
VMs
120+
FBs
20+
Jenkins
400+
clients
16
16
16
16
rsyslog
12
12
12
12
12
12
12
12
12
12
6G
40
40
40
40
18T
18T
6T
20,000+
tests
6G
12
Custom code

28. © 2017 PURE STORAGE INC.
28
INFRASTRUCTURE AGILITY
AD-HOC AND BURSTY ANALYTICS
rsyslog
1000-CORE SPARK CLUSTER
ON FLASHBLADE

29. © 2017 PURE STORAGE INC.
29
INFRASTRUCTURE SIMPLICITY
⎯ Physical Consolidation
⎯ Density: Multiple racks to a single FlashBlade
⎯ Management Consolidation
⎯ Non-disruptive upgrades
⎯ Storage capacity planning, data access, security
⎯ Backups and Restores

30. © 2017 PURE STORAGE INC.
30
FLASHBLADE
File & Object
AND
2.5 PBs (1:1)
N+2 REDUNDANCY
Purity
PLUS
Pure1
17TB
52TB
BLADES
Power
1150Watt/PB
8M IOPs
AND
75 GB/s
PERFORMANCE

31. © 2017 PURE STORAGE INC.
31
RESOURCES
Apache Spark White Papers:
1) Guide to Supporting On-Premise Spark
Deployments with a Cloud-Scale Data Platform
2) Engineering Unplugged: A Discussion with Pure
Storage's Brian Gold on Big Data Analytics for
Apache Spark
Big Data Analytics: purestorage.com/analytics
FlashBlade Product Info: purestorage.com/flashblade