Today’s exponential growth of data along with the amount of hardware required for storing and processing it creates many non-trivial challenges in different areas. At Dell-EMC, we recognized the importance of using predictive analytics for monitoring our enormous install base and increase, through it, the customer experience, using a proactive support approach. In this talk, Ran Taig from the Data Science as a Service team at Dell IT reviews some of the predictive capabilities developed by the team in the HW monitoring domain. Specifically, Ran walks the audience through the use-case of hard-drive failure prediction. Predicting a drive failure well in advance enables the optimization of business processes involved with at-field drive replacements, increasing significantly the reliability of the storage system in addition to a significant cost saving on logistics related to drive replacements. The talk also touches on some lessons learned and challenges faced in the way to deploying such a model as well as other innovative, data science, solutions in an enterprise environment.

1. Ran Taig, PhD.
Data Science as a Service Team
Dell IT
Hardware Failure Prediction at
Dell-EMC

2. In This Talk:
 Dell and Our Team – Overview
 HW Failure Prediction
 Sample Use-Case: HDD Failure Prediction
 Internal Project
 Research Project
 Key Takeaways

3. 3
The world’s largest
privately controlled
technology company
in numbers:
$74B revenue
Serving 98% Fortune 500
~140,000 team members
30,000 full time customer services &
support team members
180 countries
Huge (Enterprise) Install Base

4. 4 of 21
© Copyright 2016 Dell Inc.
The Data Science as a Service Team
• Established more than five years ago.
• A group of machine learning practitioners,
statisticians and mathematicians.
• An essential part of Dell IT service teams.
• A unique, flexible, work model
• Provide a wide range of data science services
to Dell-EMC business units:
• Offering internal successful solutions to
external customers.

5. Hardware Failure Prediction

6. 6 of 21
© Copyright 2016 Dell Inc.
Motivation and Vision
Towards
Proactive
IT Support
Prevent
Over-
Provisioning
Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Lorem ipsum
Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Avoid Data
LossData
Unavailability

7. 7 of 21
© Copyright 2016 Dell Inc.
Many Good Questions
Which drives at Customer-A site will wear out in next 6 months?
Will this HDDSSDServer is expected to fail in the next 14 days?
How probable is a data loss event to occur on this array?
What’s the risk of using 0.5 WPD drives?

8. Sample Use-Case:
HDD Failure Prediction

9. 9 of 21
© Copyright 2016 Dell Inc.
Starting Point
• Focused mainly in a drive population (age, vendor) rather then a specific drive
• (AFR, MTTF).
• Some academic work that was not applicable to our type of data and desired scale.
• Develop a classification model that can provide an alert for a specific drive failure
• Improved Logistics: Reduce the number of support engineers visits to customers sites.
• Improved customer experience.
• Prevent data loss and data unavailability events.
• Innovative Support Strategy.
Past Work
Target
Why?

10. 10 of 21
© Copyright 2016 Dell Inc.
Main Challenges:
The Data – In Reality
• S.M.A.R.T attributes – Tends to vary between different vendors.
• Each business unit and at the times, each analytics group collected and
managed the data independently:
• Varied features.
• Varied frequencies.
• Many Interpretability Issues.
• Different data resources platforms parsers formats SMEs …..
• Sparse tables.
• Missing data.
• Corrupted logs.
• Unstable frequencies (Communication Parsing Ingestion Issues.
And as Always:

11. 11 of 21
© Copyright 2016 Dell Inc.
Data Consolidation
The Business Data Lake
BU1………. BU4 TCE GDE
GS: install
base
Business Data
Lake

12. 12 of 21
© Copyright 2016 Dell Inc.
Model Overview
• Intensive feature
engineering.
• A random forest
classifier.
• Optimizing
prediction period
in sync with the
BUISNESS.

13. 13 of 21
© Copyright 2016 Dell Inc.
Model Additional Output
Replace
Test
Drive failure probability
• Help decision makers to see the BIG picture.
• What’s the $ value of each 0.1 change in the threshold?

14. 14 of 21
© Copyright 2016 Dell Inc.
Evaluation
Results were disappointing.
This was no surprise as
The data was bad
Well, as always, blame the data not
the data scientist….

15. What should we do?
Some Research!

16. 16 of 21
© Copyright 2016 Dell Inc.
PoC on Open Data
Strategy: Prove the importance of quality HDD data collection through research
• Records from 49,056 hard drives.
• Ranging over 21 principal manufacturers
• Daily collection over 2015 from each operational drive
• Basic drive information along with S.M.A.R.T statistics
• “Failed” label for the last operational day of the drive
• Structured and (almost) completely clean
An online backup company.
Released a dataset on its entire HDD install base as open-source.

17. 17 of 21
© Copyright 2016 Dell Inc.
Better Data to Better Model
Incorporating Time-Series Based
Features
Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Lorem ipsum
• For each counter attribute we computed a set
of time series features for the last 7,14 and 21
days
• We first fitted a linear function to the time
series for the specific feature and then
extracted its properties.
• In addition, for non-accumulative features we
extracted relevant features (Mean, Median,
Variability)
XX,XXX+Lorem ipsum
SMART 7
Variability
Slope
Max-diff
Intercept

18. 18 of 21
© Copyright 2016 Dell Inc.
Modelling
The Fancy Parts
Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Lorem ipsum
• To provide a model that will best simulate a
production environment we divided into traintest
by TIME rather than by DRIVE:
All features are computed w.r.t. (some)
current date.
• For train labeling and feature extraction:
We fully ignored any sample that was
dated to the TEST PERIOD.
XX,XXX+Lorem ipsum
Multiple time-series training data
Multiple
time-
series
test data
Time
Feature Engineering
Model Training
Model
Evaluation

19. 19 of 21
© Copyright 2016 Dell Inc.
Results
For 14 days horizon :
False Positive Rate = 0.2%
Recall = 26%.
This was improved to a 50% recall with 0.5% FPR
Details: Our SYSTOR’2017 Poster

20. 20 of 21
© Copyright 2016 Dell Inc.
Implementation
• ~9M samples stored in the final GBDP dataset.
• GPDB supports native SQL that are executed in an efficient distributed manner.
• We used PLR which allows the formulation of R functions and run them on a (PostGre)SQL DB.
• Using PLR we efficiently computed some 20 different time-based features using SQL “window”
functions.
• Then, the ~50K dataset was moved to a local server for a standard Scikit-learn modelling work

21. 21 of 21
© Copyright 2016 Dell Inc.
Additional Use Cases
• Endurance modelling and forecast
• Over-Drive prediction
• Flash failures
• ……..

22. Key Takeaways

23. 23 of 21
© Copyright 2016 Dell Inc.
(Hard) Lessons Learned
• Data science team as educators for the value of data.
• Data science is a MUST-HAVE not a NICE-TO-HAVE
• POC over open (quality) data to showcase potential.
• Always provide a (rough) estimation of your model’s
$-value (revenue generator or cost-saving).

24. 24 of 21
© Copyright 2016 Dell Inc.
(Hard) Lessons Learned
• IT is closest to the data and should be considered as
a key generator for the enterprise data culture
transformation.
• Be aware that you may be captured as a threat for
some traditional corporate analytics rolls.
• REAL change must come TOP-DOWN.
• Don’t waste a good mistake, learn from it!

25. Thank you
ran.taig@dell.com