Nicholas Berg presented on Seagate's use of big data analytics to manage the large amount of manufacturing data generated from its hard drive production. Seagate collects terabytes of data per day from testing its drives, which it analyzes using Hadoop to improve quality, predict failures, and gain other insights. It faces challenges in integrating this emerging platform due to the rapid evolution of Hadoop and lack of tools to fully leverage large datasets. Seagate is developing its data lake and data science capabilities on Hadoop to better optimize manufacturing and drive design.

1. Nicholas Berg, Director Enterprise Analytics
Seagate Technology
Sensor Overload!Taming The Raging Manufacturing Big Data Torrent
(DMT-1633)
IBM World of Watson 2016 Conference

2. You May Know Seagate as a
Hard Drive
Manufacturer…
§ $11B Annual Revenue
§ 45,000 employees, 26 countries
§ 1st and only to ship over 2 billion drives
§ Stores more than 40% of the world’s data
§ Technology leader with 9000+ patents

3. The Intelligent Information Infrastructure
Seagate is Evolving to Address Trends
SYSTEMS
DEVICES
ARRAYS
Converged Infrastructure
Hybrid Data
Systems
Software-defined
Storage
Flash Arrays
Cloud Backup
& Disaster Recovery
NAND PCIe
Traditional Enterprise
NAND SASHyperscale
Desktop
Notebook / Tablet
Branded NAND SATAHybrid
Kinetic
Still Growing
Expanded Offerings
Started with Our
Core Products

4. Seagate’s Manufacturing & Design
Fremont, CA
Longmont, CO
Shakopee, MN
Bloomington, MN
Springtown, N. Ireland
Gwanggyo, South Korea
Wuxi, China
Suzhou, China
Teparuk & Korat, ThailandJohor, Penang &
Seremban Malaysia
Shugart, Singapore
Woodlands, Singapore
Drive & Component Manufacturing
Design Centers
Havant, UK
• Seagate owns and operates 11 factories in 8
countries on 3 continents
• More than three million square feet of
manufacturing space
• Factories are vertically integrated from silicon
fabrication to drive assembly
• Integrate over 45 billion parts per year

5. Hard Disk Drive Components
• Seagate ships
~200M/year, or,
~500,000 drives/day
• An assembled drive
generates about
20MB of test data
• Each drive has
150-300 main
components
depending on drive
complexity
• Unqualified estimate: 100MB/drive machine data generated
• ~50TB per day
• ~20PB of data a year

6. Sensor Overload!
• What to do with 50TB of data
per day, or 20PB a year?
• Streaming and real time
analytics for factory controls
• Capture and perform deeper
analytics for product design,
quality, yields and many
many other uses

7. All Elements Working Together As One System
Solution: Big Data Analytics
End-to-End Coherent, Scalable Data Collection and Retention
Big Data Analytics Infrastructure (H/W + S/W) and Algorithms
Drive Quality
Engineering
and Assurance
Data
Drive
Assembly and
Manufacturing
Test Data
Incoming
Components
Data
Ongoing Quality
and Reliability
Test Data
Returned
Drives Test
and
Diagnostics
Data
Customer
Integration and
Field Data
(including Field
Telemetry)
Predictive Life
Models
Test auto-
Diagnostics
and Alerts
Predictive
Financial
Models
Robust
Excursion
Detection Algos
Ad-hoc Big
Data Analytics
Projects
In-situ Failure
Prediction
Big Data-Driven Quality Decision Layer

8. 4.4 ZB in 2013
85% from New Data Types
15x Machine Data by 2020
44 ZB by 2020
ZB = 1B TB
Traditional Data Architecture
Pressured
Sources: Reinsel, David. “Where in the World Is Storage: A Look at Byte Density Across the Globe” IDC October 2013, IDC/EMC Digital Universe, April 2014

9. Hadoop
Ecosystem
Architecture

10. • A Logical Data Warehouse combines traditional data warehouses with big data
systems to evolve your analytics capabilities beyond where you are today
• Hadoop does not replace your EDW. EDW is a good “general purpose” data
management solution for integrating and conforming enterprise data to produce your
everyday business analytics
• A typical EDW may have 100’s of data feeds, dozens of integrated applications and
run 1000’s to 100,000’s of queries a day
• Hadoop is more specialized and much less mature. For now it will have only a few
application integration points and run fewer queries at a lower concurrency,
answering different questions
• A Hadoop cluster of 60-100 nodes is a supercomputer. What would you use a
supercomputer for? Probably to answer the really big questions
Evolving to a Logical Data Warehouse

11. The Data Lake: Data Tiering

12. Hadoop cluster data loading and querying
EDW
Factory Data
Systems
UNIX
HDFS
SQOOP
Big SQL
Hive
SparkR Ambari
Compact
& Load
Tier 3 (Derived Data Tables)
Tier 2 (Hive Structured Data Tables)
Tier 1 (Raw Data Files)
READ
JDBC | ODBC| Other Drivers
WRITE
Data Science
ApplicaGons
(SAS, R, Python, ML) 10% Drive
Sampled
T1/T2
ApplicaFon
Spark
Yarn
Ganglia | Nagios
R Shiny
H2O
Jupyter
R Studio MapReduce
SparkSQL
HCatalog
Tez
T1/T2
ApplicaFon
Component
Component
Drive Data
100%

13. Tier 1 / Tier 2 custom data loading application
Data Transport
• Scoop: Pull EDW data to HDFS Tier 1
• Non-EDW files (Factory push):
• Trickle feed files to staging area
• Unzip, Merge, reZip small files to large files
• Push compacted files to HDFS Tier 1
Data Mapping & Loading
• Match source/target columns
• Detect and handle column changes
• Transform data
• Insert or Update data in Tier 2
• Dual feed to cluster 2 Tier 1 Tier 2
Scheduling
• Oozie backend
• Configurable frequency
• Currently Daily
• Snapshots (waits for data loads to complete)
• Meta data backups
Compaction
• Major and Minor compaction
• Minor: merges small files to large ones
• Major: remove old versions of data (updates)
• Consolidates HDFS directories
T1/T2 App

14. Enterprise Hadoop Architecture
CPUs 12
HDDs 12x3TB
RAM 144GB
Network Dual 10GBit
D
N
S
P
E
C

15. • Incremental phased delivery, or use case by use case
• Form a “data lake” or “data reservoir” for all enterprise data
• Data availability must come first, model and transform the data in place within
Hadoop
• resist moving the data again
• Lots of talk about schema on read but for DW types of uses, this is impractical
• Data modeling is still required but can be simplified
• Have multiple clusters: Development, Test and then two or more Production, one for
Ad Hoc data exploration & experimentation, one for more governed uses
• Use existing custom query/analytics solution to provide “transparent” access to
Hadoop
Some early Hadoop practices and learnings

16. • Use partitioned/tiered data sets: raw, modeled/standardized, analytics, history/archive
• Tier 0: extended history/archives (if needed)
• Tier 1: low latency raw data for power users to access using low level tooling (MR,
Python)
• Tier 2: de-duped, modeled and transformed data used by the majority of Hadoop
users
• Tier 3: specialized analytic data sets for specific needs (e.g. data pivots,
aggregations)
• Copy summarized data, derived analytics to EDW for broader use/analysis with BI
tools
• Do lots of performance testing, run benchmarks, continually optimize
Some early Hadoop practices and learnings

17. Data Science
Theme Example
Image Analytics &
Pattern Recognition
Media flaw pattern recognition
Machine Learning Reduce drive testing through
failure prediction
Anomaly Detection Multivariate SPC
Predictive Analytics Predict process interactions that
are critical to quality and yields
Prognostic Health Field telemetry analytics
Data Science
Research
Automated feature extraction
Deep learning & image analytics
Imbalanced data
Data science tooling

18. Data Science
RED: PREDICTED
BLUE: ACTUAL
Data Science
In Action

19. • Knowing which Hadoop projects to “bet on”, which data formats and compression
types to use
• Speed of change: probably has more code been written than any other IT platform
• Need to upgrade cluster software frequently (once a quarter)
• Gaps: Some things not ready like ACID, real-time queries
• Resource management for different types of workloads
• Lack of BI tools that can really take advantage of huge data sets and visualize them
• Still very batch processing orientated but interactive is gaining traction with Spark etc.
• Provisioning large numbers of machines, hardware failures
• Integrating remote clusters, cross cluster data movement and inter-cluster processing
Hadoop challenges – an emerging and evolving platform

20. Big Data Analytics Platform Evolution
Data VirtualizaGon
Storage
Data
Sources
VisualizaGon, BI,
ReporGng
Data PreparaGon
GUI AnalyGcs
Data Science
Programming Tools
Data
Engineering
Cloud
Data
Storage
Seagate Data Centers Cloud Data Centers
Data PreparaGon
GUI AnalyGcs
VisualizaGon, BI,
ReporGng
CiGzen Data
ScienGst & SME
Data Science
Programming Tools
Data
ScienGst
Business Intelligence,
Report Designer
Compute
Compute
(elasGc)
Cloud
automaGon
workflows

21. Nicholas Berg nicholas.e.berg@seagate.com