This talk was held at the second meeting of the Swiss Big Data User Group on July 16 at ETH Zürich. The topic of this meeting was: "NoSQL Storage: War Stories and Best Practices". http://www.bigdata-usergroup.ch/item/296477

1. Analyzing Logs/
Configs of 200'000
Systems with
Hadoop
christoph.schnidrig@netapp.com
1

2. What is AutoSupport?
¡ AutoSupport is NetApp's 'phone home'
mechanism
¡ Collection of
– Logfiles
– XML files
– Command output capture
– Counter Manager output
2

3. Business Challenges
Gateways ETL Data Warehouse Reporting
• 600K ASUPs • Data needs to • Only 5% of data goes into the • Numerous mining
every week be parsed and data warehouse requests are not satisfied
loaded in 15 • Oracle DBMS struggling to currently
• 40% coming over
the weekend mins scale, maintenance and • Huge untapped potential
backups challenging of valuable information for
• 2TB growth over
• No easy way to access this lead generation,
week
unstructured content supportability, and BI
Finally, the incoming load doubles every 16 months!
4

4. Hadoop Architecture
7

5. Solution Architecture
8

6. Client Apps – how the customer sees it
11

7. Physical Architecture
FAS
2040
FAS2040 A
1
GB
Ethernet
B
12 data nodes : 12 cores , 48 GB RAM each
3 E-series storage arrays (~600TB) Secondary
Job
Tracker Name
Node Name
Node
10
GB/s
Ethernet
2 4 2 4 2 4 2 4 2 4 2 4 2 4 2 4 2 4 2 4 2 4 2 4
Port 1 Port 2 8 8 8 8 Lnk Lnk Port 1 Port 2 8 8 8 8 Lnk Lnk Port 1 Port 2 8 8 8 8 Lnk Lnk
Ch 1 Ch 2 FCHost Ch 3 Ch 4 Drive Ch 1 Ch 2 FCHost Ch 3 Ch 4 Drive Ch 1 Ch 2 FCHost Ch 3 Ch 4 Drive
Expansion Expansion Expansion
ID/Diag ID/Diag ID/Diag
E
2600
Storage
Array

8. Some performance numbers
Metrics Hadoop
Raw ASUP ingest 1000 ASUPs/min
Throughput or 1.5 GB/min
ASUP Configuration data parse & 1000 ASUP/min
Load
Event messages (EMS) Process & < 1 Hour for 2 Billion records
Load ~= > 200 GB/Hour
EMS Ad-hoc analysis 4-6M records/sec ~=
200 MB/sec on compressed
(LZO) data
14
14

9. New possibilities with Hadoop
¡ Correlate disk latency (hot) with
disk type
– 24 billion records
– 4 weeks to run query
– Hadoop implementation 10.5 hours
¡ Bug detection through pattern
matching
– 240 billion records – Too large to
run
– Hadoop implementation 18 hours
15

10. Incoming AutoSupport Volumes
and TB Consumption
Flat-File Storage Requirement
3500
3000
Total Usage (tb)
2500
2000 Projected Total Usage (tb)
1500 Doubles
1000
500
0
Jan-05 Jan-06 Jan-07 Jan-08 Jan-09 Jan-10 Jan-11 Jan-12 Jan-13 Jan-14 Jan-15 Jan-16
¡ At projected current rate of growth,
total storage requirements continue
doubling every 16 months
¡ Cost Model:
> $15M per year Ecosystem costs
16

11. References
¡ NetApp Accelerates AutoSupport Analytics with
NetApp Open Solution for Hadoop
http://media.netapp.com/documents/asup-hadoop.pdf
¡ NetApp Open Solution for Hadoop Solutions Guide
http://media.netapp.com/documents/tr-3969.pdf
¡ ESG: Lab Validation Report
http://media.netapp.com/documents/ar-esg-netapp-
open-solution.pdf

12. 18