Presentation from Hadoop Summit 2016 in Dublin, Ireland. Contains tips on scaling out human and hardware infrastructure.

1. The Past, Present, and Future of
Hadoop @ LinkedIn
Carl Steinbach
Senior Staff Software Engineer
Data Analytics Infrastructure Group
LinkedIn

2. The (Not So) Distant Past

3. PYMK (People You May Know)
First version implemented in 2006
6-8 Million members
Ran on Oracle (foreshadowing!)
Found various overlaps
School, Work… etc
Used common connections
Triangle closing (?)

4. Triangle Closing
?
Mary
Dave
Steve

5. PYMK Problems
By 2008, 40-50 Million members
Still running on Oracle
Failed often
Infrequent data refresh
6 weeks – 6 months!

6. Humble Beginnings Back in ‘08

7. Success! (circa 2009)
Apache Hadoop 0.20
20 node cluster (repurposed hardware)
PYMK in 3 days!

8. The Present

9. Hadoop @ LinkedIn Circa 2016
> 10 Clusters
> 10,000 Nodes
> 1000 Users
Thousands of workflows, datasets, and ad-hoc queries
MR, Pig, Hive, Gobblin, Cubert, Scalding, Tez, Spark,
Presto, …

10. Two Types of Scaling Challenges
Machines
People and Processes

11. Scaling Machines

12. Some Tough Talk About HDFS
Conventional wisdom holds that HDFS
Scales to > 4k nodes without federation*
Scales to > 8k nodes with federation*
What’s been our experience?
Many Apache releases won’t scale past a couple thousand nodes
Vendor distros usually aren’t much better
Why?
Scale testing happens after the release, not before
Most vendors have only a handful of customers with clusters larger than 1k nodes
* Heavily dependent on NN RPC workload, block size, average file size, average container size, etc, etc

13. March 2015 Was Not a Good Month

14. What Happened?
We rapidly added 500 nodes to a 2000 node cluster
(don’t do this!)
NameNode RPC queue length and wait time skyrocketed
Jobs crawled to a halt

15. What Was the Cause?
A subtle performance/scale regression was introduced
upstream
The bug was included in multiple releases
Increased time to allocate a new file
The more nodes you had, the worse it got

16. How We Used to do Scale Testing
1. Deploy the release to a small cluster (num_nodes = 100)
2. See if anything breaks
3. If no, then deploy to next largest cluster and goto step 2
4. If yes, figure out what went wrong and fix it
Problems with this approach
Expensive: developer time + hardware
Risky: Sometimes you can’t roll back!
Doesn’t always work: overlooks non-linear regressions

17. • Scale testing and performance
investigation tool for HDFS
• High fidelity in all the dimensions that
matter
• Focused on the NameNode
• Completely Black-box
• Accurately fakes thousands of DNs on a
small fraction of the hardware
• More details in forthcoming blog post
17
HDFS Dynamometer

18. Scaling People and Processes

19. 19

20. 20
v
Hadoop Performance Tuning

21. Too many dials!
Lots of frameworks: each one is
slightly different.
Performance can change over time.
Tuning requires constant monitoring
and maintenance!
21
Why Are Most User Jobs Poorly Tuned?
* Tuning decision tree from “Hadoop In Practice”

22. 22
Dr Elephant: Running Light Without Overbyte
Automated Performance
Troubleshooting for Hadoop
Workflows
● Detects Common MR and
Spark Pathologies:
○ Mapper Data Skew
○ Reducer Data Skew
○ Mapper Input Size
○ Mapper Speed
○ Reducer Time
○ Shuffle & Sort
○ More!
● Explains Cause of Disease
● Guided Treatment Process

23. Grab the source code
github.com/linkedin/dr-elephant
Read the blog post
engineering.linkedin.com/blog
23
Dr Elephant is Now Open Source

24. Upgrades are Hard
A totally fictional story:
The Hadoop team pushes a new Pig upgrade
The next day thirty flows fail with ClassNotFoundExceptions
Angry users riot
Property damage exceeds $30mm
What happened?
The flows depended on a third-party UDF that depended on a transitive
dependency provided by the old version of Pig, but not the new version of
Pig

25. Bringing Shading Out of the Shadows
What most people think it is
Package artifact and all dependencies in the same JAR + rename some or all of
the package names
What it really is
Static linking for Java
Unfairly maligned by many people
We built an improved Gradle plugin that makes shading
easier for inexperienced users

26. Audit Hadoop flows for
incompatible and unnecessary
dependencies.
Predict failures before they happen
by scanning for dependencies
that won’t be satisfied post-
upgrade.
Proved extremely useful during
Hadoop2 migration
26
Byte-Ray: “X-Ray Goggles for JAR Files”

27. Byte-Ray in Action

28. SoakCycle:
Real World Integration Testing

29. The Future?

30. Dali
2015 was the year of the table
We want to make 2016 the year of the view
Learn more at the Dali talk tomorrow

31. ©2014 LinkedIn Corporation. All Rights Reserved.©2014 LinkedIn Corporation. All Rights Reserved.