The document discusses LinkedIn's use of Hadoop and HDFS to store and process over 1 exabyte of data across multiple clusters. Some key points: 1. LinkedIn now stores over 1 exabyte of total data across all of its Hadoop clusters, with its largest cluster being 10,000 nodes storing 500 petabytes of data. 2. The Hadoop clusters use a single NameNode for metadata management with an average latency under 10 milliseconds. High availability features help prevent single points of failure. 3. LinkedIn has optimized performance through techniques like Java tuning and satellite clusters to address issues like small files and logging directories.

1. The exabyte journey &
DataBrew with CICD
Speaker: Shu-Jeng, Hsieh (Scott)
Date: Jun 25th, 2021 (Fri)
Location: the 104 Corporation

2. The Exabyte Journey in
LinkedIn with Hadoop

3. 1 exabyte (EB)
= 1000 PB
= 1M TB
= 1B GB

4. Two Milestones
1. LinkedIn now stores 1 exabyte of total data
across all Hadoop clusters.
2. The largest cluster is a 10,000-node cluster.
a. 500 PB
b. 1 billion objects
c. A single NameNode serving RPCs
d. An average latency under 10 milliseconds

5. Historical
Scale

6. Two Milestones
1. LinkedIn now stores 1 exabyte of total data
across all Hadoop clusters.
2. The largest cluster is a 10,000-node cluster.
a. 500 PB
b. 1 billion objects
c. A single NameNode serving RPCs
d. An average latency under 10 milliseconds

7. High Availability
1. The NameNode used to be a single point of
failure (SPOF).
2. From Hadoop 2, the HA architecture is
introduced.
a. Quorum Journal Manager (QJM)
b. Network File System (NFS)
3. Aside from preventing from the SPOF, HA is
also crucial for rolling upgrades.

9. DN DN DN DN
NN
Active
NN
Standby
JN
JN JN

10. Java Tuning
380GB heap to maintain
1.1Bof namespace objects

11. Java Tuning
1. Java heap generations
a. Young generation
b. Tenured generation
2. Non-fair locking
a. Theoretically, it’s unfavorable for writers.
b. In practice, it substantially improves performance due to the
occupancy of read requests, which is 95%.

12. Java Tuning

13. Satellite cluster
1. The small files problem
2. The logging directory
3. Bootstrapping the satellite cluster
a. 60TB of data using DistCp, 12 hours
b. FailoverFS
4. Very large block reports

14. Consistent reads
from Standby Node
1. Motivation and requirements
a. HDFS-12943
2. Consistency model
3. The stale read problem
4. The consistency principle
5. Journal tailing: Fast path
a. HDFS-13150

15. Scaling beyond the
Hadoop ecosystem
1. Port-based selective wire encryption
a. GDPR, CCPA
b. HADOOP-10335, HDFS-13541
c. 36-46% reduction in read/write latency, 56-85% increase of
read/write throughput.
2. Encryption at rest
a. Dataset-level encryption
b. LiKMS

16. Scaling beyond the
Hadoop ecosystem
3. Wormhole
a. Bunches of pipelines for data transferring
b. From a single HDFS to the universe

17. Traditional Chinese Version
https://fantasticsie.medium.com/%E8%89%BE%E4
%BD%8D%E5%85%83%E4%BF%B1%E6%A8%8
2%E9%83%A8-linkedin-
%E5%9C%A8%E6%93%B4%E5%B1%95-
hadoop-
%E5%88%86%E6%95%A3%E5%BC%8F%E6%A
A%94%E6%A1%88%E7%B3%BB%E7%B5%B1%
E4%B8%8A%E7%9A%84%E6%97%85%E7%A8

18. Glue DataBrew with CICD

19. Glue DataBrew 101
1. A service that empowers those who have
needs on ETL without code skill.
2. 250+ patterns (built-in transformations)
3. Track the transforming history
of data.
4. Integrates with data pipelines

20. A Glance

21. Data Profile

22. Data Lineage
AWS Glue DataBrew SAS EG

23. DataBrew recipes
A series of transformations
on a DataBrew dataset

24. Continuous Integration &
Continuous Delivery
1. The culture enables teams to store and version
code, maintain parity between development and
production environments.
2. CI/CD practices apply beyond software delivery.
3. A demonstration of the practice on data prepartion

25. Architecture

26. A CDK construct for the demonstration with 4
programming languages available worldwide,
demos for the languages included.
cdk-databrew-cicd
https://awscdk.io/packages/cdk-databrew-cicd@0.1.13/#/
Construct for the architecture

27. import * as cdk from '@aws-cdk/core';
import { DataBrewCodePipeline } from 'cdk-databrew-cicd';
class TypescriptStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
const preproductionAccountId = 'PREPRODUCTION_ACCOUNT_ID';
const productionAccountId = 'PRODUCTION_ACCOUNT_ID';
const dataBrewPipeline = new DataBrewCodePipeline(this, 'DataBrewCicdPipeline', {
preproductionIamRoleArn:
`arn:${cdk.Aws.PARTITION}:iam::${preproductionAccountId}:role/preproduction-Databrew-Cicd-Role`,
productionIamRoleArn: `arn:${cdk.Aws.PARTITION}:iam::${productionAccountId}:role/production-
Databrew-Cicd-Role`,
// bucketName: 'OPTIONAL',
// repoName: 'OPTIONAL',
// branchName: 'OPTIONAL',
// pipelineName: 'OPTIONAL'
});
new cdk.CfnOutput(this, 'OPreproductionLambdaArn', { value:
dataBrewPipeline.preproductionFunctionArn });
new cdk.CfnOutput(this, 'OProductionLambdaArn', { value: dataBrewPipeline.productionFunctionArn
});
new cdk.CfnOutput(this, 'OCodeCommitRepoArn', { value: dataBrewPipeline.codeCommitRepoArn });
new cdk.CfnOutput(this, 'OCodePipelineArn', { value: dataBrewPipeline.codePipelineArn });
}
}
const app = new cdk.App();
new TypescriptStack(app, 'TypescriptStack', {stackName: 'DataBrew-CICD'});

28. How it looks like in the
CloudFormation Designer
after deploying
the CDK construct

30. Ask
Me
Anything