Shirshanka Das presents on building LinkedIn DataHub, a metadata platform. He discusses how enterprises need to solve data discovery, governance and data ops challenges. LinkedIn DataHub addresses this by liberating metadata from various systems and ingesting it into a centralized platform using Kafka. The metadata is then stored, indexed and served to enable data discovery, social search, data lineage and other capabilities. He outlines the overall architecture and how the stream-first approach supports use cases like operational metadata and data quality monitoring.

1. Shirshanka Das
Co-founder and CTO
LinkedIn DataHub: Lessons from
building a stream-first metadata
platform
Kafka Summit
May 12 2021 People Code
Data
DATA

2. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
Co-Founder, CTO of Acryl Data
Founder of LinkedIn DataHub, Apache Gobblin
Previously, lead architect for Big Data at LinkedIn
Shirshanka Das
@shirshanka
About: me

3. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
To talk about metadata,
we first need to talk about data
Metadata Day Workshop: Dec 14th, 2020
https://metadataday2020.splashthat.com/

4. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!4
Good News
Data
Ingestion
Data
Prep
Data
Transformation
Data
Serving
Data
Visualization
Data
Quality
Online Data Stores Data Stream Stores Data Lakes Data Warehouse
Workflow
Mgrs
Macro Trends in Data
Enterprises are rushing to be data-driven, and rushing into the cloud
Data Ecosystem has been categorized into segments with multiple
good projects attempting to solve those segments well

5. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
Case Study: LinkedIn’s Data Ecosystem

6. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!6
Pain Points
faced by Humans and Software!
Which is the authoritative customer dataset?
What are all the dashboards that are powered by the customer
dataset?
Is the customer dataset ready to be consumed?
Are we sharing customer data with third parties?
Do sales people have access to customer data on my platform? Did
they ever?

7. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
Which is the authoritative customer dataset?
What are all the dashboards that are powered by the customer dataset?
Is the customer dataset ready to be consumed?
Are we sharing customer data with third parties?
Do sales people have access to customer data on my platform? Did they ever?
What kind of questions are these?
Pop Quiz
Data Discovery
Data Governance
Data Ops
All of the above

8. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!8
3K+ GitHub stars
75+ Contributors
100+ commits / month
slack.datahubproject.io
Open Source at datahubproject.io
LinkedIn DataHub solves these problems
@datahubproject

9. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!9
How do you solve these problems?

10. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!10
What is common across all these?
Data
Ingestion
Data
Prep
Data
Transformation
Data
Serving
Data
Visualization
Data
Quality
Online Data Stores Data Stream Stores Data Lakes Data Warehouse
Workflow
Mgrs
The metadata that is at the heart of each individual system

11. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!11
What is Metadata?
"Thinking" by Elvin (CC BY-NC 2.0)

12. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!12
"Thinking" by Elvin (CC BY-NC 2.0)
What is Metadata?

13. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!13
"Thinking" by Elvin (CC BY-NC 2.0)
Models,
Features, …
What is Metadata?

14. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!14
“If we can liberate and standardize the metadata that is locked
inside these systems, we can answer these questions in a way that
does not depend on the specific technology being used”
Opportunity

15. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!15
How do you build this amazing
metadata platform?

16. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!16
Step 1: Liberate Metadata!
and

17. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!17
Step 1: Liberate Metadata!
Push
and

18. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!18
Step 1: Liberate Metadata!
Push + Pull
and

19. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!19
Step 2: Store it!
Key-Value + Commit Log

20. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!20
Step 3: Index it!
Commit Log -> Search + Graph + {your favorite index}

21. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!21
Step 4: Serve it!
Key-Value, Search, Graph, Stream, …
Stream Queries

22. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!22
The End to End Architecture
for Humans and Software!

23. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!23
Reference Architecture for Data Mesh
Stream connects metadata from different domains into a global graph
Domain 1
Metadata + Data Streams
Global Metadata Graph
Domain 2 Domain 3 Domain 4
Global Data Pane

24. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!24
Data Discovery
built on this platform

25. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!25
Data Discovery
Social search

26. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!26
Data Products on the Data Mesh
Enabling logical domain-based datasets

27. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!27
Data Discovery
Lineage

28. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!28
From data to trusted data
Operational metadata holds the key
Which system did this dataset originate from and how fresh is it?
Which other systems consume this data?
Is this a time-partitioned dataset or a snapshot dataset?
Will this dataset break my pipeline?
Who is on-call for this dataset?
Can I set alerts on this dataset for important changes?

29. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!29
DataHub is perfectly positioned!
From Technical to Operational Metadata
Stream of changes already being pushed
End to end map of all data: can non-intrusively observe data
Troubleshooting can leverage context and social signals
Efficient routing of “pages” to the right on-call
Connecting consumers and producers efficiently based on lineage
information
Pro-active data quality monitoring support will prevent incidents

30. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!30
The Future
Cloudy with a chance of observability!
Dataset Operational Health
at a glance

31. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!31
The Future
Cloudy with a chance of observability!
Deeper Insights
Dataset Level

32. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!32
The Future
Cloudy with a chance of observability!
Deeper Insights
Column Level

33. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!33
The Future
Cloudy with a chance of observability!
Data Quality Integration
Great Expectations

34. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!34
The Future
Cloudy with a chance of observability!
Event Oriented View
What happened lately

35. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!35
The Future
Cloudy with a chance of observability!
Event Log
Across all producers

36. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!36
Data Discovery - Data Observability - Data Ops
Powered by the Metadata Stream

37. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!37
Key Takeaways
LinkedIn DataHub is an open source stream-first metadata platform
Ideally positioned for data mesh implementations
Fast growing project and community
Join the streaming metadata movement!
slack.datahubproject.io

38. D
A
T
A
O
P
S
U
N
L
E
A
S
H
E
D
|
M
A
R
C
H
1
7
,
2
0
2
1
K
A
F
K
A
S
U
M
M
I
T
|
M
A
Y
1
2
,
2
0
2
1
!38
Co-Founder, CTO of Acryl Data
Founder of LinkedIn DataHub, Apache Gobblin
Previously, lead architect for Big Data at LinkedIn
@shirshanka
Thank you!
datahubproject.io
acryl.io
slack.datahubproject.io