The document discusses metadata and the need for a metadata discovery tool. It provides an overview of metadata, describes different types of users and their needs related to finding and understanding data. It also evaluates different architectural approaches for a metadata graph and considerations for security, guidelines, and other challenges in building such a tool.

1. Metadata

5. SELECT
*
FROM
default.my_table
WHERE ds=’2018-01-01’
LIMIT 100;

7. metadata
noun /ˈmedəˌdādə,ˈmedəˌdadə/
:a set of data that describes and gives information about other data.
1. What kind of
information?
2. About what
data?

8. ●
●
●
●
●
●
Terminology borrowed from Ground paper

9. Data stores Dashboard /
Reports
Schema registry
Events /
Schemas
StreamsPeople
Employees

10. Search based Lineage based Network based
Where is the
table/dashboard for X?
What does it contain?
I am changing a data
model, who are the owner
and most common users?
I want to follow a power
user in my team.
Does this analysis already
exist?
This table’s delivery was
delayed today, I want to
notify everyone
downstream.
I want to bookmark tables of
interest and get a feed of
data delay, schema change,
incidents.
Other requirements
● Leverage as much data automatically as possible
● Preferably, open source and healthy community
● API availability
● Easy to set up

12. Criteria / Products Alation Where
Hows
Airbnb
Data
Portal
Cloudera
Navigator
Apache
Atlas
Search based
Lineage based
Network based
Hive/Presto support
Redshift support
Open source (pref.)

13. First person to discover the South Pole -
Norwegian explorer, Roald Amundsen

14. Power user
● All info in their head
● Get interrupted a lot
due to questions
● Lost
● Ask “power users” a
lot of questions
● Dependencies
landing on time
● Communicating with
stakeholders
Noob user Manager

24. Postgres Hive Redshift ... Presto
Github
Source
File
Databuilder Crawler
Neo4j
Elastic
Search
Metadata Service Search Service
Frontend ServiceML
Feature
Service
Security
Service
Other Microservices
Metadata Sources

26. Pull Model Push Model
● Periodically update the index by pulling from
the system (e.g. database) via crawlers.
● Onus of integration lays on data graph
● No interface to prescribe, hard to maintain
crawlers
● The system (e.g. DB) pushes to a message
bus which downstream subscribes to.
● Onus of integration lies on database
● Message format serves as the interface
● Allows for near-real time indexing
Crawler
Database Data graph
Scheduler
Database Message
queue
Data graph
Preferred if
● Near-real time indexing is important
● Clean interface doesn’t exist
● Other tools like Wherehows are moving
towards Push Model
Preferred if
● Waiting for indexing is ok
● Working with “strapped” teams
● There’s already an interface

28. Low relevance High relevance

29. Low popularity High popularity

30. Relevance Popularity
Tables:
● Descriptions
● Table names, column names
● Tags
Dashboards:
● Description
● Chart names
Tables:
● Querying activity
● Different weights for automated vs adhoc
querying
Dashboards:
● Number of views
● Number of edits

32. “This is God’s
work” - George
X, ex-head of
Analytics, Lyft
“I was on call and
I’m confident 50%
of the questions
could have been
answered by a
simple search in
Amundsen” -
Bomee P, DS, Lyft

33. Penetration rate:
DS (aka analyst): 81%
RS (aka DS): 71%
PM: 22%
SWE: 17%
Cust Serv: 7% (12/390)
Sp. Ops: 67% (10/15)
Sp. Op Leads: 53% (9/17)
Economist: 100% (7/7)
Cust. Quality: 78% (7/9)
Growth Mktg: 25% (6/24)

42. Discovery Curation
Guidelines on:
- Where to store
data
- How to name
tables, dashes,
columns, etc.
Challenges
- How much do we push
for guidelines vs. just make
it discoverable?

43. Discovery Security
- Provide data &
metadata only on
a per need basis
Challenges
- Do we hide the existence
of a data set?
- Do expose metadata
regardless of whether the
data scientist has access
to the data?

45. Metadata
Security/ACL
DataDiscovery
&Trust
Downstream
impactanalysis
DataMonitoring
Compliance
(GDPR/CCPA)

50. Icons under Creative Commons License from https://thenounproject.com/