n this talk, we talk about the challenges at scale in an organization like Lyft. We delve into data discovery as a challenge towards democratizing data within your organization. And, go in detail about the solution to solve the challenge of data discovery.

1. May 2018
Mark Grover | @mark_grover | Product Management, Lyft
Deepak Tiwari | @_deepaktiwari_ | Product Management, Lyft
go.lyft.com/strata18
Democratizing Data within your Organization

2. Agenda
• Empowering with Data
• Data at Lyft
• Challenges with Data Discovery
• Data Discovery at Lyft
2

3. Data democratization is important...
3
democracy
noun de·moc·ra·cy di-ˈmä-krə-sē
: the absence of hereditary or arbitrary class distinctions or privileges

4. There are several challenges to data democratization...
4
• Data discovery
‒ Lack of understanding of what data exists, where, who owns it, who
uses it, and how to request access.
• Data tools
‒ Creation: Productivity and technical knowledge (e.g. ETL)
‒ Consumption: Tools for exploration and analysis (e.g. Visualization,
attribution, etc.)

5. Data Scientists spend upto 1/3rd time in Data Discovery...
5
• Data discovery
‒ Lack of
understanding of
what data exists,
where, who owns it,
who uses it, and how
to request access.

6. Lyft: Fastest growing ride hailing service in North America
6

7. 7
Lyft Data Team
Lyft Data Team
Core Data Infra Streaming Infra Visualization Experimentation BI and Logging ML Infra

8. Data platform users
8
Data Modelers Data Scientists Research
Scientists
General
Managers
Data Platform
Engineers ExperimentersProduct
Managers

9. 9
Core Infra high level architecture
Custom apps

10. 10
Life of an event
Golden
Path
Client: iOS / Android
Events stored locally
Server
Call Ingest
Pub /
Sub
(Kafka)
Core Infra:
Hadoop: Hive
/ Presto
Visualization
/ Query
Layer
Read
Monitoring / Anodot
Streamcheck
Storage
(S3)
Stream Ingest
ETLs
Event

11. Data Discovery
11

12. • My first project is to analyze and predict Strata Attendance
Hi! I am a n00b Data Scientist!
12

13. • Where is the data?
• What does it mean?
First questions
13

14. Option #1 - github search for “strata attendance”
● But which one do I use?

15. • Doesn’t scale
• Sometimes YOU are the first one!
Option #2 - Ask a power user
15

16. • What does this field mean?
‒ Does attendance data include employees?
‒ Does it include revenue?
• Let me dig in and understand
Understand the context
16

17. Explore
SELECT
*
FROM
default.my_table
WHERE ds=’2018-01-01’
LIMIT 100;

18. Exploring with SELECT * is EVIL
1. Lack of productivity for data scientists
2. Increased load on the databases
18

19. But!
The comment is out of date...
Now what?

20. Github PR hell!

21. Goal: Productive and effective tool for data discovery
Have we met our goal?

22. Goal: Productive and effective tool for data discovery
Have we met our goal?

23. Audience for data
discovery
23

24. Data Discovery - User personas
24
Data Modelers Data Scientists Research
Scientists
General
Managers
Data Platform
Engineers ExperimentersProduct
Managers

25. 3 Data Scientist personas
Power user
● All info in their head
● Get interrupted a lot
due to questions
● Lost
● Ask “power users” a
lot of questions
● Dependencies
landing on time
● Communicating with
stakeholders
Noob user Manager

26. Search based Lineage based Network based
Where is the
table/dashboard for X?
What does it contain?
I am changing a data
model, who are the owner
and most common users?
I want to follow a power
user in my team.
Does this analysis already
exist?
This table’s delivery was
delayed today, I want to
notify everyone
downstream.
I want to bookmark tables of
interest and get a feed of
data delay, schema change,
incidents.
Data Discovery answers 3 kinds of questions

27. Data graph - 3 kinds of nodes
PeopleAnalysisData sets

28. Summary
• Primarily for data scientists
• Index information about data sets, analysis and people
• Answer search based, lineage based and network based questions
28

29. Buy vs. Build vs. Adopt
29

30. 3 kinds of questions
Criteria / Products Alation Where
Hows
Airbnb
Data
Portal
Cloudera
Navigator
Apache
Atlas
Search based
Lineage based
Network based
Hive/Presto support
Redshift support
Open source (pref.)

31. Meet Amundsen
31
First person to discover the South Pole -
Norwegian explorer, Roald Amundsen

32. Amundsen - landing page

33. Amundsen - table detail page

34. Amundsen - column details

35. Amundsen - column details

36. Amundsen Architecture
36

37. Pillar #1
Building a data graph
37

38. 38
Search service Graph service PostgreSQL service
Update description
Update
metadata
Front end service

39. Pillar #2
Push model vs. Pull Model
39

40. Pull model vs. push model
40
Pull Model Push Model
● Periodically update the index by pulling from
the system (e.g. database) via crawlers.
● The system (e.g. database) pushes
metadata to a message bus which
downstream subscribes to.
Crawler
Database Data graph
Scheduler
Database Message
queue
Data graph

41. Pull model vs. push model
41
Pull Model Push Model
● Onus of integration lays on data graph
● No interface to prescribe, hard to maintain
crawlers
● Onus of integration lies on database
● Message format serves as the interface
● Allows for near-real time indexing
Crawler
Database Data graph
Scheduler
Database Message
queue
Data graph

42. Pull model vs. push model
42
Pull Model Push Model
● Onus of integration lays on data graph
● No interface to prescribe, hard to maintain
crawlers
● Onus of integration lies on database
● Message format serves as the interface
● Allows for near-real time indexing
Crawler
Database Data graph Database Message
queue
Data graph
Preferred if
● Near-real time indexing is important
● Clean interface doesn’t exist
● Other tools like Wherehows are moving
towards Push Model
Preferred if
● Waiting for indexing is ok
● Working with “strapped” teams
● There’s already an interface

43. Pillar #3
Relevance vs. Popularity
43

44. Relevance - search for “apple” on Google
44
Low relevance High relevance

45. Popularity - search for “apple” on Google
45
Low popularity High popularity

46. Striking the balance
46
Relevance Popularity
● Descriptions, owners, frequent users ● Querying activity
● Dashboarding
● Different weights for automated vs adhoc
querying

47. Summary
47

48. Summary
• High level architecture & user personas
• Data Discovery making data scientists unproductive
• 3 types of data discovery - search, lineage and network based
• 3 types of data graph nodes - data sets, analysis and users
• 3 pillars of Amundsen architecture
‒ Building a data graph
‒ Push vs. pull model
‒ Relevance vs. popularity
• Lyft’s work in progress for data discovery - Amundsen 48

49. Mark Grover | @mark_grover
Deepak Tiwari | @_deepaktiwari_
go.lyft.com/strata18
Icons under Creative Commons License from https://thenounproject.com/
49

50. Amundsen - table detail page
Relevance Popularity