Presentation on dogfooding data at Lyft by Mark Grover and Arup Malakar on Oct 25, 2017 at Big Analytics Meetup (https://www.meetup.com/SF-Big-Analytics/events/243896328/)

1. Mark Grover | @mark_grover
Arup Malakar | @amalakar
1

2. Agenda
• Goals of the Lyft data platform
• High level architecture
• Story about dogfooding
2

3. Goals of Lyft’s data platform
3
Data Modelers Data Analysts Data Scientists General Managers
Data Platform
Engineers

4. Quick demo
4

5. 5
High level architecture

6. There were still some problems
6
Are our users
happy?
Are there common
patterns we can analyze?
How can we best
plan and evaluate?

7. 7
What if we started dogfooding our
own platform to analyze its use?

8. Requirements
8
Auditing Replayability Error analysisExperimentation Performance
monitoring

9. How?
9

10. 1. Auditing
• What - Query text
• Who - User
• Where - Cluster
• How - Presto/Hive
• When - Timestamp
• Stats (Optional)
‒ CPU seconds
‒ MB seconds
10

11. Hive Hook
11

12. Presto Event Listener
12

13. Event Schema
13

14. 14
Events ingestion

15. 2. Replayability
15

16. Replayer Benefits
• Reproduce errors
• Debug
• Fix
• Validate
16

17. 3. Error analysis
17

18. 4. Performance monitoring
•
18

19. 5. Experimentation
• Enrich benchto benchmark
• Golden set of queries
• Benchmark
‒ New version
‒ Config change
• Run queries to evaluate new systems
19

20. Bonus! Dogfooding data platform
• Renaming column name disallowed
‒ Data council
‒ More reviewers
• Testing/debugging event ingestion
‒ Unit testing
‒ Developer SDKs
20

21. Quick demo
21

22. Summary
• Users - GMs, data modelers, analysts, engineers & data scientists
• Architecture
• Dogfooding
• Learning
22

23. Thank you!
23
Mark Grover
Arup Malakar
Icons under Creative Commons License from
https://thenounproject.com/