Download as PDF, PPTX
More Related Content
Dogfooding data at Lyft
- 1. Mark Grover |@mark_grover Arup Malakar | @amalakar 1
- 2. Agenda • Goals ofthe Lyft data platform • High level architecture • Story about dogfooding 2
- 3. Goals of Lyft’sdata platform 3 Data Modelers Data Analysts Data Scientists General Managers Data Platform Engineers
- 4.
- 5.
- 6. There were stillsome problems 6 Are our users happy? Are there common patterns we can analyze? How can we best plan and evaluate?
- 7. 7 What if westarted dogfooding our own platform to analyze its use?
- 8. Requirements 8 Auditing Replayability ErroranalysisExperimentation Performance monitoring
- 9.
- 10. 1. Auditing • What- Query text • Who - User • Where - Cluster • How - Presto/Hive • When - Timestamp • Stats (Optional) ‒ CPU seconds ‒ MB seconds 10
- 11.
- 12.
- 13.
- 14.
- 15.
- 16. Replayer Benefits • Reproduceerrors • Debug • Fix • Validate 16
- 17.
- 18.
- 19. 5. Experimentation • Enrichbenchto benchmark • Golden set of queries • Benchmark ‒ New version ‒ Config change • Run queries to evaluate new systems 19
- 20. Bonus! Dogfooding dataplatform • Renaming column name disallowed ‒ Data council ‒ More reviewers • Testing/debugging event ingestion ‒ Unit testing ‒ Developer SDKs 20
- 21.
- 22. Summary • Users -GMs, data modelers, analysts, engineers & data scientists • Architecture • Dogfooding • Learning 22
- 23. Thank you! 23 Mark Grover ArupMalakar Icons under Creative Commons License from https://thenounproject.com/