Downloaded 19 times
IDEAS Amundsen Presentation
- 1. Saturday, October 26th2019 Alagappan Sethuraman | Engineering Manager, Lyft Daniel Won | Software Engineer, Lyft Disrupting Data Discovery
- 2. Agenda • What isData Discovery? • Challenges in Data Discovery • Introducing Amundsen • Amundsen Architecture • Impact and Future Work 2
- 3. What is DataDiscovery? 3
- 4. Data is usedto make informed decisions 4 Analysts Data Scientists General Managers Engineers ExperimentersProduct Managers Data-driven decision making process: 1. Search & find data 2. Understand the data 3. Perform an analysis/visualization 4. Share insights and/or make a decision Make data the heart of every decision
- 5. What is DataDiscovery? Consider a data-driven decision making process: 1. Search & find data 2. Understand the data 3. Perform an analysis/create a visualization 4. Share insights and/or make a decision 5 Data Discovery
- 6.
- 7. • My firstproject is predict the attendance for IDEAS conference • Goal: Help the office team make a decision on number of chairs to provide? • Idea: Let’s take a look into attendance from previous conferences… but where do I look? Hi! I’m a new Analyst! 7
- 8. • Ask afriend/manager/coworker • Ask in a wider Slack channel • Search in the Github repos Step 1: Search & find data 8 We end up finding tables: hosted_events that seems to be the right one
- 9. • You findseveral columns that might be what you're looking for: ‒ booked, registered, and attendance • But you still have many questions such as: ‒ Does attendance include staff? ‒ What's the difference between booked and registered? ‒ How accurate are these figures? Step 2: Understand the data 9
- 10. Step 2: Understandthe data ● Look for further documentation on these columns ○ Where does this documentation live? ● Ask an expert who knows this table ○ Who is an expert? ● Run some queries to try to figure it out at the risk of being wrong 10 SELECT * FROM schema.host_events LIMIT 100;
- 11. Nearly 1/3 ofData Scientist time is spent in Data Discovery 11 • Data discovery is a problem because of the lack of understanding of what data exists, where, who owns it, & how to use it. • Data Discovery provides little to no intrinsic value • Impactful work happens in Analysis
- 12.
- 13. What is Amundsen? •Built at Lyft, official launch in late 2018 • Inspired by Google Search, Airbnb Data Portal, and Apache Gobblin • Named after Norwegian explorer Roald Amundsen ‒ Led the first expedition to the South Pole ‒ Led the first expedition through the Northwest Passage 13
- 14.
- 15.
- 16.
- 17.
- 18.
- 19. Computed Column Statistics Disclaimer:these stats are arbitrary.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25. Why choose agraph database? 25
- 26.
- 27.
- 28.
- 29. Neo4j is thesource of truth for editable metadata 29
- 30. Why not propagatethe editabled metadata back to source 30
- 31. Why not propagatethe editabled metadata back to source 31
- 32. Why not propagatethe editabled metadata back to source 32
- 33. Why not propagatethe editabled metadata back to source 33
- 34.
- 35. Amundsen’s Impact atLyft • Deployed at Lyft for over 1 year • Over 700 Weekly Active Users • 90% penetration among Data Scientists • Reduced mean time to discovery by 75% • Also used by Data Eng, Software Eng, PMs, Ops, Marketing Managers, and more 35
- 36.
- 37.
- 38.
- 39.
- 40.
- 41. • github.com/lyft/amundsen • 200+github stars, 10+ companies contributing back • Slack channel 250+ people from 30+ companies • Presented at conferences in San Francisco, Barcelona, Vilnius, Moscow, LA, NYC by Lyft employees and community Amundsen is Open Source! 41
- 42.
- 43.
- 44. Alagappan Sethuraman |/in/alagappanut Daniel Won | /in/danwon Project Code @ github.com/lyft/amundsen Icons under Creative Commons License from https://thenounproject.com/ 44
Editor's Notes
- #2 Name & Role working on an open-source data discovery tool at Lyft. It’s called “Amundsen” -- more on that name later. It leverages Neo4j, glad to share how we’ve been using Neo4j at Lyft to achieve goals of our product Amundsen.
- #3 On the agenda for this talk
- #4 Now onto challenges with data discovery
- #5 Effective data discovery is important because data is at the heart of every decision we make. It is the only way to make informed, objective decisions. Applies to many roles Data-driven decision making process Search & find data Understand the data Perform an analysis Share insights or make a decision
- #7 Now onto challenges with data discovery
- #8 To highlight some data discover pain points that occur without the proper tools, let’s walk through a hypothetical example
- #9 Your experience searching and finding data may involve doing all of the following 3 things.
- #10 Your experience understanding the data doesn’t get any easier. Each question leads to further questions - How was this data collected?
- #12 ⅓ of time on data discovery Difficult to find what exists, understand whether or not it’s what you are looking for, or trust that it is the source of truth for that information We can significantly increase productivity and impact if we can reduce this time...
- #13 We’ve talked about some pain points of data discovery and why it’s important, let’s talk about our solution -- Amundsen.
- #15 Disclaimer Representative data Amundsen circa March 2019 Our landing page is optimized for search Most common method of data discovery, presented with search bar & help text for some advanced search features We also want the landing page to be able to help users that don’t know what to search for. Created this concept of popular tables
- #16 Users presented with ranked search results Not like page-rank but based on relevance and popularity
- #24 Now onto challenges with data discovery
- #26 However graph databases are not common for many web applications, and so one might ask why choose a graph database.
- #27 Well if you remember the diagram of the data ecosystem at Lyft from the beginning of the talk, that can be modeled as a graph. This is a very powerful feature because the alternative to created these kinds of relationships with a RDBMS is joins A NoSQL database isn’t set up for this
- #28 As you may remember from the application walkthrough, Amundsen surfaces resource metadata and that is what we are storing in Neo4j
- #29 Let’s take a note of some of the features from the table detail page again and see how this is represented in Neo4j Walk through features What’s very beneficial about this is that when we have a new use case and a new piece of metadata to represent, we just have to create the new node and relationship.
- #30 Another key characteristic of our system is that neo4j is the source of truth for our editable metadata
- #31 This was actually not our original intent, we ran into a roadblock when we were first implementing the description editing feature. We originally had a setup like this
- #32 Then we realized we forgot to account for something. Tables can get rebuilt using the source code that generated the table and descriptions will be overwritten
- #33 The we thought about whether or not we could do this, update them both!
- #34 The answer was no. ...And that’s how Neo4j became the source of truth for editable metadata
- #35 Now onto challenges with data discovery
- #37 Now onto challenges with data discovery
- #41 Now onto challenges with data discovery
- #43 T