The Other 99% of a Data Science Project
•Download as PPTX, PDF•
5 likes•1,594 views
Slides from my talk at Open Data Science Conference 2016. Algorithms and models are an important (and cool) part of data science. This talk is about all the other steps that it takes to deploy a data science project that makes a product slightly smarter. Stuff that you hear from practitioners, but is not covered well enough in books.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to The Other 99% of a Data Science Project
Similar to The Other 99% of a Data Science Project (20)
Recently uploaded
Recently uploaded (20)
The Other 99% of a Data Science Project
- 1. THE OTHER 99% OF A DATA SCIENCE PROJECT Open Data Science Conference Santa Clara | November 4-6th 2016 Eugene Mandel @eugmandel
- 2. ∎ @eugmandel ∎ lead of data science at directly ∎ formerly: □ data science team at Jawbone □ co-founder qualaroo, jaxtr ABOUT ME
- 3. DATA SCIENCE NEEDS PRODUCT MANAGEMENT success of a data science project has as much to do with product management as with data science
- 4. 2 KINDS OF DATA SCIENCE B ANALYZE A BUILD
- 7. ∎ “don’t you know me?!” -> “you get me!” ∎ get smarter with every interaction ∎ reduce search space SMART PRODUCTS
- 8. SMART PRODUCTS BUT NOT THAT SMART...
- 11. Show and explain your web, app or software projects using these gadget templates. PARKING APP ON DEMAND CUSTOMER SUPPORT
- 13. PROBLEM: choose support tickets that expert users can resolve
- 16. GETTING THE DATA
- 17. GETTING ALLIES
- 18. GETTING THE DATA
- 19. CLEAN YOUR DATA Automated bug reports Surveys Bounced emails Internal tickets Email metadata Email threads ...
- 22. TRAINING - COLD START PROBLEM all tickets tickets seen by expert
- 23. TRAINING -GET LABELS “Is there a cat in this picture?” “Is this support ticket resolvable?”
- 24. TRAINING -GET LABELS ∎ label manually ∎ derive labels from user behavior ∎ derive labels from external sources ∎ mix
- 25. My favorite data science algorithm is division. Monica Rogati Former VP of Data, Jawbone & LinkedIn data scientist
- 26. Tokenization Bag of words (BOW) Tf–idf Random Forest Classifier MODEL
- 27. DEVELOPMENT
- 28. PLAYING WELL WITH ENGINEERING ∎ gaining trust ∎ development process
- 29. POINTS OF INTEGRATION online or offline?
- 32. IS IT WORKING? evaluating data products Image source: https://themouseandthewindmill.wordpress.com
- 34. IS IT WORKING? QA’ing data products Image source: https://themouseandthewindmill.wordpress.com
- 37. IN PRODUCTION
- 38. THE KNOBS: HOW TO CONTROL THE PRODUCT ∎ on/off switch per customer ∎ prediction threshold ∎ exclusions
- 39. “... SMART…” “... AI …” “...MACHINE LEARNING…” “...INTELLIGENT…” NAMING THINGS
- 40. UPDATING THE MODEL ∎ input data changes ∎ users behaviour changes ∎ dataset grows
- 41. NEGATIVE SAMPLING send small % of predicted negative as if they were positive predicted positive
- 42. NEGATIVE LABELING send small % of predicted negative for manual labeling predicted positive
- 43. ∎ “Would you be able to resolve this ticket successfully?” ∎ “Would an expert user be able to resolve this ticket successfully?” ∎ “Would an expert user be able to resolve this ticket successfully without getting a negative rating?” LABELING - HOW TO PHRASE THE QUESTION?
- 44. ∎ customers ∎ sales ∎ account managers ∎ marketing ∎ execs MESSAGING
- 46. DATA ETHICS
- 49. ∎ Presentation template by SlidesCarnival ∎ Images: □ http://jedismedicine.blogspot.com/ □ Jawbone □ Directly □ Wikipedia □ https://themouseandthewindmill.wordpress.com □ http://www.imdb.com/ CREDITS
Editor's Notes
- There are 2 big areas of data science - A for “analyze” and B for “build”. A is product development informed by data. It became adopted pretty widely by now. Having analytics, running A/B tests, doing cohort and funnel analysis became part of the product management culture. The “build” kind of data science is about building smarts into the product itself and this is the kind I want to talk about. Implementing some of this requires machine learning and it is important for product managers to understand the level of complexity of some techniques that apply to their products. However, when machine learning is discussed, too much emphasis is put on the algorithms. More needs to be said about how a smart product gains humans’ trust and make them feel good about using it.
- An app that allows you to pay for parking. You fire it up, it shows 3 choices - start a new parking session, see you old sessions. Choose “Start a new session”, go to next screen, there are several options here - select a parking zone. Done. I would not give this a second thought on a desktop.
- But when I use this app, I’m late, I hold the phone in one hand and trying to pay for parking while running to the ferry. I am running and fumbling with the phone and thinking - DON’T YOU KNOW ME?! It’s a weekday morning, I am at the parking lot next to the ferry terminal, you have seen me here before. More than once. Just give me one button - PAY NOW. And a small link to all the other features.
- Every time a user has this “DON’T YOU KNOW ME?!” moment, it is an opportunity to make a product just a little bit smarter. Smart products convert DONT YOU KNOW ME?! into YOU GET ME! Even when they don’t know my next step exactly, they reduce the search space.
- Smarter products - new problems. Complexity goes way beyond the algorithms.
- Take Nest smart thermostat - great visual design, easy to install, it is powered by machine learning that learns your preferences. It’s a good product, but even they can’t get it quite right. Got it when we just had our baby. We both like it pretty cool, but my wife felt cold after birth. This is just when Nest was learning. Once it did, for some reason it was very tough for it to adjust. Another thing - when it turns the heater on, there is no indicator is it was a human in the house or the software. I am OK correcting Nest. But not my wife. Making products smarter introduces probabilistic behavior. Because probabilistic behavior feels kind of like life, you start having different expectations. Northern California has some very hot days with cold mornings. On a day like that I would not turn the heater on in the morning. But Nest would. It just knows - get to 68 degrees. But it has no context - something that is easy and intuitive to a human is not easy to software.
- Getting the relationship of the user with a smart product right is tricky. Product managers are the best people in a company to get the tradeoffs right. Just like a pm does not have to be developer to manage a software product, she does not have to be a mathematician or a data scientist to manage a data product. But it is necessary to understand some core concepts. I'll use 2 data products to demonstrate some of these necessary concepts.
- Here is the second data product. This one is B2B and is working in the background. Directly helps companies like Airbnb, Linkedin, Pinterest with on-demand customer support. When a user submits a support ticket, some of these are sent to Directly which distributes them to a network of expert users that are ready to answer them. If experts resolve a question successfully, they get paid and Directly takes a cut. Otherwise, the experts can reroute the ticket back to the customer’s call center.
- When questions are created in the helpdesk how do we find ones that the expert users can (and want) to solve? Initially, we relied on our customers to configure some categories that their users chose when they were filling out the support form. Users are not great about categorizing their issues. We tried keywords. Very cumbersome to manage. We need to pick as many tickets as we can, but not to create too much noise for the experts.
- Getting the relationship of the user with a smart product right is tricky. Product managers are the best people in a company to get the tradeoffs right. Just like a pm does not have to be developer to manage a software product, she does not have to be a mathematician or a data scientist to manage a data product. But it is necessary to understand some core concepts. I'll use 2 data products to demonstrate some of these necessary concepts.
- Solution: let us look at at ALL your tickets as they come in and a machine learning model will choose which ones will be sent to the expert users. Here is how it works: ….. Explain the image The model is a classifier and it needs examples to learn what a good ticket looks. It can do so from watching how the experts respond to tickets they have seen earlier. If the experts took a ticket and resolved it successfully, it becomes a positive example. If the send the question back or resolve it, but the user reviews their answer negatively, this question becomes a negative example.
- ML startups ask companies “give us all your data” I was preparing for a touch conversation. Getting access to more and better data… “Is it a yes?” Think of getting data early, before you need it Legal. Stripping of anything personal. Insist on storing.
- Customer success (Account managers) - interested. One of the main metrics they are responsible for is our ticket share- percentage of tickets we are handling at a customer.
- ML startups ask companies “give us all your data” I was preparing for a touch conversation. Getting access to more and better data… “Is it a yes?” Think of getting data early, before you need it Legal. Stripping of anything personal. Insist on storing.
- The improvements that you can get from cleaning your data are great. The plot of the movie Big Short can be summarized as “guys clean a dataset, get rich”. In case of Jawbone meal logging, the biggest lyft in performance came from realizing that breakfasts are different from other meals. Spinach in the morning was probably a part of omelete. Spinach at lunch was most likely a salad. Sometimes, cleaning your data requires a good understanding of the domain you are working with. Which properties of your data you do and don’t use is to a significant degree a product management decision. For example, different cuisines disagree on what foods are eaten best together. Do you use this knowledge somehow? Depends what you know about your users.
- Monica Rogati, who used to be VP of data at Jawbone has this saying:... Yes, you could go much more advanced algorithm, but this simple one can get you pretty far. the biggest improvements were achieved by cleaning the data and understanding it deeply
- Account managers - interested
- How do we know if a model is good? When "normal software” breaks, it breaks with high visibility. An issue with ML is that it will ALWAYS give you an answer. How we compare models? An obvious metric is accuracy. Basically the percentage of predictions that the algorithm, gets right. However in product is data science this is a very bad metric. This depends on how balanced or unbalanced the classes that you are predicting are. Example: fraud detection, rare disease testing. If 0.1% of transactions are fraudulent, you can create a “very sophisticated” predictive model. When asked “Is this transaction fraudulent?” it will always say “no”. The accuracy of this model will be about 99.9%. Thinking through this is exactly the PM’s job. In this case you don’t need to know the math that underlies the predictive model. How do we QA data products?
- How do we know if a model is good? When "normal software” breaks, it breaks with high visibility. An issue with ML is that it will ALWAYS give you an answer. How we compare models? An obvious metric is accuracy. Basically the percentage of predictions that the algorithm, gets right. However in product is data science this is a very bad metric. This depends on how balanced or unbalanced the classes that you are predicting are. Example: fraud detection, rare disease testing. If 0.1% of transactions are fraudulent, you can create a “very sophisticated” predictive model. When asked “Is this transaction fraudulent?” it will always say “no”. The accuracy of this model will be about 99.9%. Thinking through this is exactly the PM’s job. In this case you don’t need to know the math that underlies the predictive model. How do we QA data products?
- How do we QA data products? When "normal software” breaks, it breaks with high visibility. An issue with ML is that it will ALWAYS give you an answer. Monitoring in production
- Unless you are making the ultimate data product - a make money while you sleep fund runner :) - your system lives in the world and interacts with people. Once the product is out, other people carry the message and you cannot control it. Listen to how an account manager talks about this with a client, how a salesperson talks with a prospect. ML/DS is uniquely susceptible to BS - how to control it?
- "Why did you show me ‘french fries’?" Well, because this is the item that is logged together most frequently with burger. "Why you decided that this transaction is fraudulent? Why did you decide that this customer support ticket is resolvable?" the simpler the model the more interpretable it is. When a model is not easily interpreted, but it performs well, it’s your task to manage expectations.