My slide-deck from a webinar on the same topic for the Institute of Product Leadership, April 4th, 2017 What does it take to build killer products in the “AI-first” era? What makes for a great Data Science-driven product and how do great Product Managers leverage Data Science to drive value for customers? Find out how to avoid the pitfalls of hype-chasing Data Science tactics. Learn how to work with Data Science and Engineering to build a compelling product and solve real problems. Mandar takes a practitioner’s approach to present his recipe for success for building Data Science-driven products that drive enduring value for customers.

1. Product
Management
in the era of
Data Science

2. Mandar Parikh
@mandarparik
h
VP Product & Engineering
@ Entytle
Ex-
Aviso
Zuora
Salesforce

3. Today’s Webinar
• What does it take to build killer
products in the “AI-first” era?
• What makes for a great Data
Science-driven product and how do
great Product Managers leverage
Data Science to drive value for
customers?
• Find out how to avoid the pitfalls of
hype-chasing Data Science tactics.
• Learn how to work with Data Science
and Engineering to build a compelling
product and solve real problems.
Product
Data
Science
Engineering

4. Agenda
1. What is Data Science? What are Data Science-enabled
Products?
2. A Mini-Primer on Data Science
3. Designing Products with Data Science: Strategies for Success
4. Data Science Pitfalls
5. Engaging with DS Teams
6. Product Marketing & Data Science
7. Data Science Essentials for Product Managers

5. 1. What is Data Science? What are Data Science-enabled
Products?
2. A Mini-Primer on Data Science
3. Designing Products with Data Science: Strategies for Success
4. Data Science Pitfalls
5. Engaging with DS Teams
6. Product Marketing & Data Science
7. Data Science Essentials for Product Managers

6. What is “Data Science”?
• Not really science: not falsifiable, often untested, no explanatory
power
• An approach that analyzes large amounts of data (where large
is relative) and extracts patterns and insights and makes
predictions from it, to impact some business driver
• New term for statistical techniques that have existed for
decades

7. • Data Science is at the “core” of the offering; differentiating IP
• Tons of examples
• LinkedIn
• Uber
• Salesforce
• I won’t talk about other applications of AI including NLP, Image
Recognition et al.
1
What are “Data Science-enabled Products”?

8. What are “Data Science-enabled
Products”?
LinkedIn
• Early adopters of Data Science
• Graph search
• Skills
• Reputation
• Recruiting
• “People You May Know”
2

9. What are “Data Science-enabled
Products”?
Uber
• Driver Positioning
• Ride Demand Forecasting
• Surge Pricing
• Estimating ETA
3

10. • Started as a Contact
Manager, no Data
Science
• Salesforce Einstein
• “Provides Insights to
Users”
4What are “Data Science-enabled
Products”?
Salesforce

11. 1. What is Data Science? What are Data Science-enabled
Products?
2. A Mini-Primer on Data Science
3. Designing Products with Data Science: Strategies for Success
4. Data Science Pitfalls
5. Engaging with DS Teams
6. Product Marketing & Data Science
7. Data Science Essentials for Product Managers

12. Data Science Basics
• Features & Feature Selection
• ”Art of Data Science is to figure out which features to use”
• Unsupervised Learning
• “What patterns exist in the data?”
• Supervised Learning
• “Make predictions using existing patterns in the data”
• Popular algorithms: Random Forest, Gradient Boosted
Machines, Kernel Regression, Logistic Regression, Nearest
Neighbors, Clustering
• Deep Learning

13. A Word On Machine Learning
• DS, ML and AI used interchangeably
• I am not here to settle the debate, but I think of it as:
• DS model extracts patterns and makes predictions
• ML automatically calibrates the model and improves predictions over
time

14. Different
Types of
Analyses
Descriptive:
What happened?
01
Predictive:
What will
happen?
02
Prescriptive:
What should I do
(to achieve some
outcome)?
03

15. DESCRIPTIVE ANALYTICS

16. PREDICTIVE
ANALYTICS

17. Prescriptive Analytics
Price Optimization
1
Expert Systems /
Decision Support
Systems
2
What-if Systems
3

18. Accuracy v. Recall v. Precision v…..
Business Metrics
N=170 Predicted: YES Predicted: NO
Actual: YES 70 5
Actual: NO 45 50
Accuracy: How many times were you right? (70+50)/(70+50+5+45) = 70%
Recall: Of all the positives, how many times were you right? 70/(70+5) = 93%
Precision: How many times did you correctly predict YES? 70/(70+45) = 61%
But none are Business Metrics!

19. 1. What is Data Science? What are Data Science-enabled
Products?
2. A Mini-Primer on Data Science
3. Designing Products with Data Science: Strategies for Success
4. Data Science Pitfalls
5. Engaging with DS Teams
6. Product Marketing & Data Science
7. Data Science Essentials for Product Managers

20. Product Management Fundamentals Don’t Change
• Customer Empathy
• Data-driven
Salesforce Lead Scoring
Tell me why a lead is an 83
v.
#1

21. Cleanliness is Next to Godliness
• The data you get will not be clean – guaranteed
• Data cleansing is the single best thing you can do for your
data, beating incremental improvements you’ll get with a
”better” model
• Cleaning data is HARD! Don’t underestimate the effort involved.
#2

22. • Don’t over think it, and don’t let
your Data Scientists tell you
otherwise
• Start simple; you might be
surprised how far you can get
• Don’t get stuck in Analysis –
Paralysis; be skeptical of
incremental improvements
#3The Best Model is Often
the Simplest One

23. Get the AI Out of the Way
Opportunity Score
Opportunity 42929 48%
Opportunity 23427 68%
Opportunity 55620 23%
Opportunity Score
Opportunity 42929 Medium
Opportunity 23427 High
Opportunity 55620 Low
#4

24. Make it Actionable
Opportunity Score Why?
Opportunity 42929 Medium Time-in-stage,
Meeting/email sentiment
Opportunity 23427 High High level of activity,
External factors
Opportunity 55620 Low Close date pushed
AKA Interpretability
#5

25. Your Users Are NOT Data Scientists
No need to
show the
confidence
level or
margin of
error
Users like plain-speak
explanations.
#6

26. Design is AI’s Best Friend
Cadence
is key
Progressive Disclosure
Drill-down
Flexible &
uncluttered
Trend arrows
are great
Name the segments
#7

27. • Less is more
• Show key insights, skip the details (but be
able to drill-in)
• Articulate the insights, don’t keep the user
guessing
• Anomaly Detection is powerful at scale and is
great for ”management by exception”
• If you’re using Clustering, it’s often insightful
to expose the output to managers
• Managers love Prescriptive Analytics;
consider What-If modeling
http://www.tatvic.com/blog/wp-
content/uploads/2017/01/fetured.jpg
Prescriptive Analytics
Anomaly Detection
#8
Managers’ Deserve Less!

28. Tools
■ BITools LikeTableau, Domo
■ Excel, R, Jupyter for quick
prototyping
■ D3.js – very popular for building
rich, interactive visualizations
Techniques
■ “TheVisual Display of Quantitative
Information”, by EdwardTufte
■ …
Don’t Visualize Data:
Tell Stories
#9

29. Be The QA You Want To See
KPIs & Metrics
01
Backtesting
02
#10

30. Backtesting:
The Basic Idea
1. Take a big slice of valid data
2.Split into training and testing data
sets
a. Training data set should be several
multiples of “business cycles”
b. Testing data set should probably
be a few “business cycles”
3. Train the model on the training
data set
4.Run the model on the testing
data set and generate predictions
over the testing period
5. Compare actuals to predictions in
the testing period

31. Backtesting: Motivations
• Basis for comparison for new stuff from Data Science; baseline for
measuring marginal value added
• Regression test for Engineering
• Quantifiable quality test for Product & Customer Success
• Aid in Sales and Marketing

32. 1. What is Data Science?What are Data Science-enabled Products?
2. A Mini-Primer on Data Science
3. Designing Products with Data Science: Strategies for Success
4. Data Science Pitfalls
5. Engaging with DSTeams
6. Product Marketing & Data Science
7. Data Science Essentials for Product Managers

33. All Big Data Is Not Equal
• Particularly in Enterprise systems, the amount of data is not “Big”
• Can make useful predictions or provide useful insights on relatively
small amount of data
• E.g.Years worth of Opportunity Data <<A month’s worth of Censor Data
• More data does not necessarily mean better predictions
• E.g.Very old data may be “out-of-date” and may impact accuracy negatively

34. • You don’t need Data Science
ALL the time
• Don’t shy away from a “simple"
solution just because there is
no DS involved
Data Science is NOT a Silver Bullet

35. 1. What is Data Science? What are Data Science-enabled
Products?
2. A Mini-Primer on Data Science
3. Designing Products with Data Science: Strategies for Success
4. Data Science Pitfalls
5. Engaging with DS Teams
6. Product Marketing & Data Science
7. Data Science Essentials for Product Managers

36. Engaging with DS Teams
• Integral part of the Product & Engineering Team
• Much of Data Science is (also) Software Engineering
• Should be part of the “normal” product cadence and processes
• Bring in Data Science early into Customer conversations (a la
Design)
• Drive use-case-based initiatives with Data Science
1

37. Engaging with DS Teams
• Help DS understand and define the problem
• Help with the art, leave the science to DS
• Help DS understand the context, assist in feature-selection
• DS picks the algorithms
• Much like API design, agree on the contract: inputs and outputs
• Don’t forget the UX part of Data Science
• Reinforce Business Metrics
2

38. 1. What is Data Science? What are Data Science-enabled
Products?
2. A Mini-Primer on Data Science
3. Designing Products with Data Science: Strategies for Success
4. Data Science Pitfalls
5. Engaging with DS Teams
6. Product Marketing & Data Science
7. Data Science Essentials for Product Managers

39. • As the spokesperson for your product, learn to speak Data
Science
• You will have to grapple with:
• When to mention DS and when not to?
• When you speak about DS, how much do you give-away?
• How do you explain algorithms to the lay user?
• Understand your audience, understand the context
1
Product Marketing with Data Science

40. Context
• Consumer App: “Powered by sophisticated AI” works
• Business App: “Powered by sophisticated AI” could back-fire
with users who have been following a process for decades
2
Product Marketing with Data Science

41. What to say & How to say it
• Check with your DS team and Legal/IP on what NOT to say
• General approach to solving the problem is probably okay
• IP is usually in the end-to-end solution, not the choice of model
• Stick with simple explanations that your grandmother could
understand
• Be prepared to back up your claims (proof-points)
3
Product Marketing with Data Science

42. 1. What is Data Science? What are Data Science-enabled
Products?
2. A Mini-Primer on Data Science
3. Designing Products with Data Science: Strategies for Success
4. Data Science Pitfalls
5. Engaging with DS Teams
6. Product Marketing & Data Science
7. Data Science Essentials for Product Managers

43. Data Science Essentials for Product Managers
• A background in basic statistics is an absolute must
• Pick up a book with a “high-level” intro to Data Science; my favorites are:
• Numsense! Data Science for the Layman: No Math Added; by Annalyn Ng, Kenneth
Soo
• Data Science for Dummies, Lillian Pierson
• An “Intro to Data Science” online class could be useful; here’s one: Intro to
Data Science (Udemy)
• Unlike Balsamiq or Sketch for UX Design, there are no “starter” toolkits for
Data Science-related prototyping
• Understanding Jupyter (or similar) output (metrics, curves, distributions) is
a good skill to develop
• Excel or analytic tools like Looker are great arsenal to have
• Ability to collaborate closely with your Data Scientists is key

44. Thank You
@mandarparikh