According to stats, 85% of Artificial Intelligence (AI) / Machine Learning (ML) / data science (DS) projects fail, which hinders companies' appetite in investing in AI/ML/DS, and holds back data scientists from getting the recognition they deserve. In this talk dated 15 June 2019, Kevin Wong presented a gentle introduction on how he applied a re-invented Product Management approach to AI projects, in order to maximise their likelihood of success.

1. STRICTLYCONFIDENTIAL
Maximising Likelihood of Success:
Applying Product Management to AI projects
Kevin Wong, Senior Product Manager, Search Ranking
Berlin, German-Chinese Association of Artificial Intelligence | June 16 2019

2. Speaker Bio
2
Senior Product Manager
@HomeToGo (Search Ranking)
Communications Tutor
@Data Science Retreat, Berlin DS bootcamp
Past Experience:
~5 years of experience in Product Management
specialised in data science projects

3. Why this talk?
3
85% of AI/data science projects fail.
What define project success?
- The product / project achieved its desired objectives within time &
budget constraints.
- The delivered product or package is attractive to users in certain ways
compared to competing products.
- It is delivered in a way that maximises value and utilised by users.
Improving likelihood of project success
→ Improving likelihood of personal success

4. Agenda
1.What is Product Management?
2.How to apply PM techniques to maximise likelihood of your AI projects’ success?
3.Q&A
4

5. Agenda
1.What is Product Management?
2.How to apply PM techniques to maximise likelihood of your AI projects’ success?
3.Q&A
5

6. Product Management
≠
Project Management

7. What is a Product?
A unit of exchange (either a physical item or intangible
service)
between value producers and value consumers who
understood its value.
7

8. Project vs Product Management
8
Project Management Product Management
Scope
Time Cost
Delivery
Users
Business Developers
Value
Discovery,
Validation,
Proposition
& Positioning
Product
What, Why &
How to get customers hooked
What, How, When

9. Product Owners vs Product Managers
9
Product Owner (SCRUM) Product Manager
Users
Business Developers
Users
Business Developers
Also true for Product Managers,
but this is still secondary concern
First and foremost - is this the most
value adding problem/solution to both
the users and the business,
given what is feasible?

10. Different ways to organise AI work
10
Team Leaders Product Manager
Data
Scientists
Data
Engineers
Software
Developers
Product
Managers
Tech
Leads
Data
Scientists
Data
Engineers
Software
Developers
Team
Leads

11. Agenda
1.What is Product Management?
2.How to apply PM techniques to maximise likelihood of your AI projects’ success?
– Reduce Uncertainties Early
– Being Agile
– Data-Driven Decisions
3.Q&A
11

12. 2a) Reduce Uncertainties Early
Typical Product Uncertainties, e.g.
• Do we really understand what the users
or the business’ needs?
• Is this the most effective / highest ROI
way to solving the problem?
• Will users use or pay for our product?
• How can we keep users engaged?
• Are we best positioned in the market
against our competitors given our
product strategy?
12
Uncertainties especially for AI, e.g.
• Can we achieve the desired precision and
recall with the amount of data we have?
• Are we optimising for the right metric?
• Given we trained the model offline, is it
really going to make an impact online?
• Do we have good-enough data quality?
• Would stakeholders understand what we
are doing so that they support our cause?
• Does our AI framework allow us to iterate
quickly?

13. 2a) Reduce Uncertainties Early
Dealing with Product Uncertainties
• Exploratory Data Analysis
• User Research
– Prototyping
– Design Sprints
• Calculate ROI & consider uncertainty
• Partner with Customers Early
• Build or Buy
• Being Agile → see subsection 2(b)
13
Uncertainties especially for AI, e.g.
• Set success criteria & metric to optimise early
→ see subsection 2(c)
• Exploratory Data Analysis
• Simplier Model & Model Explainability
• Data Quality Monitoring
• Invest in AI Infrastructure
• OKR adjusted for AI projects
• Bring your stakeholders along step by step

14. 2b) Being Agile - Agile Manifesto
Principle 2: Welcome changing
requirements, even late in development.
Agile processes harness change for the
customer's competitive advantage.
Principle 3: Deliver working software
frequently, from a couple of weeks to a
couple of months, with a preference to the
shorter timescale.
14
Principle 5: The most efficient and effective
method of conveying information to and
within a development team is face-to-face
conversation.
Principle 10: Simplicity--the art of
maximizing the amount of work not done--is
essential.

15. 2b) Being Agile - Building Product
15
This might not fit half of data science projects
though. see next slide for my modification.
Learn
Build
Measure
Feedback

16. 2b) Being Agile - Feedback Loop
16
Building AI products - Re-interpretating Agile Manifesto
1. Breakdown uncertainties in AI projects into small,
tangible steps with clearly defined success criteria.
2. Set sprint goals to set reasonable expectations before
start of sprint.
3. Demonstrate progress in terms of answered
hypothesis or reduced risks, if prototypes are not
possible.
Learn
Build
Measure
Feedback

17. 2c) Data Driven Decisions
17
1. Exploratory Data Analysis (If it is important, repeat 3 times)
2. Setting KPIs & success criteria
3. A/B Testing
– Do we have a good A/B testing framework?
– How long does an A/B test takes?
– Do we have strong offline metrics that correlate well with A/B tests success?
– How should we design experiment variants to understand whether success is a
direct result of our projects?

18. STRICTLYCONFIDENTIAL
Thank you for your attention!
Berlin, German-Chinese Association of Artificial Intelligence | June 16 2019

19. Agenda
1.What is Product Management?
2.How to apply PM techniques to maximise likelihood of your AI projects’ success?
– Reduce Uncertainties Early
– Being Agile
– Data-Driven Decisions
3.Q&A
19