Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
www.productschool.com
How Machine Learning Delivers
Personalized Experiences by Expedia
Group Product Manager
FREE INVITE
Join 35,000+ Product Managers on
COURSES
Product Management
Learn the skills you need to land a product manager job
COURSES
Coding for Managers
Build a website and gain the technical knowledge to lead software engineers
COURSES
Data Analytics for Managers
Learn the skills to understand web analytics, SQL and machine learning concepts
COURSES
Digital Marketing for Managers
Learn how to acquire more users and convert them into clients
COURSES
UX Design for Managers
Gain a deeper understanding of your users and deliver an exceptional end-to-end
experience
COURSES
Product Leadership for Managers
For experienced Product Managers looking to gain strategic skills needed for top l...
COURSES
Corporate Training
Level up your team’s product management skills
Speaker
Ammar Jawad
Expedia Group
Product Manager
How Machine Learning
Delivers Personalized
Experiences
Ammar Jawad
Product Manager, Personalisation & ML Platform
Expedia ...
Agenda
- Understanding AI, ML and Deep Learning
- AI and Product Management
- Degree of Personalisation
- Feedback loops a...
Difference between software engineering and ML
In traditional software engineering...
13
In (most) ML applications...
A hu...
AI, Machine Learning and Deep Learning
14
AI is...
A discipline that has to do
with the theory and
methods to build machin...
Impact of ML on Products
15
No more one-size-fits-all solutions Increased commercial value
● Adapt to users
● Anticipate u...
When should a consumer-facing feature be powered by ML?
16
Would some users find
the feature more
relevant than others?
Ma...
Types of ML features
17
Feature-level
● Recommended destinations custom-tailored to
each user
● Ordering images based on w...
Degree of Personalization - Considerations for Product
18
Non-Personalized Targeted Highly Targeted Semi-Personalized Pers...
Personalization: Linearity & Nonlinearity
19
How most product people think... How we should be thinking...
Targeted products: The case for Multi-Armed Bandits
20
Variant AControl Variant B
Variant
C
25%
25% 25%
25%
A/B Testing
Mu...
Targeted products: Multi-Armed Bandits (Part II)
Online decision making
- Should I exploit? (make the best decision given ...
Targeted products: Multi-Armed Bandits (Part III)
22
Three main approaches to exploration:
1. Random exploration
Explore b...
Reinforcement Learning in Online decision making
23
Action Reward
Action RewardContext
Multi-Armed Bandits
Contextual Band...
Highly Targeted Products: Contextual Bandits
24
Region Time of Day Variants
Latin America Morning Control
EMEA Night Varia...
Optimising the right KPI in Reinforcement Learning
25
The AI agent is optimising for “score earned” which
has unintended c...
Personalized products: Recommender Systems (RS)
Recommender systems are based on calculating similarity and distance betwe...
Personalized products: Humans-in-the-loop in RS
27
User1 User2
Dominican Republic Dominican Republic
Thailand Thailand
Cam...
Feedback Loops: Personalization is data-hungry
28
MODEL TUNING
PROCESS
INSIGHT
GENERATION
INPUTS OUTPUTS
Adapted from “How...
Collective Intelligence
29
Source: “Collective Intelligence in Action”, Satnam Alag
As users interact on the web and expre...
Rise of the AI Product Manager
● 20% of jobs in the UK expected to be displaced by AI over the next 20 years 1
○ Approxima...
Roadmap to become a great AI Product Manager
● Excel (formulas, pivot tables, vlookups)
● Statistics (descriptive, inferen...
Summary
● Understanding AI, ML and DL
● Product and ML
● Degree of Personalization
● Feedback loops in features
● Collecti...
Contact and links
- LinkedIn: linkedin.com/in/ammarjawad
- Email: b-ajawad@hotmail.com
- Quora: www.quora.com/profile/Amma...
www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design and Product Leade...
Upcoming SlideShare
Loading in …5
×

How Machine Learning Delivers Personalized Experiences by Expedia Group Product Manager

2,631 views

Published on

Product Management Event at #ProductCon London about Machine Learning and how it Delivers Personalized Experiences by Product Manager at Expedia Group, Ammar Jawad.

Published in: Technology

How Machine Learning Delivers Personalized Experiences by Expedia Group Product Manager

  1. 1. www.productschool.com How Machine Learning Delivers Personalized Experiences by Expedia Group Product Manager
  2. 2. FREE INVITE Join 35,000+ Product Managers on
  3. 3. COURSES Product Management Learn the skills you need to land a product manager job
  4. 4. COURSES Coding for Managers Build a website and gain the technical knowledge to lead software engineers
  5. 5. COURSES Data Analytics for Managers Learn the skills to understand web analytics, SQL and machine learning concepts
  6. 6. COURSES Digital Marketing for Managers Learn how to acquire more users and convert them into clients
  7. 7. COURSES UX Design for Managers Gain a deeper understanding of your users and deliver an exceptional end-to-end experience
  8. 8. COURSES Product Leadership for Managers For experienced Product Managers looking to gain strategic skills needed for top leadership roles
  9. 9. COURSES Corporate Training Level up your team’s product management skills
  10. 10. Speaker Ammar Jawad Expedia Group Product Manager
  11. 11. How Machine Learning Delivers Personalized Experiences Ammar Jawad Product Manager, Personalisation & ML Platform Expedia Group 11
  12. 12. Agenda - Understanding AI, ML and Deep Learning - AI and Product Management - Degree of Personalisation - Feedback loops and Collective intelligence - AI Product Manager 12
  13. 13. Difference between software engineering and ML In traditional software engineering... 13 In (most) ML applications... A human analyses a problem, writes code, turns into a program which then translates inputs to outputs. A computer figures out the best program to write using statistics by looking at a set of examples and their desired output.
  14. 14. AI, Machine Learning and Deep Learning 14 AI is... A discipline that has to do with the theory and methods to build machines to resemble humans. ML is... A toolset which can be used to solve certain kinds of AI problems. Machines do not start out intelligent but become so after being trained. DL is... A subfield of ML which learn representations of data by layers of increasingly meaningful representations. Layered representations are learned through neural networks. Machine Learning Deep Learning Artificial Intelligence
  15. 15. Impact of ML on Products 15 No more one-size-fits-all solutions Increased commercial value ● Adapt to users ● Anticipate users’ needs ● Fulfill user intent ● De-averaging ● Upsell, cross sell and the right complimentary products ● Human-level expertise at scale
  16. 16. When should a consumer-facing feature be powered by ML? 16 Would some users find the feature more relevant than others? Machine Learning Heuristic rules Yes No
  17. 17. Types of ML features 17 Feature-level ● Recommended destinations custom-tailored to each user ● Ordering images based on what the user is most likely interested to see ● Chatbots to answer simple customer service tickets Delivery-level ● Multi-Armed Bandits experimenting to identify the right widget size per user segment ● Contextual Bandits testing different string translations based on behaviour/technological differences of users in a particular region, e.g. youth/adults.
  18. 18. Degree of Personalization - Considerations for Product 18 Non-Personalized Targeted Highly Targeted Semi-Personalized Personalized Source: BBC Source: Katar Investments Source: Copenhagen Airport Source: Manning Publications Source: Spotify Low High
  19. 19. Personalization: Linearity & Nonlinearity 19 How most product people think... How we should be thinking...
  20. 20. Targeted products: The case for Multi-Armed Bandits 20 Variant AControl Variant B Variant C 25% 25% 25% 25% A/B Testing Multi-Armed Bandits 75% of all traffic is sent to suboptimal variants. Control Variant A Variant B Variant C 25% 25% 25% 25% Day 1 Control Variant A Variant B Variant C 13% 28% 5% 54% Day 2 Variant A Variant C 19% 81% Day n
  21. 21. Targeted products: Multi-Armed Bandits (Part II) Online decision making - Should I exploit? (make the best decision given current info) - Should I explore? (gather more info) Best long-term strategy may involve short-term sacrifices to maximise long-term gain Other examples may include: 21
  22. 22. Targeted products: Multi-Armed Bandits (Part III) 22 Three main approaches to exploration: 1. Random exploration Explore based on a probability to take a random action, e.g. explore 20% of the time. 1. Optimism in the face of uncertainty When faced with options for which we know the value of each except one action which value is unknown then there is a bias towards the action with an unknown outcome. 1. Information state space Consider agent’s information as part of its state Look ahead to see how information helps reward
  23. 23. Reinforcement Learning in Online decision making 23 Action Reward Action RewardContext Multi-Armed Bandits Contextual Bandits
  24. 24. Highly Targeted Products: Contextual Bandits 24 Region Time of Day Variants Latin America Morning Control EMEA Night Variant A APAC Morning Variant B Contextual Bandits are Multi-Armed Bandits for each segment
  25. 25. Optimising the right KPI in Reinforcement Learning 25 The AI agent is optimising for “score earned” which has unintended consequences.
  26. 26. Personalized products: Recommender Systems (RS) Recommender systems are based on calculating similarity and distance between a set of users or items 26 UserId MovieId Rating 0 0 4 0 1 5 0 2 4 2 0 4 2 1 4 2 2
  27. 27. Personalized products: Humans-in-the-loop in RS 27 User1 User2 Dominican Republic Dominican Republic Thailand Thailand Cambodia Cambodia The Netherlands The Netherlands Spain Spain Brazil Brazil Philippines Hypothetical example of destinations travelled by two users
  28. 28. Feedback Loops: Personalization is data-hungry 28 MODEL TUNING PROCESS INSIGHT GENERATION INPUTS OUTPUTS Adapted from “How Googles does Machine Learning”, Coursera
  29. 29. Collective Intelligence 29 Source: “Collective Intelligence in Action”, Satnam Alag As users interact on the web and express their opinions, they influence others.
  30. 30. Rise of the AI Product Manager ● 20% of jobs in the UK expected to be displaced by AI over the next 20 years 1 ○ Approximately equal to the additional jobs created by AI in the same period. ● Companies increasingly leverage ML to gain competitive advantages ● Users increasingly demand better experiences only delivered via AI ● Personalisation only possible through ML ● Tech companies are looking for product leaders with AI expertise to help navigate 30 1. PWC, UK Economic Outlook July 2018
  31. 31. Roadmap to become a great AI Product Manager ● Excel (formulas, pivot tables, vlookups) ● Statistics (descriptive, inferential) ● Maths (algebra, linear algebra, calculus) ● Coding in Python (variables, functions, objects) ● SQL (joins, order by, group by, data aggregation, basic subqueries) ● Machine Learning (supervised, unsupervised) ● Deep Learning (ANNs, CNNs, NLP) 31
  32. 32. Summary ● Understanding AI, ML and DL ● Product and ML ● Degree of Personalization ● Feedback loops in features ● Collective intelligence in products ● AI Product Managers 32
  33. 33. Contact and links - LinkedIn: linkedin.com/in/ammarjawad - Email: b-ajawad@hotmail.com - Quora: www.quora.com/profile/Ammar-Jawad - Presentation available here: https://goo.gl/1yd25g © Expedia, Inc. All rights reserved. 33
  34. 34. www.productschool.com Part-time Product Management, Coding, Data Analytics, Digital Marketing, UX Design and Product Leadership courses in San Francisco, Silicon Valley, New York, Santa Monica, Los Angeles, Austin, Boston, Boulder, Chicago, Denver, Orange County, Seattle, Bellevue, Washington DC, Toronto, London and Online

×