We will look at ways of applying data science and machine learning to better understand customers and improve their user experience. From a practical industry application perspective, we will discuss the following: measuring popularity, statistical significance in A/B testing, survival analysis, predictive lifetime value and recommendation systems. We will review the concepts and some of the math behind these, while also addressing the real world challenges faced by many of these implementations.

1. DATA
SCIENCE
POP UP
AUSTIN
Predicting Customer Behavior &
Enhancing Customer Experience
Vinny Senguttuvan
Sr. Data Scientist, Metis

2. DATA
SCIENCE
POP UP
AUSTIN
#datapopupaustin
April 13, 2016
Galvanize, Austin Campus

4. PREDICTING CUSTOMERS
Vinny Senguttuvan
Senior Data Scientist
METIS

5. 5PREDICTING CUSTOMERS
What is the company/product?
High 5 Games
A facebook and mobile app
A suite of about 100 games
Over 10 million users

6. 6PREDICTING CUSTOMERS
What to Predict?
User-Product Interaction (A/B testing)
Popularity
Survival Analysis
Lifetime Value
Recommendation systems

7. 7PREDICTING CUSTOMERS
The main challenge:
Most users play for free
2% of the users contribute to all revenue
2% of those 2% are responsible for over half the entire revenue
Double singularity

8. A/B TESTING

9. 9A/B TESTING
Classic Hypothesis Testing
There is a current product design/feature: A
There is a suggested improvement or variation: B
We split the users into two groups and offer them A or B

10. 10A/B TESTING
Classic Hypothesis Testing
An Example hypothesis:
Moving the “buy credits” button from top-right to a more
prominent position in the mid-right of the screen and making it
bigger and brighter will lead to at least 20% more clicks.

11. 11A/B TESTING
Classic Hypothesis Testing
An Example hypothesis:
Moving the “buy credits” button from top-right to a more
prominent position in the mid-right of the screen and making it
bigger and brighter will lead to at least 20% more clicks.
It is important to have a minimum improvement requirement, since
there is a cost involved in making the change.

12. 12A/B TESTING
Classic Hypothesis Testing
Need enough data to:
• separate the probability density functions
• get the required sample size
• achieve statistical significance

13. 13A/B TESTING
Classic Hypothesis Testing
Need enough data to:
• separate the probability density functions
• get the required sample size
• achieve statistical significance
1. A long list of changes means not enough time per change

14. 14A/B TESTING
Classic Hypothesis Testing
Need enough data to:
• separate the probability density functions
• get the required sample size
• achieve statistical significance
1. A long list of changes means not enough time per change
2. Often the high monetizers are the primary test subjects, so the
double singularity is a big issue

15. 15A/B TESTING
Work around
When we don’t have enough samples, we could:
• we could bring down the confidence value
• we could bias on the side of one hypothesis
• we could assume independence between various A/B and run
them simultaneously or at least overlap them

16. 16A/B TESTING
Work around
When we don’t have enough samples, we could:
• we could bring down the confidence value
• we could bias on the side of one hypothesis
• we could assume independence between various A/B and run
them simultaneously or at least overlap them
We do all three.

17. 17A/B TESTING
Work around
When we don’t have enough samples:
• we bring down the confidence value
• we bias towards hypothesis A, since B has implementation cost
• we order the various A/B tests such that the dependences
between adjacent ones are minimal

18. POPULARITY METRICS

19. 19POPULARITY METRIC
Rank or classify products or games
• New products added continuously.
• Some locked and slowly unlocked.
• Others fully unlocked to all users.

20. 20POPULARITY METRIC
Rank or classify products or games
• New products added continuously.
• Some locked and slowly unlocked.
• Others fully unlocked to all users.
• They are released at vastly different conditions:
• one is game #15 when there are 2 million users
• another is game #77 when there are 10 million users

21. 21POPULARITY METRIC
Rank or classify products or games
Open problem
Also faced with music and other products

22. 22POPULARITY METRIC
Rank or classify products or games
Open problem
Also faced with music and other products
We whiteboarded various approaches including N dimensional
product spaces and harmonic distributions.
Either they were too complex to build or the data was too noisy.

23. 23POPULARITY METRIC
Rank or classify products or games
Went with two simple metrics:
1. An average-weighted-spin: Spins per game, over total spins,
times number of games unlocked, time log number of days
since the unlock of the game.
2. A Bayesian method (my colleague implemented this)

24. 24POPULARITY METRIC
Rank or classify products or games
Went with two simple metrics:
1. An average-weighted-spin: Spins per game, over total spins,
times number of games unlocked, time log number of days
since the unlock of the game.
2. A Bayesian method (my colleague implemented this)
Both performed well but had flaws. Used an intersection of the two.

25. SURVIVAL ANALYSIS

26. 26SURVIVAL ANALYSIS
Survival of users
Wanted to know:
1. How many users are still active after a given time since the
creation of their account
2. When is a user at risk of discontinuing to use the product

27. 27SURVIVAL ANALYSIS
Survival of users
Wanted to know:
1. How many users are still active after a given time since the
creation of their account
2. When is a user at risk of discontinuing to use the product
This problem had its beginnings in the medical industry but is
widely among internet services and products

28. 28SURVIVAL ANALYSIS
Survival of users
A lot can be learned by grouping users by cohorts and plotting
survival rates over time
We can also observe return rates of users after each additional day
of absence.
Cox survival model (my colleague worked on this)

29. 29SURVIVAL ANALYSIS
Survival of users
Our observations:
The survival rate of new users were significantly lesser than earlier
users and continues to drop
There was a clear day span, if a user was absent by more than that,
the chance of their return dropped significantly

30. 30SURVIVAL ANALYSIS
Survival of users
Our actions:
Factor in the decline in quality of the users while estimating
revenue. Also focus on acquisition channels that provided better
players
Reach out to users at risk of discontinuation with notifications and
offers before they hit the absence threshold

31. PREDICTIVE LIFETIME VALUE

32. 32PREDICTIVE LIFETIME VALUE
Lifetime Value
The total revenue from a specific user over entire time with product
Often defined as the revenue during the first 365 days of use
Very significant because of costs:
• 10% of the revenue spent on acquisition ads
• 30% revenue goes to app hosts (facebook/apple)

33. 33PREDICTIVE LIFETIME VALUE
Lifetime Value
The cost of ads are per new player acquired
While the revenue comes from a small subset of those players

34. 34PREDICTIVE LIFETIME VALUE
Lifetime Value
The cost of ads are per new player acquired
While the revenue comes from a small subset of those players
So we wanted to build an aggregated LTV for various acquisition
sources

35. 35PREDICTIVE LIFETIME VALUE
Lifetime Value
It’s a time series
Modeling that way would use all available information, but is
difficult

36. 36PREDICTIVE LIFETIME VALUE
Lifetime Value
It’s a time series
Modeling that way would use all available information, but is
difficult
Flipped it into a regression problem

37. 37PREDICTIVE LIFETIME VALUE
Regression Problem
Predict 365 day revenue based on user’s first month data.

38. 38PREDICTIVE LIFETIME VALUE
Regression Problem
Predict 365 day revenue based on user’s first month data.
Spend
Purchase count
Days of play
Player level achieved

39. 39PREDICTIVE LIFETIME VALUE
Regression Problem
Predict 365 day revenue based on user’s first month data.
Spend
Purchase count
Days of play
Player level achieved
Date joined (because of the decline in player quality)

40. 40PREDICTIVE LIFETIME VALUE
Regression Problem
Including the date joined as a feature makes it a sort of time series
Had to be careful in the setting up of the train and test data

41. 41PREDICTIVE LIFETIME VALUE
Regression Problem
Including the date joined as a feature makes it a sort of time series
Had to be careful in the setting up of the train and test data
Prediction rates were 80%, 90% and 95% after 1, 2 & 3 months
Surprising discoveries which made us change strategies

42. RECOMMENDATION SYSTEMS

44. 44RECOMMENDATION SYSTEMS
How does Pandora do it?
Specialists listened to every song and gave each of them various
attributes (about 30 to a hundred)
Two songs are similar if their distance in that n-dimensional space
is minimal
This works with a binary scale or a point system

46. 46RECOMMENDATION SYSTEMS
How does YouTube do it?
Algorithmic:
• Find all users who listened to the song
• Find the song that set of people had listened to the most
The actual solution is a little different but concept is the same

47. 47RECOMMENDATION SYSTEMS
Pandora:
• intrinsic recommendation system
• internal elements (features) of the songs are used to
measure similarity
YouTube:
• extrinsic recommendation system
• recommendations are solely based on peer preferences
• hence the term “collaborative filtering”

48. 48RECOMMENDATION SYSTEMS
DIEHARD BRAVEHEART MULAN CASABLANCA HOME ALONE
5
3
1
1
3
4
5
1
2
3
3
4
2
5
5
4
3
2
1
2
TOM
SALLY
NIKIL
KIM

49. 49RECOMMENDATION SYSTEMS
ACTION COMEDY HISTORY COMING-OF-AGE ROMANCE
5
3
1
1
5
5
1
4
1
3
5
4
1
2
1
4
1
5
1
3
TOM
SALLY
NIKIL
KIM
DIE HARD BRAVEHEART MULAN CASABLANCA HOME ALONE
5
2
1
2
1
5
1
5
2
4
4
2
4
4
4
2
2
4
2
5
3
4
1
4
1
ACTION
COMEDY
HISTORY
COMING-OF-AGE
ROMANCE

50. 50RECOMMENDATION SYSTEMS
svd: single value decomposition
You provide the MxN matrix and a K. And you get:
M x K (user-feature) matrix
K x N (feature-product) matrix

51. 51RECOMMENDATION SYSTEMS
Collaborative Filtering
User & Product Matrix can be decomposed:
(User & Features) and (Features & Product)

52. 52RECOMMENDATION SYSTEMS
Collaborative Filtering
User & Product Matrix can be decomposed:
(User & Features) and (Features & Product)
Then you can predict rating, find similar users or similar products.
AP News example: Articles and words

53. 53RECOMMENDATION SYSTEMS
Collaborative Filtering
How to choose “K”?

54. 54RECOMMENDATION SYSTEMS
Collaborative Filtering
How to choose “K”?
Could try various K, decompose, multiply and compare (RMSE) to
original matrix (cross-validation)
Or look at Eigen Values

55. 55RECOMMENDATION SYSTEMS
Collaborative Filtering
What if the ratings are not clearly defined?

56. 56RECOMMENDATION SYSTEMS
Collaborative Filtering
What if the ratings are not clearly defined?
Could use click, percentage of view, number of views, like, share

57. 57RECOMMENDATION SYSTEMS
Collaborative Filtering
For Missing values:
Use averages. Or sparse matrix solvers.
Normalization is important.
Outliers and noise get magnified.

58. 58RECOMMENDATION SYSTEMS
Collaborative Filtering
What if there is a progression or order to the content?

59. 59RECOMMENDATION SYSTEMS
Collaborative Filtering
What if there is a progression or order to the content?
Have to do some supervision.

60. 60RECOMMENDATION SYSTEMS
Intrinsic recommendation systems
Features are well defined
Don’t need user data to begin
Manual

61. 61RECOMMENDATION SYSTEMS
Extrinsic recommendation systems
Automatic
Features are undefined
Need a lot of user data
Not automatically adaptive to special cases (like order-specific
entries)

62. 62RECOMMENDATION SYSTEMS
Combined technique
Begin with the intrinsic knowledge
Better understanding of the content
Effective initial recommendation
Once enough data points are accumulated, switch to the extrinsic
model

63. 63RECOMMENDATION SYSTEMS
Combined technique
Double the work
Most effective, especially for timely content
This is what Netflix does… likely the best recommendation system

64. 64RECOMMENDATION SYSTEMS
What more, with recommender systems?

70. 70RECOMMENDATION SYSTEMS
Plus recommender systems can be more than
mere recommendation..

71. DATA
SCIENCE
POP UP
AUSTIN
@datapopup
#datapopupaustin