Proprietary + Confidential
UA focused on LTV
On steroids with Feature Selection
Przemyslaw Modrzewski, Principal Analytical Lead, Google
@modrzewski
Proprietary + ConfidentialProprietary + Confidential
We are facing a marketing revolution for…… last 20 years...
Clicks
Installs
ROAS / ROI
Profit
Customer Equity
Customer Lifetime Value
Proprietary + ConfidentialProprietary + Confidential
What LTV is after all?
Profits
Acquisition
Retention
Cross,up-selling
Costs
Customers
life-cycle
Proprietary + ConfidentialProprietary + Confidential
Problem with game players !
Proprietary + ConfidentialProprietary + Confidential
Reality…. (we all know that)
98%
to
analyze
intent
All In-App Users
Most Apps
have only
<2%
Paid Users
Paid Users
>60%
One-time
Purchasers
Big Whales<10%
Proprietary + ConfidentialProprietary + Confidential
Acquisition strategy - Options we have….
Paid Users
Will-Pay Users
Active Users
TOPLayer -- to Predict pLTV
● Goal KPI:ROI, to acquire more high value users
● Data Needed:All Paid User in-app behavior & payment data
MIDDLE Layer -- to Predict Conversion Probability
● Goal KPI:Payer Rate, to acquire more paid users
● Data Needed:Feature Selected Active Users in-app behavior data
Baseline -- Feature Selection
● Goal: Select what user behaviors are important to payments
● Data Needed:All Active Users in-app behavior data
Proprietary + ConfidentialProprietary + Confidential
Feature selection makes a difference
OBJECTIVE
A mobile advertiser
wants to identify the
best event for UAC
Action where IAP as
the optimization goal
does not work well
FEATURE SELECTION APPROACH
Run through all performance related
features and find the one(s) with the
both high relevance to IAP and high
occurrence itself
PREVIOUS APPROACH
Use install or IAP as UAC
optimization goal
IAP
Install
Proprietary + ConfidentialProprietary + Confidential
Events can extend “the pool” of users
Events in game
Whales
Conversions
Proprietary + Confidential
What makes Feature Selection
Proprietary + ConfidentialProprietary + Confidential
TECHNOLOGY
Use Cutting edge Machine
Learning (e.g.
ExtraRandomForest) to find
optimum feature for action
OUTCOME
Each feature gets an Importance, Frequency and Correlation
scores.
Selected features will need a high score on these 4 criterions.
IMPLEMENTATION
Directly leverage the in-app events in
AdWords
Importance:
Unique User Frequency:
Active User Correlation:
Correlation:
PRIMARY criterion for sorting the list. The higher the number, the
more important the feature is for user classification
Number of unique users triggered this feature/action
Criterion measuring how likely increasing the frequency of
actions can make a user convert to paid user
Criterion measuring how likely a user triggered a
certain action will be converting to a paid user
Feature Selection System
Proprietary + Confidential
Some games is community-like
where valuable users are not only
who made in-app-purchase but
also non-paid but highly active.
Select the most suitable
features used in UAC to
enhance the performance of
UAC campaign
Feature Selection - what can do with it?
Proprietary + Confidential
Payment Amount
above threshold*
(Y/N)
Paid user
(Y/N)
Pay_Count
above threshold*
(Y/N)
Active Days
above threshold*
(Y/N)
Login Times
above threshold*
(Y/N)
Online Time
above threshold*
(Y/N)
*threshold is determined by clients*threshold is determined by clients
ROI/PURCHASE OBJECTIVE RETENTION OBJECTIVE
Different objectives of conversion
Proprietary + Confidential
Segment markets into 4 tiers by
Quadrant Analysis with:
● Install to In-app-Purchase Conversion Rate
● Conversion Lag in Days
X-axis, Install to In-app-Purchase Conversion Rate, Y-axis, Conversion Lag in Days, Each dot represents a
market, Quadrant Segment Threshold, average statics of x and y axis
Pay Rate
ConversionLag
average
average
IAP event is very often difficult to recognize by UA systems
Proprietary + ConfidentialProprietary + Confidential
We have to remember about an important criteria for
events in UAC
DELAY
No later than 7 days post
install
Ideally D2/3
The shorter the better
RELEVANCY
Has to be deterministic
and correlated with the
business objective
CONVERSION VOLUME
Min. 20 unique user
converted per day
Ideally 50+
The higher the better
Purchase
Registration
Add to cart
Proprietary + Confidential
Identify different proxy events, with Feature Selection, for different markets that are
on the path to purchase
T3/T4 Markets: combine the
upper funnel events
● Android: Buy Item
● iOS: Group and clan
message
All Markets:
● IAP
T1/T2 Markets: lower funnel
event
● IAP
● Start payment
Results
T2 IAP -> BuyItem*
AU/CA/SG
Android
NZ/CH IOS
Tier Optimize
T2
T3
25%
7 Days’ ROI
6X
2X
AU/CA/SG
IOS
None -> Group & Clan
Message
IAP -> Group & Clan
Message *
Approach
* BuyItem : Buy items i.e. weapon, skins, etc. with gaming currency *Group & Clan Message : How many messages sent in group or clan
We can go wider (upper-funnel) but still be careful
Proprietary + Confidential
UAC for in-app actions with Feature Selection (3 events)
Campaign 1 for IAP
Campaign 2 for IAP
Campaign 3 for IAP
Campaign 4 for IAP
Campaign 5 for IAP
Campaign 1 for FS events
We can acquire 35 users (ave.) instead of one.
Proprietary + Confidential
We can be flexible in picking events if FS shows significance
Proprietary + Confidential
UAC for in-app actions with Feature Selection
Proprietary + Confidential
Events are 250 cheaper then IAP…… ;)
Event 1
Event 2
Event 3
Event 4
Event 5
Proprietary + Confidential
Do we have a correlation in event and revenue?
Proprietary + Confidential
Be careful with counting profitability with events
Proprietary + Confidential
Last but not least
Deep but less Volume
Condition A AND Condition B
1. Use “AND” to combine
features only if no single
feature is insightful and
you don’t want to explore
new features
2. Use “AND” when you are
ROI driven
3. KPI: ROI, CPA, CVR to pay
Shadow but more
Volume
Condition A OR Condition B
1. Use “OR” to combine
features only if multiple
single features are insightful
2. Use “OR” when you are
Ecosystem driven
3. KPI: ROI, Retention Rate,
Cost per Retention
Proprietary + ConfidentialProprietary + Confidential
Thanks to Feature Selection we can “swim in bigger pool” ;)
98%
to
analyze
intent
All In-App Users
Most Apps
have only
<2%
Paid Users
Paid Users
>60%
One-time
Purchasers
Big Whales<10%
Proprietary + Confidential
Feature Selection is easy in GCP :)
Cloud AutoML
Model is now trained and ready to make predictions
This model can scale as needed to adapt to customer demands
Upload labeled structured
data
Train your model Evaluate
(Classification/Regression)
Class A
Class B
Class C
Class D
Class E
ID Feature Feature ... Label
...
...
ID Feature Feature ...
...
...
Proprietary + Confidential
Thank You Thank
You
Thank
You
Contact: modrzewski@google.com

User Acquisition focused on LTV on steroids

  • 1.
    Proprietary + Confidential UAfocused on LTV On steroids with Feature Selection Przemyslaw Modrzewski, Principal Analytical Lead, Google @modrzewski
  • 2.
    Proprietary + ConfidentialProprietary+ Confidential We are facing a marketing revolution for…… last 20 years... Clicks Installs ROAS / ROI Profit Customer Equity Customer Lifetime Value
  • 3.
    Proprietary + ConfidentialProprietary+ Confidential What LTV is after all? Profits Acquisition Retention Cross,up-selling Costs Customers life-cycle
  • 4.
    Proprietary + ConfidentialProprietary+ Confidential Problem with game players !
  • 5.
    Proprietary + ConfidentialProprietary+ Confidential Reality…. (we all know that) 98% to analyze intent All In-App Users Most Apps have only <2% Paid Users Paid Users >60% One-time Purchasers Big Whales<10%
  • 6.
    Proprietary + ConfidentialProprietary+ Confidential Acquisition strategy - Options we have…. Paid Users Will-Pay Users Active Users TOPLayer -- to Predict pLTV ● Goal KPI:ROI, to acquire more high value users ● Data Needed:All Paid User in-app behavior & payment data MIDDLE Layer -- to Predict Conversion Probability ● Goal KPI:Payer Rate, to acquire more paid users ● Data Needed:Feature Selected Active Users in-app behavior data Baseline -- Feature Selection ● Goal: Select what user behaviors are important to payments ● Data Needed:All Active Users in-app behavior data
  • 7.
    Proprietary + ConfidentialProprietary+ Confidential Feature selection makes a difference OBJECTIVE A mobile advertiser wants to identify the best event for UAC Action where IAP as the optimization goal does not work well FEATURE SELECTION APPROACH Run through all performance related features and find the one(s) with the both high relevance to IAP and high occurrence itself PREVIOUS APPROACH Use install or IAP as UAC optimization goal IAP Install
  • 8.
    Proprietary + ConfidentialProprietary+ Confidential Events can extend “the pool” of users Events in game Whales Conversions
  • 9.
    Proprietary + Confidential Whatmakes Feature Selection
  • 10.
    Proprietary + ConfidentialProprietary+ Confidential TECHNOLOGY Use Cutting edge Machine Learning (e.g. ExtraRandomForest) to find optimum feature for action OUTCOME Each feature gets an Importance, Frequency and Correlation scores. Selected features will need a high score on these 4 criterions. IMPLEMENTATION Directly leverage the in-app events in AdWords Importance: Unique User Frequency: Active User Correlation: Correlation: PRIMARY criterion for sorting the list. The higher the number, the more important the feature is for user classification Number of unique users triggered this feature/action Criterion measuring how likely increasing the frequency of actions can make a user convert to paid user Criterion measuring how likely a user triggered a certain action will be converting to a paid user Feature Selection System
  • 11.
    Proprietary + Confidential Somegames is community-like where valuable users are not only who made in-app-purchase but also non-paid but highly active. Select the most suitable features used in UAC to enhance the performance of UAC campaign Feature Selection - what can do with it?
  • 12.
    Proprietary + Confidential PaymentAmount above threshold* (Y/N) Paid user (Y/N) Pay_Count above threshold* (Y/N) Active Days above threshold* (Y/N) Login Times above threshold* (Y/N) Online Time above threshold* (Y/N) *threshold is determined by clients*threshold is determined by clients ROI/PURCHASE OBJECTIVE RETENTION OBJECTIVE Different objectives of conversion
  • 13.
    Proprietary + Confidential Segmentmarkets into 4 tiers by Quadrant Analysis with: ● Install to In-app-Purchase Conversion Rate ● Conversion Lag in Days X-axis, Install to In-app-Purchase Conversion Rate, Y-axis, Conversion Lag in Days, Each dot represents a market, Quadrant Segment Threshold, average statics of x and y axis Pay Rate ConversionLag average average IAP event is very often difficult to recognize by UA systems
  • 14.
    Proprietary + ConfidentialProprietary+ Confidential We have to remember about an important criteria for events in UAC DELAY No later than 7 days post install Ideally D2/3 The shorter the better RELEVANCY Has to be deterministic and correlated with the business objective CONVERSION VOLUME Min. 20 unique user converted per day Ideally 50+ The higher the better Purchase Registration Add to cart
  • 15.
    Proprietary + Confidential Identifydifferent proxy events, with Feature Selection, for different markets that are on the path to purchase T3/T4 Markets: combine the upper funnel events ● Android: Buy Item ● iOS: Group and clan message All Markets: ● IAP T1/T2 Markets: lower funnel event ● IAP ● Start payment Results T2 IAP -> BuyItem* AU/CA/SG Android NZ/CH IOS Tier Optimize T2 T3 25% 7 Days’ ROI 6X 2X AU/CA/SG IOS None -> Group & Clan Message IAP -> Group & Clan Message * Approach * BuyItem : Buy items i.e. weapon, skins, etc. with gaming currency *Group & Clan Message : How many messages sent in group or clan We can go wider (upper-funnel) but still be careful
  • 16.
    Proprietary + Confidential UACfor in-app actions with Feature Selection (3 events) Campaign 1 for IAP Campaign 2 for IAP Campaign 3 for IAP Campaign 4 for IAP Campaign 5 for IAP Campaign 1 for FS events We can acquire 35 users (ave.) instead of one.
  • 17.
    Proprietary + Confidential Wecan be flexible in picking events if FS shows significance
  • 18.
    Proprietary + Confidential UACfor in-app actions with Feature Selection
  • 19.
    Proprietary + Confidential Eventsare 250 cheaper then IAP…… ;) Event 1 Event 2 Event 3 Event 4 Event 5
  • 20.
    Proprietary + Confidential Dowe have a correlation in event and revenue?
  • 21.
    Proprietary + Confidential Becareful with counting profitability with events
  • 22.
    Proprietary + Confidential Lastbut not least Deep but less Volume Condition A AND Condition B 1. Use “AND” to combine features only if no single feature is insightful and you don’t want to explore new features 2. Use “AND” when you are ROI driven 3. KPI: ROI, CPA, CVR to pay Shadow but more Volume Condition A OR Condition B 1. Use “OR” to combine features only if multiple single features are insightful 2. Use “OR” when you are Ecosystem driven 3. KPI: ROI, Retention Rate, Cost per Retention
  • 23.
    Proprietary + ConfidentialProprietary+ Confidential Thanks to Feature Selection we can “swim in bigger pool” ;) 98% to analyze intent All In-App Users Most Apps have only <2% Paid Users Paid Users >60% One-time Purchasers Big Whales<10%
  • 24.
    Proprietary + Confidential FeatureSelection is easy in GCP :) Cloud AutoML Model is now trained and ready to make predictions This model can scale as needed to adapt to customer demands Upload labeled structured data Train your model Evaluate (Classification/Regression) Class A Class B Class C Class D Class E ID Feature Feature ... Label ... ... ID Feature Feature ... ... ...
  • 25.
    Proprietary + Confidential ThankYou Thank You Thank You Contact: modrzewski@google.com