Description of StackAdapt's machine learning pipeline for presentation at University of Toronto School of Continuing Studies

1. Presentation for School of Continuing Studies

2. Data Science / Engineering
Section I: Advertising Technology Landscape

3. About Me
- Name: Larkin Liu
- Role: Data Scientist @ StackAdapt since 2016
- Specialties: Apache Spark, Scala, Python, R
- Education: MASc in Industrial Engineering, Specializing
in Operations Research, University of Toronto
- Other Fun Facts:
- Chinese / Canadian
- Competitive MMA fighter, and kickboxer.
- I really like race cars.

4. What I do ?

5. Agenda
Increase Profitability of Campaigns
- Ad Tech Landscape
- ML Models
- Logistic Regression
- Bagging: Random Forest
- Boosting: Adaboost (Gradient Boosted Trees, xgboost)
- Survival Regression (Proportional Hazards, Accelerated Failure Time Model)
- (Natural Language Processing)
- AB Testing
- RTB Auction Strategy

6. Real Time Bidding
- Online advertising goes through a process
known as Real Time Bidding (RTB)
- StackAdapt is a Demand Side Platform
(DSP).
- DSP’s interface with clients running
advertising campaigns, facing the Ad
Exchange.
- Our objective is to win valuable ad
impressions for our client’a campaigns.

7. Overview (Objectives)
- The ad exchange is a second price
auction.
- We bid on advertisements that are
valuable to our client.
- To accomplish we predict the likelihood
of a defined conversion, based on ML
modelling.
- We set our bid price proportional to our
predicted probability of a conversion.

8. Key Terms
- KPI - Key Performance Metric
- Win Price - the win price of the advertisement on the ad exchange, actual cost.
- Bid Price - what the DSP bid for the advertisement.
- CPM - Cost per Mille, cost per 1000 impressions.
- eCPC - Effective cost per click (total cost/number of clicks)
- eCPE - Effective cost per engagement (total cost/number of engagements)
- eCPA - Effective cost per action (total cost/number of conversions)
- AB Testing - split testing algorithms between control group and treatment algorithm.

9. Expectation
Initially we believed that each optimizer we designed will have a desired effect on the intermediate KPI’s
(CTR, eCPC, eCPE, eCPA, etc.), which in turn affect the overall profit of each campaign.

10. Reality
In reality, we discovered that the effect of each optimizer on various intermediate KPI’s follow a more
complex interaction scheme, which is also dependent on the market dynamics.

11. Data Science / Engineering
Section II: ML Models

12. Logistic Regression
Logistic Regression
We interpret the probability of p
i
provided predictor variables x
0,i
, x
1,i
, ..., x
m,i
.
Univariate logistic regression model F(x)
Can be re-written as, interpreted as the Odds ratio, where
F(x) is interpreted as probability of response = 1 (p)

13. Logistic Regression with
Interaction (IX) Terms
- Basic logistic regression makes a key assumption that all observations are independent of one
another. This is not the case in our data set.
- Interaction terms take into account the interaction between variables. For example, where variables X
and Z may not be independent, and the interaction between X and Z produce an effect on the log
odds.
- When deploying logistic regression for prediction of key KPI’s, the addition of interaction terms crucial
for accurate prediction, as variables are not independent, and the interaction between variables may
have a key effect in predicting KPI’s.

14. AdaBoost
- Adaptive Boosting (AdaBoost) is a well-established boosting algorithm.
- Unlike bagging, it produces a linear combination of tree results.
- Each weak classifier is trained on the entire dataset.
- Misclassified results are accentuated, and correctly classified results are diminished, depending on
each of the weak classifier results.
- The result is a linear combination of weak classifiers.
- Boosting can resolve the inherent capabilities of a specific class of classifiers, as well as reduce
class imbalance.

15. AdaBoost
Illustrative example of boosting.

16. AdaBoost Algorithm

17. Random Forest
- Random Forest (Breiman 2001) is a very
established bagging classification
algorithm allowing us to perform
classification and regression.
- An extension of the decision tree algorithm,
RF combines a random sampling of the
data, sample of the features, and sample of
the in and combines the result of many
small weak predictors.
- This approach makes RF much more
robust. Preventing overfitting and bias.

18. Survival Regression
- Proportional Hazards Model
- Accelerated Failure Time Model
- Models were evaluated using Akaike Information
Criterion (AIC), and Root Mean Square Error (RMSE).
- Primarily used to measure the time it takes for users to remain on a site (time on site). The
longer a user remains on a site, the lower the probability.

19. Survival Modelling

20. Survival Modelling
- We used a Random Forest model. Parameters,
- m: 33% of Total No. of Features
- No. Trees: 100
- Max Depth: 10 Layers
- Average RMSE across 10-fold cross validation
of 145. (A 25% Improvement from the Survival
Models investigated earlier).

21. Data Science / Engineering
Section III: RTB Deployment

22. AB Testing
- Currently our tests run 50/50 splits (S = 0.5), 50% goes to A group (control) 50% goes to B group
(experimental treatment).
- Our goal is to maximize profit, and minimize eCPC, something which we can achieve by deploying
our ML models.
- However, the effect of any model on any specific campaign can vary.

23. EMR-AB13-IX-5day-dailyUpdate
- Experimental Model Avg eCPC: 0.819
- Control Group eCPC: 0.833
- Experimental Model Profit: 2457.11
- Control Group Profit: 2773.32

24. EMR-AB14-mean_encoded_logisti
c_regression
- Experimental Model Avg eCPC: 1.246
- Control Group eCPC: 0.675
- Experimental Model Profit: 2285.23
- Control Group Profit: 1086.80

25. But wait
- Models perform differently with regards to various KPI’s and models on a
campaign-specific basis….
- Solution: a larger proportion of bid requests should go to the model with better KPI
performance.

26. RTB Optimizer
Our Min*/Max* framework is based off of a PID controller, where we adjust the split (S) proportional to
our objective of attaining a minimum or maximum value.
- Proportional: Immediate Error
- Integral: Cumulative Error
- Derivative: Rate of Change

27. References
- Zhang, Weinan, “Optimal Real-Time Bidding for Display Advertising”, 2016
- Freund & Schapire, “Experiments with a New Boosting Algorithm”, 1996

28. QUESTIONS