Presenting our experiences in using Stan for productizing Bayesian revenue estimation at Helsinki analytics meetup. Based on our recent blog post.

1. Bayesian Revenue
Estimation with Stan
Save time, work smarter, improve results
Smartly - Oct 2017
Markus Ojala
https://www.smartly.io/blog/tutorial-how-we-prod
uctized-bayesian-revenue-estimation-with-stan

2. 1 billion spend
Managed yearly
Smartly.io
Facebook &
Instagram Partner

4. Use case: campaign budget allocation

5. Multi-Armed
Bandit

6. Bayesian Bandits / Thompson sampling
The number of pulls for a given lever should match
its actual probability of being the optimal lever
Sample from the posterior for the mean of each lever

7. Modeling results per spend

8. Separate revenue model into two parts
ROAS = revenue / cost
= revenue / conversions * conversions / cost
= revenue / conversions * 1 / CPA
ROAS = return on ad spend
CPA = cost per action
Existing model
● Lot of data
● Varies fast
● Big differences
New model
● Little data
● Varies slowly
● Small real differences
● Lot of random variation

9. Modeling the revenue per purchase
Revenue follows usually a long-tailed distribution, use log-normal

10. We observe only aggregates
Goal: estimate log-normal parameters for ad sets
Challenge: observation i is aggregate of multiple events ni
Solution: Estimate by another log-normal

11. Simple
Stan model

12. Multilevel
Model
Account
Campaign
Ad set

13. ● Share information between ad sets and campaigns
● Ad set level means are expected to be close to campaign level mean etc
● Campaigns have mean and scale, as well as the account
Multilevel model

14. Multilevel model in Stan

15. Posterior predictive checks

16. Results 1
Common case:
ad sets don’t
really differ

17. Results 2
Campaigns
differ, ad sets
not

18. Results 3
Lot of data
Ad sets differ

19. ● Easy way to write Bayesian models and do inference
● Stan automates sampling from the posterior (MCMC, HMC, NUTS)
● Sampling can be slow
● We use Stan’s variational inference ADVI that approximates posterior
● Stan Python interface PyStan is not widely adopted, some glitches in usage
● Alternatives: PyMC3, Edward
Experiences in Stan

20. Thank you.
Markus.Ojala@smartly.io