Dynamic Creative (Non Contextual)
•
1 like•122 views
A presentation on the Bayesian Bandit algorithm me and my team developed at Adform :-)
Recommended
Recommended
More Related Content
Similar to Dynamic Creative (Non Contextual)
Similar to Dynamic Creative (Non Contextual) (20)
Recently uploaded
Recently uploaded (20)
Dynamic Creative (Non Contextual)
- 1. Non-contextual Dynamic Creative August 2018 Morten Arngren Lead Data Scientist Vlad Sandulescu Senior Data Scientist Jan Kremer Data Scientist
- 2. Simulations Finding optimal model parameters ExcludedIntroduction Quick recap from last time 📈 Real world results 🌎
- 5. Weight-based
- 6. 🚀 🌴 🎧 Normalised CTR 🍷 weights are calculated as normalised CTR values for each version samples are drawn from this multinomial distribution at serving time Weight based serving 🚀 The chosen one… Sampling (weighted) ServeWeight-based
- 7. in the initial impressions, the weights are all the same to learn the true distribution afterwards the weights are updated frequently Weight based serving Learning Initial impressions time 🚀 🌴 🎧🍷 🚀 🌴 🎧🍷 +1 period 🚀 🌴 🎧🍷 Campaign start
- 8. Reserve Ratio part of the weights are reserved for equal uniform random weighting. Weight based parameters 🚀 🌴 🎧 Reserve ratio 🍷 🚀 🌴 🎧🍷 Sensitivity increases the weight for the version with the highest weight eventually into a “winner-takes-all”
- 9. Dynamic Creative Recency time -5 -4 -2-3 -1 0… 1 hour 🚀 🌴 🎧🍷 Weights 💻 CTR weighted CVR weighted the impressions, clicks and conversion are all weighted by a decaying function too limit the influence of historical data Recency 💻 Impressions Clicks Conversions weighted counts are used to calculate the final weights
- 10. Dynamic Creative Recency the impressions, clicks and conversion are all weighted by a decaying function too limit the influence of historical data Recency basic form, where represents decay after 24 hours final form time -5 -4 -2-3 -1 0… 1 hour 💻 Impressions Clicks Conversions decay function Example
- 11. Recency Auto Calculation time -5 -4 -2-3 -1 0… auto calculating the recency becomes estimating the right value Recency 💻 Impressions this is a balance between adapting to the environment by limiting older data and still having enough data to estimate the model parameters adaption sufficient data 🔋⌖ Example
- 12. Recency Auto Calculation time -5 -4 -2-3 -1 0… the decay parameter is calculated from the time it took to reach 10k raw impressions - looking backwards in time Recency 💻 Impressions 5k imprs. [] hours time it takes to reach threshold of 10k impressions in the first hour only 5k impressions are reached and the decay parameter is set to its default value. upper bound on decay parameter Example
- 13. Recency Auto Calculation time -5 -4 -2-3 -1 0… the decay parameter is calculated from the time it took to reach 10k raw impressions - looking backwards in time Recency 💻 Impressions 9k imprs. [] hours in the first two hours only 9k impressions are reached and the decay parameter is set to its default value. upper bound on decay parameter time it takes to reach threshold of 10k impressions Example
- 14. Recency Auto Calculation time -5 -4 -2-3 -1 0… the decay parameter is calculated from the time it took to reach 10k raw impressions - looking backwards in time Recency 💻 Impressions 10k imprs. 3 hours final form time it takes to reach threshold of 10k impressions target recency calculate the parameter ensuring 10k impressions back are attenuated with 80% Example
- 15. Limitations Impressions threshold wasting ~50k impression before we can go optimal. Cold start learn from existing version by taking the average and inject new version. Version B Version A Injected avg. traffic Version added Own organic traffic TimeAd created Missing period 🎧 🚀 🌴 🚀 🌴 🎧🍷
- 16. Architecture v1.0 Scheduled Training Serving-Team Weighted sampling Model parameters Dyn. Ads. Data Pre-filtering Calc. parameters Model ready Request Serving Kafka Serve parameters Impression 🚀 low-latency server hyper parameters 🚀 🌴 🎧🍷 Kafka
- 17. 🚀 🌴 🍷 Bayesian Beta Bandits
- 18. Bayesian Beta Bandits ? ? Versions inside 🚀 🌴 🍷 Dynamic Creative CTR p(click) 🚀 🍷 🌴 🍷 The chosen one… 3x uniform probability distributions 💻 Impressions Clicks Zero impressions randomrandomrandom 🍷 Update (<1 hour) Thompson Sampling
- 19. ? ? Versions inside 🚀 🌴 🍷 CTR p(click) 🚀 🍷 Few impressions Uniform probability distribution 🌴 random random random Bayesian Beta Bandits Thompson Sampling 2 Beta distributions 🚀 The chosen one… 💻 Impressions Clicks 🚀 Update (<1 hour) Dynamic Creative
- 20. ? ? Versions inside 🚀 🌴 🍷 CTR p(click) 🍷 More impressions 🌴 🚀 random random random Bayesian Beta Bandits Thompson Sampling 🌴 The chosen one… 💻 Impressions Clicks 🌴 Update (<1 hour) Dynamic Creative
- 21. ? ? Versions inside 🚀 🌴 🍷 CTR p(click) 🍷 More impressions 🌴 🚀 random random random Bayesian Beta Bandits Thompson Sampling 🚀 The chosen one… 💻 Impressions Clicks 🚀 Update (<1 hour) Dynamic Creative
- 22. ? ? Versions inside 🚀 🌴 🍷 🎧 CTR p(click) 🍷 Cold start 🌴 🎧 🚀 random random randomrandom Bayesian Beta Bandits Thompson Sampling 🎧 The chosen one… 💻 Impressions Clicks 🎧 Update (<1 hour) Dynamic Creative
- 24. ? ? Versions inside 🚀 🌴 🍷 CTR p(click) 🍷 Recency 🚀 randomrandom Bayesian Beta Bandits Thompson Sampling 🍷 The chosen one… 💻 Impressions Clicks Update (<1 hour) random Recency 🌴 🍷 Dynamic Creative
- 25. Evaluation 📈
- 26. environment impression agent recencysensitivity impr. threshold 🍷 🚀 randomrandom random thompson sampling 🍷 The chosen one… click? recency target noise true ctr click probability cold start 🌴 ctr 0.05 0.10 0.15 🚀 🌴 🍷 0.002 0.20 🎧 🍼
- 28. stable environment 0.05 0.10 0.15 🚀 🌴 🍷 noise environment 0.05 0.10 0.15 🚀 🌴 🍷 0.05 0.10 0.15 🚀 🌴 🍷 cold start environment 0.002 0.20 🎧 🍼 adding at every period𝒩(0,0.1) ctr = ctr + 𝒩(0,0.1) nothing happens…. ….ZZZZzzzzz adding two additional version in the middle of the experiment
- 29. regret - average metric ρ = Tμ* − T ∑ t=1 ̂rt μ* = max k (μk) (max_reward_mean - n_optimal_selections_b) / imp_per_period Average number of times the optimal ad was not selected.in words in math in code lower is better
- 30. regret - auc metric Cumulative DistributionRegret Distribution higher is better in graphs in words Area under the curve of the cumulative distribution of the regret. auc
- 31. 10k imp. threshold 2k 1k recency target 0.8 0.1 0.5 0.2 I T algo parameters 0.95 0.8 1 100 auto 10 0.01 sensitivity S R recency 0.1 0.2 50 80
- 32. algo parameters Find top optimal parameters for all environments One set to rule them all….
- 34. auc: 0.91 / 1.00 stable environment recency sensitivity 10 flat regret - learning fast weights/ parameters at latest iteration almost perfect metrics 0.95
- 35. stable environment 0.95 recency sensitivity 101-100 auc sensitivity adapts ‘perfectly’ Regret_w Regret_b
- 36. stable environment recency sensitivity 100 Many peaks of high regret…. …exploring latest iteration exploring has a cost 0.1 auc: 0.93 / 0.97
- 37. stable environment 0.1 recency sensitivity 101-100 forgets to fast to learn efficiently. Regret_w Regret_b explores too many times. auc sensitivity
- 40. noise environment 0.95 recency sensitivity 10 bandit trying to adapt still one clear optimal version bandit still better auc: 0.67 / 0.73
- 41. noise environment 0.95 recency sensitivity 10 bandit can not decide similar CTR values. Beta dist. overlapping bandit still better auc: 0.63 / 0.70
- 43. noise environment 0.95 recency sensitivity 101-100 high regrets no clear best results Regret_w Regret_b auc sensitivity
- 44. noise environment 0.1 recency sensitivity 101-100 lower regrets sensitivity > 10 Regret_w Regret_b auc sensitivity
- 45. noise environment recency sensitivity 100 bandit adapting well-defined bandit again… auc: 0.66 / 0.71 auto
- 46. noise environment auto recency sensitivity 101-100 Very high regrets! Regret_w Regret_b auc sensitivity imp. threshold recency target 10000 0.8
- 47. noise environment auto - 0.95 recency sensitivity 100 auto recency made it worse…! Regret_w Regret_b imp. threshold recency target 10000 0.8 auto recency 0.01 + 0.95 recency
- 48. Static environment sensitivity 100 0.1 recency Noise regret imp. threshold auto recencyrecency target
- 50. cold start environment auto recency sensitivity 100 bandit adapting new optimal version found bandit rules… auc: 0.87 / 0.90
- 51. cold start environment auto recency sensitivity 100 Looks good Regret_w Regret_b auc sensitivity imp. threshold recency target 1000 0.2
- 52. 10k imp. threshold 2k 1k recency target 0.8 0.1 0.5 0.2 I T algo optimal parameters 0.95 0.8 1 100 auto 10 0.01 sensitivity S R recency 0.1 0.2 50 80
- 53. Architecture v2.0 Scheduled Training Serving-Team Thompson sampling Model parameters Dyn. Ads. Data Pre-filtering Calc. parameters Models ready Request Serving Kafka Serve parameters Impression 🚀 low-latency server hyper parameters Kafka sample from all beta distributions and select the version with the highest probability.
- 54. Architecture v2.0 Scheduled Training Serving-Team Weighted sampling Model parameters Dyn. Ads. Data Pre-filtering Calc. parameters Request Serving Kafka Serve parameters Impression 🚀 low-latency server hyper parameters Kafka sample from all beta distributions and select the version with the highest probability. Convert to weights Models ready