This is my talk given at eXelate's "big data eXposed 2015" event in Tel-Aviv. It touches the common problem of trying to optimize websites for better conversion rates, and covers the range of solutions - starting at classic A/B testing (unsuitable for rapidly changing content), going through Multi-Arm Bandits and finally arriving at Contextual Bandits - online machine learning algorithms allowing for true personalization in real-time.

2. Learn Like a Human – Taking Machine
Learning from Batch to Real-Time
Elad Rosenheim

3. Who am I
 Architect at Dynamic Yield,
“Predictors” Team Lead
 Previously:
 AlphaCSP
 SAP
 Performance & Scale, DevOps
 Measure All the Things!
 East-Asia & Japan

4. Who’s Dynamic Yield?
We’re optimizing & personalizing websites since 2011
 Start-up in Tel-Aviv
headed by Liad Agmon
 I Joined as 5th employee, we’re 50 now and growing fast

5. On the Agenda
Our clients’ problem
Old School Solutions
Meet the ML Bandits

6. Our clients’ problem
Publishers, retailers, SaaS
all share a common problem
They know their domain
but not how to optimize for each user
Screen real-estate is limited
yet everyone sees the same thing

7. What top videos to show on
NBC News’ site?
What user segments should see
this element at this location?
What’s the best layout for this
element?

8. Both the layout of this page and
each element in it deserve testing
What’s the best layout?
What types of products to show
whom?

9. What articles to show on
ynet’s homepage?
What titles and images?
In what order?

10. What is the best default sort order for products on Adika?
Does is significantly differ between user segments?

11. The Beginning
 First, there was the educated guess
 Then, there was the A/B test
 "Data Beats Opinion“
 Freedom to experiment (with nice tools)
 Hopefully: less fear of change, less politics
 How does it work?
 Split traffic between baseline and alternative variations
 In theory: sit & wait for significant results
 In practice: peek at the numbers till the nice “95% confidence”

12. A/B Tests: Already Old School?
While you wait, you're bleeding clicks
clicks == money
What about the really dynamic stuff?
Campaigns, Current Headlines, Products on Sale

13. Enter the Multi-Arm Bandits
 A Single-Arm Bandit
 Suppose I have multiple arms in front of me,
each with its unknown mean reward…
 How do I optimize income from multiple machines?
 Caution or Haste?
 Explore vs. Exploit
 In our context:
How do I optimize multiple variations?

14. Bandits - A Classic Problem
 (Very) Simple Solutions
 ε-greedy, ε–decreasing
 First 100% random explore, then ~90% exploit?
 Magic numbers, built-in revenue loss
 Bayesian-based approaches
 Smoother curve from explore to exploit
 “Winner” is now a less relevant term

15. Bandits work well when…
 We want to find the variation “best on average“
…but we’re not improving the conversion rate of any single variation
2.4% 1.7% 0.4%

16. Enter Personalization
 Each of us is a beautiful and unique feature vector!
 By showing the right variation to the right people,
we can improve conversions per variation
and beat the best variation
 ML Challenge Accepted

17. The Usual Suspects
Collaborative Filtering?
 Very big, very sparse matrix
 Cold Start
 Batch
 Not suitable in this case
Classifiers?
Logistic Regression, Random Forest et al.
 Periodically learn over all converters so far
 More data == more time, bigger model
 Not the classic question

18. What We Need
 Like a bandit, we need to learn as we go (not in batch),
but this time with “context” - the user’s data
 Incremental Learning over the stream of impressions & rewards
(“Partial Fit”)
 We’re looking to…
 Start learning from the first impression
 Handle the explore-exploit curve
 Run fast (enough)
 In the worst case: converge on the best variation, like a bandit

19. Meet the Contextual Bandits
 They “eat” the data stream
 They demand fast access to user data
 Historical or immediate
 Their model is always ready for action
 In the Papers
 Linear Bayes, LinUCB
 What we do: Per-Variation Logistic Regression
 A variant supporting updates in “mini-batches”
 Exploration-on-top
 Worst case: “Garbage In  Multi Arm Bandit Out”
 Light on memory, compact output

20.  Online should be fast & scale
 Offline: a testbed for iteratively testing new ideas
 New algorithms
 Tweaked parameters
 Feature transformations
How We Do It: Online & Offline

21. The Online Flow
DY Web Servers
a. get our script
b. log impressions,
conversions
Queue
Per Test
Learn
Workers
User
DB
Persist
ModelLoad to
Predict Server
Queue
Per Test
A B C
A B C
A B C
Predictions

22. The Offline Evaluator
 Test, Improve, Iterate
 Using real-world data
 Using generated data
 From easy to hard

23. Going Global
 Learn in the center site, fast predict in each geo. How?
 Push models via local Redis slaves
 Compressed SSH tunnel
 User data - daily aggregation
 Storage into LMDB (simple, fast memory-mapped K/V DB)
 Sync via S3 (LZ4 compressed), read from SSD
 Learn & Predict services
 Python as ML lingua franca: NumPy, SciPy, scikit-learn

24. Elad & Idan Say Goodbye
 Better data beats better algorithms
 Reduce aggressively
 Keep It Simple, Smart!
 Elad Rosenheim
 Idan Michaeli
 Read our blog
 Hiring? but of course! What’s with the Groundhog?