Devoxx Real-time Learning

3,423 views

Published on

An expanded description of real-time learning including system designs that I presented at Devox France in March 2013

Published in: Technology
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,423
On SlideShare
0
From Embeds
0
Number of Embeds
2,003
Actions
Shares
0
Downloads
46
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Devoxx Real-time Learning

  1. 1. Real-time Learning©MapR Technologies - Confidential 1
  2. 2. whoami – Ted Dunning Chief Application Architect, MapR Technologies Committer, member, Apache Software Foundation – particularly Mahout, Zookeeper and Drill (we’re hiring) Contact me at tdunning@maprtech.com tdunning@apache.com ted.dunning@gmail.com @ted_dunning©MapR Technologies - Confidential 2
  3. 3.  Slides and such (available late tonight): – http://www.mapr.com/company/events/devoxx-3-29-2013 Hash tags: #mapr #devoxxfr©MapR Technologies - Confidential 3
  4. 4. Agenda What is real-time learning? A sample problem Philosophy, statistics and the nature of the knowledge A solution System design©MapR Technologies - Confidential 4
  5. 5. What is Real-time Learning? Training data arrives one record at a time The system improves a mathematical model based on a small amount of training data We retain at most a fixed amount of state Each learning step takes O(1) time and memory©MapR Technologies - Confidential 5
  6. 6. We have a product to sell … from a web-site©MapR Technologies - Confidential 6
  7. 7. What tag- What line? picture? Bogus Dog Food is the Best! Now available in handy 1 ton bags! Buy 5! What call to action?©MapR Technologies - Confidential 7
  8. 8. The Challenge Design decisions affect probability of success – Cheesy web-sites don’t even sell cheese The best designers do better when allowed to fail – Exploration juices creativity But failing is expensive – If only because we could have succeeded – But also because offending or disappointing customers is bad©MapR Technologies - Confidential 8
  9. 9. A Quick Diversion You see a coin – What is the probability of heads? – Could it be larger or smaller than that? I flip the coin and while it is in the air ask again I catch the coin and ask again I look at the coin (and you don’t) and ask again Why does the answer change? – And did it ever have a single value?©MapR Technologies - Confidential 9
  10. 10. A Philosophical Conclusion Probability as expressed by humans is subjective and depends on information and experience©MapR Technologies - Confidential 10
  11. 11. So now you understand Bayesian probability©MapR Technologies - Confidential 11
  12. 12. Another Quick Diversion Let’s play a shell game This is a special shell game It costs you nothing to play The pea has constant probability of being under each shell (trust me) How do you find the best shell? How do you find it while maximizing the number of wins?©MapR Technologies - Confidential 12
  13. 13. Pause for short con-game©MapR Technologies - Confidential 13
  14. 14. Conclusions Can you identify winners or losers without trying them out? No Can you ever completely eliminate a shell with a bad streak? No Should you keep trying apparent losers? Yes, but at a decreasing rate©MapR Technologies - Confidential 14
  15. 15. So now you understand multi-armed bandits©MapR Technologies - Confidential 15
  16. 16. Is there an optimum strategy?©MapR Technologies - Confidential 16
  17. 17. Thompson Sampling Select each shell according to the probability that it is the best Probability that it is the best can be computed using posterior é ùP(i is best) = ò I êE[ri | q ] = max E[rj | q ]ú P(q | D) dq ë j û But I promised a simple answer©MapR Technologies - Confidential 17
  18. 18. Thompson Sampling – Take 2 Sample θ q ~ P(q | D) Pick i to maximize reward i = argmax E[r | q ] j Record result from using i©MapR Technologies - Confidential 18
  19. 19. Nearly Forgotten until Recently Citations for Thompson sampling©MapR Technologies - Confidential 19
  20. 20. Bayesian Bandit for the Shells Compute distributions based on data so far Sample p1, p2 and p3 from these distributions Pick shell i where i = argmaxi pi Lemma 1: The probability of picking shell i will match the probability it is the best shell Lemma 2: This is as good as it gets©MapR Technologies - Confidential 20
  21. 21. And it works! 0.12 0.11 0.1 0.09 0.08 0.07 regret 0.06 ε- greedy, ε = 0.05 0.05 0.04 Bayesian Bandit with Gam m a- Norm al 0.03 0.02 0.01 0 0 100 200 300 400 500 600 700 800 900 1000 1100 n©MapR Technologies - Confidential 21
  22. 22. Video Demo©MapR Technologies - Confidential 22
  23. 23. The Basic Idea We can encode a distribution by sampling Sampling allows unification of exploration and exploitation Can be extended to more general response models©MapR Technologies - Confidential 23
  24. 24. The Original Problem x2 x1 Bogus Dog Food is the Best! Now available in handy 1 ton bags! Buy 5! x3©MapR Technologies - Confidential 24
  25. 25. Mathematical Statement Logistic or probit regression P(conversion) = w (å x q ) i ij 1 w(x) = 1+ e- x erf(x) +1 w(x) = 2©MapR Technologies - Confidential 25
  26. 26. Same Algorithm Sample θ q ~ P(q | D) Pick design x to maximize reward x* = argmax E[rx | q ] = argmax å xiqij x x©MapR Technologies - Confidential 26
  27. 27. Context Variables x2 x1 Bogus Dog Food is the Best! Now available in handy 1 ton bags! Buy 5! x3 y1=user.geo y2=env.time y3=env.day_of_week y4=env.weekend©MapR Technologies - Confidential 27
  28. 28. Two Kinds of Variables The web-site design - x1, x2, x3 – We can change these – Different values give different web-site designs The environment or context – y1, y2, y3, y4 – We can’t change these – They can change themselves Our model should include interactions between x and y©MapR Technologies - Confidential 28
  29. 29. Same Algorithm, More Greek Letters Sample θ, π, φ (q, P, F)~ P(q, P, F | D) Pick design x to maximize reward, y’s are constant x* = argmax E[rx | q ] x = argmax å xiqi + å xi y j p ij + å yij i x i i, j i This looks very fancy, but is actually pretty simple©MapR Technologies - Confidential 29
  30. 30. Surprises We cannot record a non-conversion until we wait We cannot record a conversion until we wait for the same time Learning from conversions requires delay We don’t have to wait very long©MapR Technologies - Confidential 30
  31. 31. ©MapR Technologies - Confidential 31
  32. 32. ©MapR Technologies - Confidential 32
  33. 33. ©MapR Technologies - Confidential 33
  34. 34. ©MapR Technologies - Confidential 34
  35. 35. Required Steps Learn distribution of parameters from data – Logistic regression or probit regression (can be on-line!) – Need Bayesian learning algorithm Sample from posterior distribution – Generally included in Bayesian learning algorithm Pick design – Simple sequential search Record data©MapR Technologies - Confidential 35
  36. 36. Required system design©MapR Technologies - Confidential 36
  37. 37. Hadoop is Not Very Real-time Unprocessed now Data t Fully Latest full Hadoop job processed period takes this long for this data©MapR Technologies - Confidential 37
  38. 38. Real-time and Long-time together Blended now View view t Hadoop works Storm great back here works here©MapR Technologies - Confidential 38
  39. 39. Traditional Hadoop Design Can use Kafka cluster to queue log lines Can use Storm cluster to do real time learning Can host web site on NAS Can use Flume cluster to import data from Kafka to Hadoop Can record long-term history on Hadoop Cluster How many clusters?©MapR Technologies - Confidential 39
  40. 40. HDFS Data Flume Hadoop Users Kafka Kafka Kafka Cluster Cluster Kafka Cluster API Storm Kafka Web Site Design Targeting Web Service NAS©MapR Technologies - Confidential 40
  41. 41. That is a lot of moving parts!©MapR Technologies - Confidential 41
  42. 42. Alternative Design Can host log catcher on MapR via NFS Storm can read data directly from queue Can host web server directly on cluster Only one cluster needed – Total instances drops by 3x – Admin burden massively decreased©MapR Technologies - Confidential 42
  43. 43. Users http Web-server Catcher Storm Topic Web Queue Data MapR©MapR Technologies - Confidential 43
  44. 44. You can do this yourself!©MapR Technologies - Confidential 44
  45. 45. Contact Me! We’re hiring at MapR in US and Europe MapR software available for research use Contact me at tdunning@maprtech.com or @ted_dunning Share news with @apachemahout Tweet #devoxxfr #mapr #mahout @ted_dunning©MapR Technologies - Confidential 45

×