1©MapR Technologies - Confidential
News From Mahout
2©MapR Technologies - Confidential
whoami – Ted Dunning
 Chief Application Architect, MapR Technologies
 Committer, memb...
3©MapR Technologies - Confidential
 Slides and such (available late tonight):
– http://www.mapr.com/company/events/nyhug-...
4©MapR Technologies - Confidential
New in Mahout
 0.8 is coming soon (1-2 months)
 gobs of fixes
 QR decomposition is 1...
5©MapR Technologies - Confidential
New in Mahout
 0.8 is coming soon (1-2 months)
 gobs of fixes
 QR decomposition is 1...
6©MapR Technologies - Confidential
New in Mahout
 0.8 is coming soon (1-2 months)
 gobs of fixes
 QR decomposition is 1...
7©MapR Technologies - Confidential
Real-time Learning
8©MapR Technologies - Confidential
We have a product
to sell …
from a web-site
9©MapR Technologies - Confidential
Bogus Dog Food is the Best!
Now available in handy 1 ton
bags!
Buy 5!
What
picture?
Wha...
10©MapR Technologies - Confidential
The Challenge
 Design decisions affect probability of success
– Cheesy web-sites don’...
11©MapR Technologies - Confidential
More Challenges
 Too many designs
– 5 pictures
– 10 tag-lines
– 4 calls to action
– 3...
12©MapR Technologies - Confidential
Example – AB testing in real-time
 I have 15 versions of my landing page
 Each visit...
13©MapR Technologies - Confidential
A Quick Diversion
 You see a coin
– What is the probability of heads?
– Could it be l...
14©MapR Technologies - Confidential
A Philosophical Conclusion
 Probability as expressed by humans is subjective and depe...
15©MapR Technologies - Confidential
I Dunno
16©MapR Technologies - Confidential
5 heads out of 10 throws
17©MapR Technologies - Confidential
2 heads out of 12 throws
18©MapR Technologies - Confidential
So now you understand
Bayesian probability
19©MapR Technologies - Confidential
Another Quick Diversion
 Let’s play a shell game
 This is a special shell game
 It ...
20©MapR Technologies - Confidential
Pause for short
con-game
21©MapR Technologies - Confidential
Interim Thoughts
 Can you identify winners or losers without trying them out?
 Can y...
22©MapR Technologies - Confidential
So now you understand
multi-armed bandits
23©MapR Technologies - Confidential
Conclusions
 Can you identify winners or losers without trying them out?
No
 Can you...
24©MapR Technologies - Confidential
Is there an optimum
strategy?
25©MapR Technologies - Confidential
Bayesian Bandit
 Compute distributions based on data so far
 Sample p1, p2 and p2 fr...
26©MapR Technologies - Confidential
And it works!
11000 100 200 300 400 500 600 700 800 900 1000
0.12
0
0.01
0.02
0.03
0.0...
27©MapR Technologies - Confidential
Video Demo
28©MapR Technologies - Confidential
The Code
 Select an alternative
 Select and learn
 But we already know how to count...
29©MapR Technologies - Confidential
The Basic Idea
 We can encode a distribution by sampling
 Sampling allows unificatio...
30©MapR Technologies - Confidential
The Original Problem
Bogus Dog Food is the Best!
Now available in handy 1 ton
bags!
Bu...
31©MapR Technologies - Confidential
Response Function
p(win) = w qi
i
å xi
æ
è
ç
ö
ø
÷
6- 6 - 4 - 2 0 2 4
1
0
0.5
x
y
32©MapR Technologies - Confidential
Generalized Banditry
 Suppose we have an infinite number of bandits
– suppose they ar...
33©MapR Technologies - Confidential
Context Variables
Bogus Dog Food is the Best!
Now available in handy 1 ton
bags!
Buy 5...
34©MapR Technologies - Confidential
Caveats
 Original Bayesian Bandit only requires real-time
 Generalized Bandit may re...
35©MapR Technologies - Confidential
You can do this
yourself!
36©MapR Technologies - Confidential
Super-fast k-means Clustering
37©MapR Technologies - Confidential
Rationale
38©MapR Technologies - Confidential
What is Quality?
 Robust clustering not a goal
– we don’t care if the same clustering...
39©MapR Technologies - Confidential
An Example
40©MapR Technologies - Confidential
An Example
41©MapR Technologies - Confidential
Diagonalized Cluster Proximity
42©MapR Technologies - Confidential
Clusters as Distribution Surrogate
43©MapR Technologies - Confidential
Clusters as Distribution Surrogate
44©MapR Technologies - Confidential
Theory
45©MapR Technologies - Confidential
For Example
Grouping these
two clusters
seriously hurts
squared distance
D4
2
(X) >
1
...
46©MapR Technologies - Confidential
Algorithms
47©MapR Technologies - Confidential
Typical k-means Failure
Selecting two seeds
here cannot be
fixed with Lloyds
Result is...
48©MapR Technologies - Confidential
Ball k-means
 Provably better for highly clusterable data
 Tries to find initial cen...
49©MapR Technologies - Confidential
Still Not a Win
 Ball k-means is nearly guaranteed with k = 2
 Probability of succes...
50©MapR Technologies - Confidential
Still Not a Win
 Ball k-means is nearly guaranteed with k = 2
 Probability of succes...
51©MapR Technologies - Confidential
Surrogate Method
 Start with sloppy clustering into lots of clusters
κ = k log n clus...
52©MapR Technologies - Confidential
Algorithm Costs
 Surrogate methods
– fast, sloppy single pass clustering with κ = k l...
53©MapR Technologies - Confidential
Algorithm Costs
 Surrogate methods
– fast, sloppy single pass clustering with κ = k l...
54©MapR Technologies - Confidential
Algorithm Costs
 How much faster for the sketch phase?
– take k = 2000, d = 10, n = 1...
55©MapR Technologies - Confidential
Algorithm Costs
 How much faster for the sketch phase?
– take k = 2000, d = 10, n = 1...
56©MapR Technologies - Confidential
How It Works
 For each point
– Find approximately nearest centroid (distance = d)
– I...
57©MapR Technologies - Confidential
Implementation
58©MapR Technologies - Confidential
But Wait, …
 Finding nearest centroid is inner loop
 This could take O( d κ ) per po...
59©MapR Technologies - Confidential
Projection Search
total ordering!
60©MapR Technologies - Confidential
LSH Bit-match Versus Cosine
0 8 16 24 32 40 48 56 64
1
- 1
- 0.8
- 0.6
- 0.4
- 0.2
0
0...
61©MapR Technologies - Confidential
Results
62©MapR Technologies - Confidential
Parallel Speedup?
1 2 3 4 5 20
10
100
20
30
40
50
200
Threads
Timeperpoint(μs)
2
3
4
5...
63©MapR Technologies - Confidential
Quality
 Ball k-means implementation appears significantly better than
simple k-means...
64©MapR Technologies - Confidential
Contact Me!
 We’re hiring at MapR in US and Europe
 MapR software available for rese...
Upcoming SlideShare
Loading in...5
×

News From Mahout

136

Published on

Presentation to the NYC HUG on March 5, 2013 regarding the Mahout release.

Published in: Technology, Design
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
136
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The basic idea here is that I have colored slides to be presented by you in blue. You should substitute and reword those slides as you like. In a few places, I imagined that we would have fast back and forth as in the introduction or final slide where we can each say we are hiring in turn.The overall thrust of the presentation is for you to make these points:Amex does lots of modelingit is expensivehaving a way to quickly test models and new variables would be awesomeso we worked on a new project with MapRMy part will say the following:Knn basic pictorial motivation (could move to you if you like)describe knn quality metric of overlapshow how bad metric breaks knn (optional)quick description of LSH and projection searchpicture of why k-means search is coolmotivate k-means speed as tool for k-means searchdescribe single pass k-means algorithmdescribe basic data structuresshow parallel speedupOur summary should state that we have achievedsuper-fast k-means clusteringinitial version of super-fast knn search with good overlap
  • News From Mahout

    1. 1. 1©MapR Technologies - Confidential News From Mahout
    2. 2. 2©MapR Technologies - Confidential whoami – Ted Dunning  Chief Application Architect, MapR Technologies  Committer, member, Apache Software Foundation – particularly Mahout, Zookeeper and Drill (we’re hiring)  Contact me at tdunning@maprtech.com tdunning@apache.com ted.dunning@gmail.com @ted_dunning
    3. 3. 3©MapR Technologies - Confidential  Slides and such (available late tonight): – http://www.mapr.com/company/events/nyhug-03-05-2013  Hash tags: #mapr #nyhug #mahout
    4. 4. 4©MapR Technologies - Confidential New in Mahout  0.8 is coming soon (1-2 months)  gobs of fixes  QR decomposition is 10x faster – makes ALS 2-3 times faster  May include Bayesian Bandits  Super fast k-means – fast – online (!?!)
    5. 5. 5©MapR Technologies - Confidential New in Mahout  0.8 is coming soon (1-2 months)  gobs of fixes  QR decomposition is 10x faster – makes ALS 2-3 times faster  May include Bayesian Bandits  Super fast k-means – fast – online (!?!) – fast  Possible new edition of MiA coming – Japanese and Korean editions released, Chinese coming
    6. 6. 6©MapR Technologies - Confidential New in Mahout  0.8 is coming soon (1-2 months)  gobs of fixes  QR decomposition is 10x faster – makes ALS 2-3 times faster  May include Bayesian Bandits  Super fast k-means – fast – online (!?!) – fast  Possible new edition of MiA coming – Japanese and Korean editions released, Chinese coming
    7. 7. 7©MapR Technologies - Confidential Real-time Learning
    8. 8. 8©MapR Technologies - Confidential We have a product to sell … from a web-site
    9. 9. 9©MapR Technologies - Confidential Bogus Dog Food is the Best! Now available in handy 1 ton bags! Buy 5! What picture? What tag- line? What call to action?
    10. 10. 10©MapR Technologies - Confidential The Challenge  Design decisions affect probability of success – Cheesy web-sites don’t even sell cheese  The best designers do better when allowed to fail – Exploration juices creativity  But failing is expensive – If only because we could have succeeded – But also because offending or disappointing customers is bad
    11. 11. 11©MapR Technologies - Confidential More Challenges  Too many designs – 5 pictures – 10 tag-lines – 4 calls to action – 3 back-ground colors => 5 x 10 x 4 x 3 = 600 designs  It gets worse quickly – What about changes on the back-end? – Search engine variants? – Checkout process variants?
    12. 12. 12©MapR Technologies - Confidential Example – AB testing in real-time  I have 15 versions of my landing page  Each visitor is assigned to a version – Which version?  A conversion or sale or whatever can happen – How long to wait?  Some versions of the landing page are horrible – Don’t want to give them traffic
    13. 13. 13©MapR Technologies - Confidential A Quick Diversion  You see a coin – What is the probability of heads? – Could it be larger or smaller than that?  I flip the coin and while it is in the air ask again  I catch the coin and ask again  I look at the coin (and you don’t) and ask again  Why does the answer change? – And did it ever have a single value?
    14. 14. 14©MapR Technologies - Confidential A Philosophical Conclusion  Probability as expressed by humans is subjective and depends on information and experience
    15. 15. 15©MapR Technologies - Confidential I Dunno
    16. 16. 16©MapR Technologies - Confidential 5 heads out of 10 throws
    17. 17. 17©MapR Technologies - Confidential 2 heads out of 12 throws
    18. 18. 18©MapR Technologies - Confidential So now you understand Bayesian probability
    19. 19. 19©MapR Technologies - Confidential Another Quick Diversion  Let’s play a shell game  This is a special shell game  It costs you nothing to play  The pea has constant probability of being under each shell (trust me)  How do you find the best shell?  How do you find it while maximizing the number of wins?
    20. 20. 20©MapR Technologies - Confidential Pause for short con-game
    21. 21. 21©MapR Technologies - Confidential Interim Thoughts  Can you identify winners or losers without trying them out?  Can you ever completely eliminate a shell with a bad streak?  Should you keep trying apparent losers?
    22. 22. 22©MapR Technologies - Confidential So now you understand multi-armed bandits
    23. 23. 23©MapR Technologies - Confidential Conclusions  Can you identify winners or losers without trying them out? No  Can you ever completely eliminate a shell with a bad streak? No  Should you keep trying apparent losers? Yes, but at a decreasing rate
    24. 24. 24©MapR Technologies - Confidential Is there an optimum strategy?
    25. 25. 25©MapR Technologies - Confidential Bayesian Bandit  Compute distributions based on data so far  Sample p1, p2 and p2 from these distributions  Pick shell i where i = argmaxi pi  Lemma 1: The probability of picking shell i will match the probability it is the best shell  Lemma 2: This is as good as it gets
    26. 26. 26©MapR Technologies - Confidential And it works! 11000 100 200 300 400 500 600 700 800 900 1000 0.12 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.11 n regret ε- greedy, ε = 0.05 Bayesian Bandit with Gamma- Normal
    27. 27. 27©MapR Technologies - Confidential Video Demo
    28. 28. 28©MapR Technologies - Confidential The Code  Select an alternative  Select and learn  But we already know how to count! n = dim(k)[1] p0 = rep(0, length.out=n) for (i in 1:n) { p0[i] = rbeta(1, k[i,2]+1, k[i,1]+1) } return (which(p0 == max(p0))) for (z in 1:steps) { i = select(k) j = test(i) k[i,j] = k[i,j]+1 } return (k)
    29. 29. 29©MapR Technologies - Confidential The Basic Idea  We can encode a distribution by sampling  Sampling allows unification of exploration and exploitation  Can be extended to more general response models
    30. 30. 30©MapR Technologies - Confidential The Original Problem Bogus Dog Food is the Best! Now available in handy 1 ton bags! Buy 5! x1 x2 x3
    31. 31. 31©MapR Technologies - Confidential Response Function p(win) = w qi i å xi æ è ç ö ø ÷ 6- 6 - 4 - 2 0 2 4 1 0 0.5 x y
    32. 32. 32©MapR Technologies - Confidential Generalized Banditry  Suppose we have an infinite number of bandits – suppose they are each labeled by two real numbers x and y in [0,1] – also that expected payoff is a parameterized function of x and y – now assume a distribution for θ that we can learn online  Selection works by sampling θ, then computing f  Learning works by propagating updates back to θ – If f is linear, this is very easy – For special other kinds of f it isn’t too hard  Don’t just have to have two labels, could have labels and context E z[ ] = f (x, y |q)
    33. 33. 33©MapR Technologies - Confidential Context Variables Bogus Dog Food is the Best! Now available in handy 1 ton bags! Buy 5! x1 x2 x3 user.geo env.time env.day_of_week env.weekend
    34. 34. 34©MapR Technologies - Confidential Caveats  Original Bayesian Bandit only requires real-time  Generalized Bandit may require access to long history for learning – Pseudo online learning may be easier than true online  Bandit variables can include content, time of day, day of week  Context variables can include user id, user features  Bandit × context variables provide the real power
    35. 35. 35©MapR Technologies - Confidential You can do this yourself!
    36. 36. 36©MapR Technologies - Confidential Super-fast k-means Clustering
    37. 37. 37©MapR Technologies - Confidential Rationale
    38. 38. 38©MapR Technologies - Confidential What is Quality?  Robust clustering not a goal – we don’t care if the same clustering is replicated  Generalization is critical  Agreement to “gold standard” is a non-issue
    39. 39. 39©MapR Technologies - Confidential An Example
    40. 40. 40©MapR Technologies - Confidential An Example
    41. 41. 41©MapR Technologies - Confidential Diagonalized Cluster Proximity
    42. 42. 42©MapR Technologies - Confidential Clusters as Distribution Surrogate
    43. 43. 43©MapR Technologies - Confidential Clusters as Distribution Surrogate
    44. 44. 44©MapR Technologies - Confidential Theory
    45. 45. 45©MapR Technologies - Confidential For Example Grouping these two clusters seriously hurts squared distance D4 2 (X) > 1 s 2 D5 2 (X)
    46. 46. 46©MapR Technologies - Confidential Algorithms
    47. 47. 47©MapR Technologies - Confidential Typical k-means Failure Selecting two seeds here cannot be fixed with Lloyds Result is that these two clusters get glued together
    48. 48. 48©MapR Technologies - Confidential Ball k-means  Provably better for highly clusterable data  Tries to find initial centroids in each “core” of each real clusters  Avoids outliers in centroid computation initialize centroids randomly with distance maximizing tendency for each of a very few iterations: for each data point: assign point to nearest cluster recompute centroids using only points much closer than closest cluster
    49. 49. 49©MapR Technologies - Confidential Still Not a Win  Ball k-means is nearly guaranteed with k = 2  Probability of successful seeding drops exponentially with k  Alternative strategy has high probability of success, but takes O(nkd + k3d) time
    50. 50. 50©MapR Technologies - Confidential Still Not a Win  Ball k-means is nearly guaranteed with k = 2  Probability of successful seeding drops exponentially with k  Alternative strategy has high probability of success, but takes O( nkd + k3d ) time  But for big data, k gets large
    51. 51. 51©MapR Technologies - Confidential Surrogate Method  Start with sloppy clustering into lots of clusters κ = k log n clusters  Use this sketch as a weighted surrogate for the data  Results are provably good for highly clusterable data
    52. 52. 52©MapR Technologies - Confidential Algorithm Costs  Surrogate methods – fast, sloppy single pass clustering with κ = k log n – fast sloppy search for nearest cluster, O(d log κ) = O(d (log k + log log n)) per point – fast, in-memory, high-quality clustering of κ weighted centroids O(κ k d + k3 d) = O(k2 d log n + k3 d) for small k, high quality O(κ d log k) or O(d log κ log k) for larger k, looser quality – result is k high-quality centroids • Even the sloppy surrogate may suffice
    53. 53. 53©MapR Technologies - Confidential Algorithm Costs  Surrogate methods – fast, sloppy single pass clustering with κ = k log n – fast sloppy search for nearest cluster, O(d log κ) = O(d ( log k + log log n )) per point – fast, in-memory, high-quality clustering of κ weighted centroids O(κ k d + k3 d) = O(k2 d log n + k3 d) for small k, high quality O(κ d log k) or O( d log k ( log k + log log n ) ) for larger k, looser quality – result is k high-quality centroids • For many purposes, even the sloppy surrogate may suffice
    54. 54. 54©MapR Technologies - Confidential Algorithm Costs  How much faster for the sketch phase? – take k = 2000, d = 10, n = 100,000 – k d log n = 2000 x 10 x 26 = 500,000 – d (log k + log log n) = 10(11 + 5) = 170 – 3,000 times faster is a bona fide big deal
    55. 55. 55©MapR Technologies - Confidential Algorithm Costs  How much faster for the sketch phase? – take k = 2000, d = 10, n = 100,000 – k d log n = 2000 x 10 x 26 = 500,000 – d (log k + log log n) = 10(11 + 5) = 170 – 3,000 times faster is a bona fide big deal
    56. 56. 56©MapR Technologies - Confidential How It Works  For each point – Find approximately nearest centroid (distance = d) – If (d > threshold) new centroid – Else if (u > d/threshold) new cluster – Else add to nearest centroid  If centroids > κ ≈ C log N – Recursively cluster centroids with higher threshold
    57. 57. 57©MapR Technologies - Confidential Implementation
    58. 58. 58©MapR Technologies - Confidential But Wait, …  Finding nearest centroid is inner loop  This could take O( d κ ) per point and κ can be big  Happily, approximate nearest centroid works fine
    59. 59. 59©MapR Technologies - Confidential Projection Search total ordering!
    60. 60. 60©MapR Technologies - Confidential LSH Bit-match Versus Cosine 0 8 16 24 32 40 48 56 64 1 - 1 - 0.8 - 0.6 - 0.4 - 0.2 0 0.2 0.4 0.6 0.8 X Axis YAxis
    61. 61. 61©MapR Technologies - Confidential Results
    62. 62. 62©MapR Technologies - Confidential Parallel Speedup? 1 2 3 4 5 20 10 100 20 30 40 50 200 Threads Timeperpoint(μs) 2 3 4 5 6 8 10 12 14 16 Threaded version Non- threaded Perfect Scaling ✓
    63. 63. 63©MapR Technologies - Confidential Quality  Ball k-means implementation appears significantly better than simple k-means  Streaming k-means + ball k-means appears to be about as good as ball k-means alone  All evaluations on 20 newsgroups with held-out data  Figure of merit is mean and median squared distance to nearest cluster
    64. 64. 64©MapR Technologies - Confidential Contact Me!  We’re hiring at MapR in US and Europe  MapR software available for research use  Get the code as part of Mahout trunk (or 0.8 very soon)  Contact me at tdunning@maprtech.com or @ted_dunning  Share news with @apachemahout
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×