0
1©MapR Technologies - Confidential
Super-Fast Clustering
Report from MapR workshop
2©MapR Technologies - Confidential
 For Book Discount: @ellen_friedman
 Contact:
– tdunning@maprtech.com
– @ted_dunning
...
3©MapR Technologies - Confidential
Company Background
 MapR provides the industry’s best Hadoop Distribution
– Combines t...
4©MapR Technologies - Confidential
We Also Do …
 Open source development
– Zookeeper
– Hadoop
– Mahout
– Stuff
 Partner ...
5©MapR Technologies - Confidential
We Also Do …
 Open source development
– Zookeeper
– Hadoop
– Mahout
– Stuff
 Partner ...
6©MapR Technologies - Confidential
The Problem
 A certain bank
– had lots of customers
– had lots of prospective customer...
7©MapR Technologies - Confidential
But …
 These models were arduous to build
 And hard to test
 So people suggested som...
8©MapR Technologies - Confidential
What’s that?
 Find the k nearest training examples
 Use the average value of the targ...
9©MapR Technologies - Confidential
What We Did
 Mechanism for extending Mahout Vectors
– DelegatingVector, WeightedVector...
10©MapR Technologies - Confidential
Projection Search
11©MapR Technologies - Confidential
K-means Search
12©MapR Technologies - Confidential
But These Require k-means!
 Need a new k-means algorithm to get speed
 Streaming k-m...
13©MapR Technologies - Confidential
How It Works
 For each point
– Find approximately nearest centroid (distance = d)
– I...
14©MapR Technologies - Confidential
Parallel Speedup?
1 2 3 4 5 20
10
100
20
30
40
50
200
Threads
Timeperpoint(μs)
2
3
4
5...
15©MapR Technologies - Confidential
Warning, Recursive Descent
 Inner loop requires finding nearest centroid
 With lots ...
16©MapR Technologies - Confidential
Warning, Recursive Descent
 Inner loop requires finding nearest centroid
 With lots ...
17©MapR Technologies - Confidential
 Contact:
– tdunning@maprtech.com
– @ted_dunning
 Slides and such:
– http://info.map...
18©MapR Technologies - Confidential
Thank You
Upcoming SlideShare
Loading in...5
×

London Data Science - Super-Fast Clustering Report

177

Published on

Talk given to the London Data Science meetup about work on k-nearest neighbor search and fast clustering.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
177
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • MapR combines the best of the open source technology with our own deep innovations to provide the most advanced distribution for Apache Hadoop.MapR’s team has a deep bench of enterprise software experience with proven success across storage, networking, virtualization, analytics, and open source technologies.Our CEO has driven multiple companies to successful outcomes in the analytic, storage, and virtualization spaces.Our CTO and co-founder M.C. Srivas was most recently at Google in BigTable. He understands the challenges of MapReduce at huge scale. Srivas was also the chief software architect at Spinnaker Networks which came out of stealth with the fastest NAS storage on the market and was acquired quickly by NetAppThe team includes experience with enterprise storage at Cisco, VmWare, IBM and EMC. Our VP of Engineering led emerging technologies and a 600 person for EMC’s NAS engineering team. We also have experience in Business Intelligence and Analytic companies and open source committers in Hadoop, Zookeeper and Mahout including PMC members.MapR is proven technology with installs by leading Hadoop installations across industries and OEM by EMC and Cisco.
  • Transcript of "London Data Science - Super-Fast Clustering Report"

    1. 1. 1©MapR Technologies - Confidential Super-Fast Clustering Report from MapR workshop
    2. 2. 2©MapR Technologies - Confidential  For Book Discount: @ellen_friedman  Contact: – tdunning@maprtech.com – @ted_dunning  Twitter for this talk – #mapr_uk  Slides and such: – http://info.mapr.com/ted-uk-05-2012
    3. 3. 3©MapR Technologies - Confidential Company Background  MapR provides the industry’s best Hadoop Distribution – Combines the best of the Hadoop community contributions with significant internally financed infrastructure development  Background of Team – Deep management bench with extensive analytic, storage, virtualization, and open source experience – Google, EMC, Cisco, VMWare, Network Appliance, IBM, Microsoft, Apache Foundation, Aster Data, Brio, ParAccel  Proven – MapR used across industries (Financial Services, Media, Telcom, Health Care, Internet Services, Government) – Strategic OEM relationship with EMC and Cisco – Over 1,000 installs
    4. 4. 4©MapR Technologies - Confidential We Also Do …  Open source development – Zookeeper – Hadoop – Mahout – Stuff  Partner workshops – Machine learning – Information architecture – Cluster design
    5. 5. 5©MapR Technologies - Confidential We Also Do …  Open source development – Zookeeper – Hadoop – Mahout – Stuff  Partner workshops – Machine learning – Information architecture – Cluster design
    6. 6. 6©MapR Technologies - Confidential The Problem  A certain bank – had lots of customers – had lots of prospective customers – had a non-trivial number of fraudulent customers – had a non-trivial number of fraudulent merchants  They also – collected data – built models – collected more data – built more models
    7. 7. 7©MapR Technologies - Confidential But …  These models were arduous to build  And hard to test  So people suggested something simpler  Like k-nearest neighbor
    8. 8. 8©MapR Technologies - Confidential What’s that?  Find the k nearest training examples  Use the average value of the target variable from them  This is easy … but hard – easy because it is so conceptually simple and you don’t have knobs to turn or models to build – hard because of the stunning amount of math – also hard because we need top 50,000 results  Initial prototype was massively too slow – 3K queries x 200K examples takes hours – needed 20M x 25M in the same time
    9. 9. 9©MapR Technologies - Confidential What We Did  Mechanism for extending Mahout Vectors – DelegatingVector, WeightedVector, Centroid  Searcher interface – ProjectionSearch, KmeansSearch, LshSearch, Brute  Super-fast clustering – Kmeans, StreamingKmeans
    10. 10. 10©MapR Technologies - Confidential Projection Search
    11. 11. 11©MapR Technologies - Confidential K-means Search
    12. 12. 12©MapR Technologies - Confidential But These Require k-means!  Need a new k-means algorithm to get speed  Streaming k-means is – One pass (through the original data) – Very fast (20 us per data point with threads) – Very parallelizable
    13. 13. 13©MapR Technologies - Confidential How It Works  For each point – Find approximately nearest centroid (distance = d) – If d > threshold, new centroid – Else possibly new cluster – Else add to nearest centroid  If centroids > K ~ C log N – Recursively cluster centroids with higher threshold  Result is large set of centroids – these provide approximation of original distribution – we can cluster centroids to get a close approximation of clustering original – or we can just use the result directly
    14. 14. 14©MapR Technologies - Confidential Parallel Speedup? 1 2 3 4 5 20 10 100 20 30 40 50 200 Threads Timeperpoint(μs) 2 3 4 5 6 8 10 12 14 16 Threaded version Non- threaded Perfect Scaling ✓
    15. 15. 15©MapR Technologies - Confidential Warning, Recursive Descent  Inner loop requires finding nearest centroid  With lots of centroids, this is slow  But wait, we have classes to accelerate that!
    16. 16. 16©MapR Technologies - Confidential Warning, Recursive Descent  Inner loop requires finding nearest centroid  With lots of centroids, this is slow  But wait, we have classes to accelerate that! (Let’s not use k-means searcher, though)
    17. 17. 17©MapR Technologies - Confidential  Contact: – tdunning@maprtech.com – @ted_dunning  Slides and such: – http://info.mapr.com/ted-uk-05-2012
    18. 18. 18©MapR Technologies - Confidential Thank You
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×