Downloaded 32 times
Uploaded byDataWorks Summit
How to tell which algorithms really matter
This document discusses how to determine which algorithms and techniques really matter for practical applications. It provides examples of techniques that had a big impact on recommendations, such as result dithering which improved user experience more than advances in recommendation algorithms. It also discusses how techniques like Bayesian bandits and online clustering with sketches can enable effective exploration and handle large, streaming data. Additionally, it describes how search and recommendations can recursively improve each other through indicator matrices.
More Related Content
Similar to How to tell which algorithms really matter
More from DataWorks Summit
How to tell which algorithms really matter
- 1. How to TellWhich Algorithms Really Matter Ted Dunning MapR Technologies
- 2. © 2014 MapRTechnologies 2
- 3. © 2014 MapRTechnologies 3 00:011.65TB WITH 298 SERVERS
- 4. © 2014 MapRTechnologies 4 129K RECCOMENDATIONS 00:02
- 5. © 2014 MapRTechnologies 5 Advertising Automation Cloud Sellers Cloud Buyers Cloud 63M AD AUCTIONS 00:03
- 6. © 2014 MapRTechnologies 6 00:04422.2K GENETIC SEQUENCES
- 7. © 2014 MapRTechnologies 7 Largest Biometric Database 00:054.73M AUTHENTICATIONS
- 8. © 2014 MapRTechnologies 8© 2014 MapR Technologies But How is This Done? What really matters?
- 9. © 2014 MapRTechnologies 9 Topic For Today • What is important? What is not? • Why? • What is the difference from academic research? • Some examples
- 10. © 2014 MapRTechnologies 10 What is Important? • Deployable • Robust • Transparent • Skillset and mindset matched? • Proportionate
- 11. © 2014 MapRTechnologies 11 What is Important? • Deployable – Clever prototypes don’t count if they can’t be standardized • Robust • Transparent • Skillset and mindset matched? • Proportionate
- 12. © 2014 MapRTechnologies 12 What is Important? • Deployable – Clever prototypes don’t count • Robust – Mishandling is common • Transparent – Will degradation be obvious? • Skillset and mindset matched? • Proportionate
- 13. © 2014 MapRTechnologies 13 What is Important? • Deployable – Clever prototypes don’t count • Robust – Mishandling is common • Transparent – Will degradation be obvious? • Skillset and mindset matched? – How long will your fancy data scientist enjoy doing standard ops tasks? • Proportionate – Where is the highest value per minute of effort?
- 14. © 2014 MapRTechnologies 14 Academic Goals vs Pragmatics • Academic goals – Reproducible – Isolate theoretically important aspects – Work on novel problems • Pragmatics – Highest net value – Available data is constantly changing – Diligence and consistency have larger impact than cleverness – Many systems feed themselves, exploration and exploitation are both important – Engineering constraints on budget and schedule
- 15. © 2014 MapRTechnologies 15 Example 1: Making Recommendations Better
- 16. © 2014 MapRTechnologies 16 Recommendation Advances • What are the most important algorithmic advances in recommendations over the last 10 years? • Cooccurrence analysis? • Matrix completion via factorization? • Latent factor log-linear models? • Temporal dynamics?
- 17. © 2014 MapRTechnologies 17 The Winner – None of the Above • What are the most important algorithmic advances in recommendations over the last 10 years? 1. Result dithering (random noise) 2. Anti-flood (don’t repeat yourself)
- 18. © 2014 MapRTechnologies 18 The Real Issues • Exploration • Diversity • Speed • Not the last fraction of a percent
- 19. © 2014 MapRTechnologies 19 Result Dithering • Dithering is used to re-order recommendation results – Re-ordering is done randomly • Dithering is guaranteed to make off-line performance worse • Dithering also has a near perfect record of making actual performance much better
- 20. © 2014 MapRTechnologies 20 Result Dithering • Dithering is used to re-order recommendation results – Re-ordering is done randomly • Dithering is guaranteed to make off-line performance worse • Dithering also has a near perfect record of making actual performance much better “Made more difference than any other change”
- 21. © 2014 MapRTechnologies 22 Example … ε = 0.5 1 2 6 5 3 4 13 16 1 2 3 8 5 7 6 34 1 4 3 2 6 7 11 10 1 2 4 3 15 7 13 19 1 6 2 3 4 16 9 5 1 2 3 5 24 7 17 13 1 2 3 4 6 12 5 14 2 1 3 5 7 6 4 17 4 1 2 7 3 9 8 5 2 1 5 3 4 7 13 6 3 1 5 4 2 7 8 6 2 1 3 4 7 12 17 16
- 22. © 2014 MapRTechnologies 23 Example … ε = log 2 = 0.69 1 2 8 3 9 15 7 6 1 8 14 15 3 2 22 10 1 3 8 2 10 5 7 4 1 2 10 7 3 8 6 14 1 5 33 15 2 9 11 29 1 2 7 3 5 4 19 6 1 3 5 23 9 7 4 2 2 4 11 8 3 1 44 9 2 3 1 4 6 7 8 33 3 4 1 2 10 11 15 14 11 1 2 4 5 7 3 14 1 8 7 3 22 11 2 33
- 23. © 2014 MapRTechnologies 24 Exploring The Second Page
- 24. © 2014 MapRTechnologies 25 Lesson 1: Exploration is good
- 25. © 2014 MapRTechnologies 26 Example 2: Bayesian Bandits
- 26. © 2014 MapRTechnologies 27 Bayesian Bandits • Based on Thompson sampling • Very general sequential test • Near optimal regret • Trade-off exploration and exploitation • Possibly best known solution for exploration/exploitation • Incredibly simple
- 27. © 2014 MapRTechnologies 30 Fast Convergence 11000 100 200 300 400 500 600 700 800 900 1000 0.12 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.11 n regret ε- greedy, ε = 0.05 Bayesian Bandit with Gamma- Normal
- 28. © 2014 MapRTechnologies 31 Thompson Sampling on Ads An Empirical Evaluation of Thompson Sampling - Chapelle and Li, 2011
- 29. © 2014 MapRTechnologies 32 Bayesian Bandits versus Result Dithering • Many useful systems are difficult to frame in fully Bayesian form • Thompson sampling cannot be applied without posterior sampling • Can still do useful exploration with dithering • But better to use Thompson sampling if possible
- 30. © 2014 MapRTechnologies 33 Lesson 2: Exploration is easy to do and pays big benefits.
- 31. © 2014 MapRTechnologies 34 Example 3: On-line Clustering
- 32. © 2014 MapRTechnologies 35 The Problem • K-means clustering is useful for feature extraction or compression • At scale and at high dimension, the desirable number of clusters increases • Very large number of clusters may require more passes through the data • Super-linear scaling is generally infeasible
- 33. © 2014 MapRTechnologies 36 The Solution • Sketch-based algorithms produce a sketch of the data • Streaming k-means uses adaptive dp-means to produce this sketch in the form of many weighted centroids which approximate the original distribution • The size of the sketch grows very slowly with increasing data size • Many operations such as clustering are well behaved on sketches Fast and Accurate k-means For Large Datasets. Michael Shindler, Alex Wong, Adam Meyerson. Revisiting k-means: New Algorithms via Bayesian Nonparametrics . Brian Kulis, Michael Jordan.
- 34. © 2014 MapRTechnologies 37 An Example
- 35. © 2014 MapRTechnologies 38 An Example
- 36. © 2014 MapRTechnologies 43 Streaming k-means Ideas • By using a sketch with lots (k log N) of centroids, we avoid pathological cases • We still get a very good result if the sketch is created – in one pass – with approximate search • In fact, adaptive dp-means works just fine • In the end, the sketch can be used for clustering or …
- 37. © 2014 MapRTechnologies 44 Lesson 3: Sketches make big data small.
- 38. © 2014 MapRTechnologies 45 Example 4: Search Abuse
- 39. © 2014 MapRTechnologies 46 Recommendations Alice got an apple and a puppy Charles got a bicycle Alice Charles
- 40. © 2014 MapRTechnologies 47 Recommendations Alice got an apple and a puppy Charles got a bicycle Bob got an apple Alice Bob Charles
- 41. © 2014 MapRTechnologies 48 Recommendations What else would Bob like?? Alice Bob Charles
- 42. © 2014 MapRTechnologies 49 Log Files Alice Bob Charles Alice Bob Charles Alice
- 43. © 2014 MapRTechnologies 50 History Matrix: Users by Items Alice Bob Charles ✔ ✔ ✔ ✔ ✔ ✔ ✔
- 44. © 2014 MapRTechnologies 51 Co-occurrence Matrix: Items by Items - 1 2 1 1 1 1 2 1 How do you tell which co-occurrences are useful?. 0 0 0 0
- 45. © 2014 MapRTechnologies 53 Indicator Matrix: Anomalous Co-Occurrence ✔ ✔ Result: The marked row will be added to the indicator field in the item document…
- 46. © 2014 MapRTechnologies 54 Indicator Matrix ✔ id: t4 title: puppy desc: The sweetest little puppy ever. keywords: puppy, dog, pet indicators: (t1) That one row from indicator matrix becomes the indicator field in the Solr document used to deploy the recommendation engine. Note: data for the indicator field is added directly to meta-data for a document in Solr index. You don’t need to create a separate index for the indicators.
- 47. © 2014 MapRTechnologies 56 Internals of the Recommender Engine 56
- 48. © 2014 MapRTechnologies 58 Real-life example
- 49. © 2014 MapRTechnologies 59 Lesson 4: Recursive search abuse pays Search can implement recs Which can implement search
- 50. © 2014 MapRTechnologies 60 How Does This Apply?
- 51. © 2014 MapRTechnologies 61 How Can I Start?
- 52. © 2014 MapRTechnologies 62 Q&A @ted_dunning @mapr maprtech tdunning@mapr.com Engage with us! MapR maprtech mapr-technologies
- 53. © 2014 MapRTechnologies 64
Editor's Notes
- #3 I just have 5 minutes for this talk. Given the short time I thought I’d share with you some of the more interesting things you can do with Hadoop in 5 minutes or less…
- #4 The Minutesort benchmark is technology agnostic, until recently the record was held by Microsoft using custom software and dedicated high end hardware. Yahoo broke the record and sorted 1.6TB in one minute using 2200 servers. This is not limited to just the originators of Hadoop…One of our customers over a weekend recently broke this record performing 1.65TB in one minute with 298 servers. This performance is key to their use of Hadoop as it is a critical part of their business operations.
- #5 After just a few minutes of work here with Hadoop I could use a minute to relax – Beats headphones by Dr. Dre have swept the audio market. Beats has launched a new Beats Music service thatis able to personalize music selections and select the perfect song in a minute from over 20 million songs. It joins a crowded space for online music, but now by using Hadoop Beats is able to provide a completely new personalized service from over 20 million songs in their library. In their very first day, they were processing 129,000 music interactions per minute. A number that is only growing.11 Million events per day
- #6 In our third minute you could perform 63M ad auctions with the Rubicon Project. The company’s pioneering technology created a new model for the advertising industry – similar to what NASDAQ did for stock trading. Rubicon Project’s automated advertising platform is used by more than 500 of the world’s premium publishers to transact with over 100,000 ad brands globally.63M might seem like a lot but that’s just the average not the peak performance of the Rubicon Project that perform 90B ad auctions each day -- providing the most extensive ad reach in the industry touching 96% of internet users in the US. You might ask how do we know who has the largest ad reach. Well this was measured by comscore.
- #7 You can use a second minute to change Healthcare. Doctors, particularly oncologists, are faced with an enormous amount of data regarding patient treatments, outcomes, and disease states. Hadoop is having an impact across the health care industry but for this minute we will focus on its use for developing better treatments. In one minute Hadoop can analyze more than 20,000 genes across hundreds of thousands of patients. The outcome of this analysis is to get a better understanding of genomic factors and integrate imaging and clinical analytics to better understand, predict, and impact survival. In any given minute our cluster is sequencing 422,000 genes per minute.
- #8 In 1 minute you can perform 4.73 million concurrent authentications in the largest biometric database in the worldIn India, there is no social security card. It’s difficult for the average citizen to set up a bank account, access benefit programs, and enjoy economic mobility. It’s difficult for the government as well with over a $1B of government aid classified as leakage, the result of fraud and corruption. The Aadhaar program is poised to change all that by leveraging the unique IDs that all people are born with. The program aims to get fingerprints and retina scans for all 1.2 billion citizens. The scale of this project required an integrated in-Hadoop database that was capable of 200 millisecond response times while supporting millions of concurrent look-ups.
- #9 Hadoop is making CIO’s rethink their data architecture. It is a fundamental shift in the economics of data storage/processing/analytics, and is opening up entirely new business opportunities. Let’s talk about 3 key trends we are seeing, as well as 3 realities or implications on your business and “readiness” to harness the power of big data and Hadoop.
- #47 * A history of what everybody has done. Obviously this is just a cartoon because large numbers of users and interactions with items would be required to build a recommender* Next step will be to predict what a new user might like…
- #48 *Bob is the “new user” and getting apple is his history
- #49 *Here is where the recommendation engine needs to go to work…Note to trainer: you might see if audience calls out the answer before revealing next slide…
- #50 Note to trainer: This is the situation similar to that in which we started, with three users in our history. The difference is that now everybody got a pony. Bob has apple and pony but not a puppy…yet
- #51 *Binary matrix is stored sparsely
- #52 *Convert by MapReduce into a binary matrixNote to trainer: Whether consider apple to have occurred with self is open question
- #53 Old joke: all the world can be divided into 2 categories: Scotch tape and non-Scotch tape… This is a way to think about the co-occurrence
- #54 Only important co-occurrence is puppy follows apple
- #55 *Take that row of matrix and combine with all the meta data we might have…*Important thing to get from the co-occurrence matrix is this indicator..Cool thing: analogous to what a lot of recommendation engines do*This row forms the indicator field in a Solr document containing meta-data (you do NOT have to build a separate index for the indicators)Find the useful co-occurrence and get rid of the rest. Sparsify and get the anomalous co-occurrence
- #56 Note to trainer: take a little time to explore this here and on the next couple of slides. Details enlarged on next slide
- #57 *This indicator field is where the output of the Mahout recommendation engine are stored (the row from the indicator matrix that identified significant or interesting co-occurrence. *Keep in mind that this recommendation indicator data is added to the same original document in the Solr index that contains meta data for the item in question
- #58 This is a diagnostics window in the LucidWorksSolr index (not the web interface a user would see). It’s a way for the developer to do a rough evaluation (laugh test) of the choices offered by the recommendation engine.In other words, do these indicator artists represented by their indicator Id make reasonable recommendations Note to trainer: artist 303 happens to be The Beatles. Is that a good match for Chuck Berry?
- #61 In 1 minute you can perform 4.73 million concurrent authentications in the largest biometric database in the worldIn India, there is no social security card. It’s difficult for the average citizen to set up a bank account, access benefit programs, and enjoy economic mobility. It’s difficult for the government as well with over a $1B of government aid classified as leakage, the result of fraud and corruption. The Aadhaar program is poised to change all that by leveraging the unique IDs that all people are born with. The program aims to get fingerprints and retina scans for all 1.2 billion citizens. The scale of this project required an integrated in-Hadoop database that was capable of 200 millisecond response times while supporting millions of concurrent look-ups.
- #62 In 1 minute you can perform 4.73 million concurrent authentications in the largest biometric database in the worldIn India, there is no social security card. It’s difficult for the average citizen to set up a bank account, access benefit programs, and enjoy economic mobility. It’s difficult for the government as well with over a $1B of government aid classified as leakage, the result of fraud and corruption. The Aadhaar program is poised to change all that by leveraging the unique IDs that all people are born with. The program aims to get fingerprints and retina scans for all 1.2 billion citizens. The scale of this project required an integrated in-Hadoop database that was capable of 200 millisecond response times while supporting millions of concurrent look-ups.