Download to read offline
Uploaded byNikhil Ketkar
Predictive Models at Scale
Dumbo is a Python framework that allows Python code to run on Hadoop for predictive modeling at scale. It bridges the gap between the Python data science ecosystem and the Hadoop ecosystem by enabling Python code to run as MapReduce jobs. This allows leveraging Python libraries for machine learning and data analysis while taking advantage of Hadoop's distributed processing. The presentation discusses challenges with existing approaches, introduces Dumbo, and walks through an example of using it to extract model part numbers from product titles at scale.
Predictive Models at Scale
- 1. Predictive Models at Scaleusing Dumbo Nikhil Ketkar
- 2. 40k+ Brands 600k+ Sellers 700+Million Products 7k+ Categories 10k+ Attributes Motivation: Problem Space @ Indix
- 3. Developing Predictive Models UnlabelledData Sample Hand Label Model Predict Data with Predicted Labels
- 4.
- 5. The Two Giants Native,C/C++ Fortran Numpy Scipy, Pandas, Matplotlib scikit-learn, scikit-image, statsmodels JVM Java/Scala HDFS, Hadoop MapReduce Cascading/Scalding PyData Ecosystem Hadoop Ecosystem Model Predict
- 6. The Standard Options ●Port to Java/Scala use as Library in Mapper ○ Time Consuming ○ Need to port parts of the PyData Stack ○ Reduced Velocity ○ Error prone ● Write a REST API/Service for the model and call from Mapper ○ Slow due to Network Latency ○ Deployment is a nightmare ● Use Disco
- 7. Can we dobetter? ● Hadoop Streaming with Typedbytes Support ● Python Wrappers over Hadoop Streaming ○ Dumbo ○ MRJob ○ Hadoopy ○ Pydoop Reference: http://blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/
- 8. Two Minute MapReduceRefresher Reference: https://tarnbarford.net/journal/mapreduce-on-mongo
- 9. Sample Problem: ExtractMPN from Product Titles ● 0.5 Billion Product Titles ● Many contain MPNs ● Humans can detect MPNs ● Can a model do the same? ● Use CRF on Full Title ● Use RF on Tokens Moen CSIMC000BN Brushed Nickel Decorative Mirror Frame Corner Rosette from Mirrorscapes 000 Series Set of 4 Rohl A3608/6.5LPAPC 2 Polished Chrome Country Kitchen Low Lead Bar Faucet with Porcelain Lever Handle Newport Brass 3 447/ORB Oil Rubbed Bronze Hand Relieved Diverter / Volume Control Handle from the Metropole Collection Bosch HCFC2044B 1/4" SDS Plus X5L with Optimized Flute Surface Pack of 25 Sterling 7214120 Ensemble 0" x 30" Shower Receptor with Right hand Drain Pack 6 U12 23252 KUB QUATRON INDX DRILL MPNs in Product Titles
- 10.
- 11.
- 12. Important Learnings ● DumboFairly Stable, Mature and Ready for Production ● Gets the 2 giants working together! ● Found just one issue over 6 months of usage (patch submitted) ● Support for Typedbytes is critical if making predictions over binary data (Images etc.)