Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Andy Feng, Distinguished Architect, Yahoo at MLconf SF

3,531 views

Published on

Abstract: Scalable Machine Learning at Yahoo

Yahoo scientists have developed variety of machine learning libraries (supervised learning, unsupervised learning, deep learning) for online search, advertising and personalization. The emerging business needs require us to address 2 problems:

- Can we apply these libraries against massive datasets (billions of training examples, and millions of features) using commodity hardware clusters?
- Can we reduce the learning time from days to minutes or seconds?

We have thus examined system architecture options (including Hadoop, Spark and Storm), and developed a fault-tolerant MPI solution that allows hundreds of machines to jointly build a model. We are collaborating with open source community for a better system architecture for next-gen machine learning applications. Yahoo ML libraries are being revised for much better scalability and latency. In the talk, we will share system architecture of our ML platform and its use cases.

Published in: Technology
  • Be the first to comment

×