•11 likes•7,205 views

Report

Share

Download to read offline

Distributed machine learning

Follow

- 1. DISTRIBUTED MACHINE LEARNING STANLEY WANG SOLUTION ARCHITECT, TECH LEAD @SWANG68 http://www.linkedin.com/in/stanley-wang-a2b143b
- 2. What is Machine Learning?
- 3. Mathematics 101 for Machine Learning
- 5. Types of Machine Learning
- 6. Types of ML Algorithms • Clustering • Association learning • Parameter estimation • Recommendation engines • Classification • Similarity matching • Neural networks • Bayesian networks • Genetic algorithms
- 7. Top Machine Learning Algorithms
- 9. Typical Machine Learning Cases
- 10. Machine Learning Customers Examples
- 11. Machine Learning in Big Data Infrastructure
- 12. Big Data Machine Learning Pipeline
- 13. Benefits of Big Data Machine Learning
- 17. Distributed ML Framework • Data Centric: Train over large data Data split over multiple machines Model replicas train over different parts of data and communicate model information periodically • Model Centric: Train over large models Models split over multiple machines A single training iteration spans multiple machines • Graph Centric: Train over large graphs Partitions data as graph associated with every vertex/edge; Parallel apply update functions are operations on a vertex and transforming data in the scope of the vertex;
- 19. Data Parallel ML - MapReduce
- 25. Model Parallel ML – Parameter Server
- 26. Graph Parallel ML – BSP, Pregel, GAS
- 28. Graph Parallel vs Data Parallel Graph parallel is new technique to partition and distribute data and execute machine learning algorithm orders of magnitude faster than data parallel approach!
- 29. Efficient Scaling Up • Businesses Need to Compute Hundreds of Distinct Tasks on the Same Graph o Example: personalized recommendations; Parallelize each task Parallelize across tasks Tas k Tas k Tas k Tas k Tas k Tas k Tas k Tas k Tas k Tas k Tas k Complex Simple Expensive to scale 2x machines = 2x throughput
- 30. Another Approach Task Parallelism: Simple, But Practical • What about scalability? Use cluster of single-machine systems to solve many tasks in parallel, homogeneous graph data, but, heterogeneous algorithm; • What about learning ability? Use hybrid data fusion approach ; • What about memory? Using Parallel Sliding Windows (PSW) algorithm enable computation on very large graphs on disk;
- 31. Parallel Sliding Windows • PSW processes the graph one sub-graph a time: • In one iteration, the whole graph is processed. – And typically, next iteration is started.
- 32. Scalable Distributed ML Frameworks • Yahoo Vowpal Wabbit - Fast and scalable out-of-core online ML algorithms library ; Hadoop compatible Allreduce; • Hadoop Mahout - Scalable Java ML library using map/reduce paradigm; supports 3”C”s+Extras use cases; • Spark MLlib - Memory based distributed ML framework; 10 times as fast as Mahout and even scales better; • Dato GraphLab - Graph based parallel ML framework; • Apache Giraph: Bulk Synchronous Parallel (BSP) based large scale graph processing framework ; • CMU Parameter Server - Distributed ML framework; • CMU Petuum - Iterative-Convergent Big ML ; • 0xdata H2O - Scalable memory efficient deep learning system;
- 33. ML and Big Data is Breakthrough