This document discusses using Apache Spark for large scale machine learning problems with big data. It describes how Spark can be used to parallelize training and prediction tasks across large datasets that do not fit in memory. Spark allows using scikit-learn algorithms for machine learning tasks on big data by running the algorithms in a distributed manner across a Spark cluster. It also discusses alternative approaches to large scale machine learning, such as online/partial learning with stochastic gradient descent.

1. Data science using Big Data.
Pragmatic approach.
Pavel Mesentsev
Dmitri Babaev
Tinkoff Bank

2. Machine learning in business

3. Other ML tasks examples

4. Size of data
~ 1 mln obj
~ 1 bln obj
~ 1 trln obj
Small Data
Big Data
Huge Data

5. Predictive models design

6. Small data case

10. Big data case

11. Big data case

12. Apache Spark case

13. How does its works

14. How does its works

15. How does its works

16. How does its works

17. How does its works

18. How does its works

19. How does its works

20. Spark Sql

21. Spark Sql

22. Spark Sql

23. Spark Sql

24. Spark Sql

25. Spark Sql

26. Spark Sql

27. Spark Sql

28. Spark Sql

29. Apache Spark case

30. Apache Spark
in
Large Scale Machine Learning

31. LSML issues
• Too much samples to classify
• Training data does not fit in memory
• Too much training samples
• Too much models to train

32. Why not MLLib?
• MLLib is less stable
• too few algorithms comparing to scikit-learn
• ML pipelines are not so mature than in scikit-learn
• e. g. there is no simple way to use logistic regression for feature selection
• MLLib python API falls behind Java/Scala API
• MLLib is actively developed and may be feasible choice in near future

33. Spark + scikit-learn = ?
• Parallel training
• meta-parameter grid search
• parallel one-vs-rest for multi-class models
• same features but different targets
• parallel bagging and ensembles
• parallel learning for multi-step classification
• Parallel prediction

34. Distributed learning

35. Distributed learning

36. Distributed learning

37. Distributed learning

38. Distributed learning

39. Distributed learning

40. Distributed learning

41. Distributed learning

42. Distributed learning

43. Distributed prediction

44. Distributed prediction

45. Distributed prediction

46. Distributed prediction

47. Distributed prediction

48. Distributed prediction

49. Distributed prediction

50. Models management
• A complex ML task can be expressed as scikit-learn
pipeline
•e. g. feature scaling, then LR feature selection then
GBT classification
• Trained models can be stored on HDFS / S3 along with
training report and loaded for prediction

51. Alternative approaches to LSML
• partial / online learning
•e. g. stochastic gradient descent
• distributed stochastic gradient descent
•is implemented in MLLib

52. Spark is everywhere
• Mahout on spark aka Samsara
• H20 Sparkling water

53. Questions?
Pavel Mezentsev pavel@mezentsev.org
Dmitri Babaev dmitri.lb@gmail.com