Краткое введение в Spark и Machine Learning с использованием Spark на примере подбора рекламной кампании для целевой аудитории.

1. DIFFERENCE BETWEEN
SPARK AND HADOOP
MAPREDUCE

2. SPARK IS MUCH FASTER
Spark tries to keep things in memory, whereas MapReduce keeps shuffling things in and out of disk.

3. LOGISTICS REGRESSION
PERFORMANCE

4. WORDCOUNT WITH HADOOP

5. WORDCOUNT WITH SPARK
It’s easier to develop for Spark.

6. Spark also adds libraries for doing things like machine learning, streaming, graph programming and SQL

7. SPARK GENERAL FLOW

8. SOME ACTIONS AND
TRANSFORMATIONS
map(func)
flatMap(func)
froupByKey()
reduceByKey(func)
mapValues(func)
sample(…)
union(other)
distinct()
sortByKey()
..
reduce(func)
collect()
count()
first()
take(n)
saveAsTextFile(path)
countByKey()
foreach(func)
…

9. CREATE INPUT RDDs

10. SPLIT INTO TRAINING,
VALIDATION AND TEST DATASETS
FIND OUT OPTIMAL RANK AND
NUMBER OF ITERATIONS

11. RMSE (ROOT MEAN SQUARE ERROR)
CALCULATION METHOD
EVALUATE THE BEST MODEL
ON THE TEST SET

12. CREATE A NAIVE BASELINE AND COMPARE
IT WITH THE BEST MODEL
OUTPUT

13. RECOMMEND SOME NEW PRODUCTS FOR
USER WITH ID #150
AND SOME OUTPUT...

14. USER ALREADY REACTED ON
SOME CAMPAIGNS

15. USE THIS INFORMATION FOR PREDICTION
AND SOME OUTPUT...

16. RDD FAULT TOLERANCE

17. SPARK DEPLOYMENT

18. MACHINE LEARNING
Types of Machine Learning

20. ALS Algorithm

21. ALS MODEL AND ALGORITHM
Model Ratings as product of User (A) and Movie Feature (B) matrices of size UxK and MxK
Alternating Least Squares (ALS)
• Start with random A nd B vectors
• Optimize user vectors (A) based on movies
• Optimize movie vectors (B) based on users
• Repeat until converged

22. ALS ALGORITHM