Cvetanka Eftimoska: How we can use PySpark for building and training an ML model Data Science Online Camp 2021 Website - https://dscamp.org/ Youtube - https://www.youtube.com/startuplviv FB - https://www.facebook.com/Data-Science-Camp-103012708431833

1. 1
How we can use PySpark for
building and training an ML
model

2. 2
About me.
Cvetanka Eftimoska
⋮IWConnect, North Macedonia
Senior Technical Consultant

3. 3
KEY FACTS ABOUT
⋮IWConnect .
350+ projects in over 20
industries and over 10 countries
in Europe and USA
11 practices providing
diversified solutions
4 locations: Bitola, Prilep, and
Skopje, North Macedonia and
Amsterdam, Netherlands.
16 years of working
experience

4. 4
The goal of the presentation.
• The concept of AI and Machine Learning
• What is machine learning?
• Types of machine learning
• Machine learning flow
• Define PySpark
• PySpark Mllib
• Data preprocessing
• Linear regression
• GBT Regressor
• Example

5. 5
The concept of AI.

6. 6
What is Machine Learning?

7. 7

8. 8
Machine Learning flow

9. 9
Define PySpark

10. 10
PySpark Mllib

11. 11
Data preprocessing
• Better results
• Formatted Data
• More meaningful data
• Data that can be trusted
• Inaccurate data
• Missing data
• Presence of noisy data
• Inconsistent data

12. 12
Start with PySpark

13. 13
Define the schema

14. 14
Explore the dataset

15. 15
Missing values

16. 16
Missing values

17. 17
Missing values

18. 18
Handle missing values - estimate

19. 19
Handle missing values - estimate
2. Iterative Imputer (Multivariate)
• Models each feature with missing values as
a function of other features

20. 20
Handle missing values - eliminate

21. 21
Tip

22. 22
Engine volume analyses

23. 23
Check the data

24. 24
OLS assumptions – multicollinearity

25. 25
OLS assumptions - multicollinearity

26. 26
OLS assumptions - linearity

27. 27
OLS assumptions - linearity

28. 28
Dealing with outliers

29. 29
Dealing with outliers

30. 30
Dealing with outliers

31. 31
Check the data

32. 32
Categorical features

33. 33
Mean = 0
Standard deviation = 1
𝑥𝑖 − m𝑒𝑎𝑛 𝑥
stdev 𝑥
Standardization

34. 34
Standardization

35. 35
Split into train and test

36. 36
Linear regression

37. 37
Linear regression

38. 38
Linear regression

39. 39
Linear regression

40. 40
GBT Regressor

41. 41
GBT Regressor

42. 42
Conclusion.

43. 43
Thank You!
cvetanka.eftimoska@iwconnect.com
iwconnect life_at_iw
.com/iwconnect
Follow us 👇