Data Science Process Walkthrough as part of session hosted by Big data Trunk (www.BigDataTrunk.com) for below Meetup group https://www.meetup.com/Big-Data-IOT-101/ Presented by Antony Ross

1. The Data Science Process
Antony Ross

2. The Data Science Process
1.) Identify a useful question
2.) Acquire the data
3.) Clean the data
4.) Explore the data
5.) Model the data
6.) Communicate the results

3. Identify a Useful Question

4. 2

5. age gender monthly rate
membership
months
cancelled
23 female 10.99 3 no
57 female 12.99 27 no
25 male 12.99 11 yes
37 male 12.99 29 no
44 male 7.99 18 no
62 female 10.99 16 yes
24 female 12.99 5 yes
Customer Data

7. Three Types of Returners
• Always Returners
• Never Returners
• Contingent Returners

9. Get the Data

10. Structured Data

11. Unstructured Data

12. Clean the Data

14. Age
Years Since
Graduation
Income
Loan
Balance
Next
Payment
Defaulted
22 0 39247 14947.63 123 N
37 9 57981 27172.43 365 N
26 5 49031 18102.54 273 Y
42 16 63989 93429.79 491 Y
26 1 41250 13456.54 215 N
29 7 56482 29048.47 316 N
43 9 71202 26279.13 385 N
30 11 27649 6826.10 55 N
Student Loan Data

15. Age
Years Since
Graduation
Income
Loan
Balance
Next
Payment
count 4947 4947 4947 4947 4933
mean 32.17 6.33 57981.98 37172.43 376.51
std 6.12 4.21 9031.62 18102.54 137.09
min 2 0.00 0.00 93.00 5.00
25% 26 3 41250.00 13456.00 112.00
50% 29 7 56482.00 39084.00 381.00
75% 43 9 71202.00 66212.00 485.00
max 67 31 276498.00 269910.00 755.00
Student Loan Data
Summary

16. Clean the Data

17. Loan ID# Graduated Loan Type
Loan
Balance
Next
Payment
Months
Delinquent
Defaulted
10148975 Yes 1 31567 327 0 N
19773966 Yes 3 27909 - 3 N
25220947 Yes 2 11,463 243 25 Y
17090812 No 2 29801 255 15 Y
23956341 Yes 3 18755 173 0 N
12680900 Yes 1 16,211 122 7 N
23435111 No 1 5064 84 0 N
Student Loan Data

18. Graduated Loan Type
Loan
Balance
Next
Payment
Months
Delinquent
Defaulted
Yes 1 31567 327 0 N
Yes 3 27909 - 3 N
Yes 2 11,463 243 25 Y
No 2 29801 255 15 Y
Yes 3 18755 173 0 N
Yes 1 16,211 122 7 N
No 1 5064 84 0 N
Student Loan Data

19. Graduated Loan Type
Loan
Balance
Next
Payment
Months
Delinquent
Defaulted
Yes 1 31567 327 0 N
Yes 3 27909 - 3 N
Yes 2 11463 243 25 Y
No 2 29801 255 15 Y
Yes 3 18755 173 0 N
Yes 1 16211 122 7 N
No 1 5064 84 0 N
Student Loan Data

20. Graduated Loan Type
Loan
Balance
Next
Payment
Months
Delinquent
Defaulted
Yes 1 31567 327 0 N
Yes 3 27909 NaN 3 N
Yes 2 11463 243 25 Y
No 2 29801 255 15 Y
Yes 3 18755 173 0 N
Yes 1 16211 122 7 N
No 1 5064 84 0 N
Student Loan Data

21. Transform the Data

22. Graduated Loan Type
Loan
Balance
Next
Payment
Months
Delinquent
Defaulted
Yes 1 31567 327 0 N
Yes 3 27909 NaN 3 N
Yes 2 11463 243 25 Y
No 2 29801 255 15 Y
Yes 3 18755 173 0 N
Yes 1 16211 122 7 N
No 1 5064 84 0 N
Student Loan Data

23. Graduated Loan Type
Loan
Balance
Next
Payment
Months
Delinquent
Defaulted
1 1 31567 327 0 N
1 3 27909 NaN 3 N
1 2 11463 243 25 Y
0 2 29801 255 15 Y
1 3 18755 173 0 N
1 1 16211 122 7 N
0 1 5064 84 0 N
Student Loan Data

24. One-Hot Encoding

25. Graduated Loan 1 Loan 2 Loan 3
Loan
Balance
Next
Payment
Months
Delinquent
Defaulted
1 1 0 0 31567 327 0 N
1 0 0 1 27909 NaN 3 N
1 0 1 0 11463 243 25 Y
0 0 1 0 29801 255 15 Y
1 0 0 1 18755 173 0 N
1 1 0 0 16211 122 7 N
0 1 0 0 5064 84 0 N
Student Loan Data

26. Graduated Loan 1 Loan 2 Loan 3
Loan
Balance
Next
Payment
Months
Delinquent
Defaulted
1 1 0 0 31567 327 0 0
1 0 0 1 27909 NaN 3 0
1 0 1 0 11463 243 25 1
0 0 1 0 29801 255 15 1
1 0 0 1 18755 173 0 0
1 1 0 0 16211 122 7 0
0 1 0 0 5064 84 0 0
Student Loan Data

27. Pair Plot

28. Explore the Data

31. Select Features
Song Duration Pitch Timbre Tempo Popularity Genre

32. Timbre in a selected track

36. Top 30 Genres in My “Liked” and “Rejected” Playlists

38. Model the Data

39. Logistic Regression
Naive Bayes
Support Vector Machine
Decision Tree
K-Nearest Neighbor
K-Means
Neural Network
Common Machine Learning Algorithms

40. Test/Train Split

41. Test Set
70% 30%
DATA SET
Training Set

42. 70%
DATA SET
Training Set

43. Model Selection

44. 10-Fold Cross Validation
1 2 3 4 5 6 7 8 9 10
Training Set
evaluation

45. 10-Fold Cross Validation
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
Training Set
evaluation

46. 10-Fold Cross Validation
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
Training Set
evaluation

47. 10-Fold Cross Validation
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
Training Set

48. Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 6 Fold 7 Fold 8 Fold 9 Fold 10 mean
0.69 0.64 0.73 0.82 0.64 0.70 0.68 0.71 0.70 0.69 0.70
Logistic Regression

49. Logistic
Regression
Support Vector
Machine
Decision Tree
K-Nearest
Neighbor
Naive Bayes
0.705 0.722 0.635 0.675 0.607

50. 1 2 3 4 5 6 7 8 9 10
Training Set

51. Test Set
70% 30%
DATA SET
Training Set

52. Test Set
30%
DATA SET

53. Confusion Matrix

54. trueclass
predicted class

55. trueclass
predicted class
negative class

56. trueclass
predicted class
positive class

57. Confusion Matrix
TN FPTrue Negative False Positive
FNFalse Negative
TPTrue Positive
Negative Class
Positive Class
Predicted Negative Predicted Positive

58. Model Evaluation Metrics

59. Accuracy
0.796
Spotify “Liked” Classification

60. Precision Recall F1-score
Rejected 0.89 0.72 0.79
Liked 0.73 0.90 0.81
Spotify “Liked” Classification

61. Test Set
100%
DATA SET
Training Set