These are slides from a lecture I gave at the School of Applied Sciences in Münster. In this lecture, I talked about **Real-World Data Science** at showed examples on **Fraud Detection, Customer Churn & Predictive Maintenance**.

1. Real-World Data Science
Fraud Detection, Customer Churn & Predictive Maintenance
Dr. Shirin Glander

2. About me

3. Fraud Detection

4. What is fraud and why is it interesting for Data
Science?
“The crime of getting money by deceiving people” (Cambridge Dictionary)
E-commerce and online transactions pose risks for companies

5. What is fraud and why is it interesting for Data
Science?
“The crime of getting money by deceiving people” (Cambridge Dictionary)
E-commerce and online transactions pose risks for companies
Traditional Fraud Detection
Rule-based
Inflexible
Leads to arms-race between companies and fraudsters

6. Leveraging Big Data and Machine Learning in
modern Fraud Detection
Collect information about customers & transactions
Learn from this data

7. Leveraging Big Data and Machine Learning in
modern Fraud Detection
Collect information about customers & transactions
Learn from this data

8. Unsupervised
Data without labels or class
information
Aim
Finding similar instances or
patterns
Ranking instances
Finding outliers
The two main approaches in Machine Learning

9. Unsupervised
Data without labels or class
information
Aim
Finding similar instances or
patterns
Ranking instances
Finding outliers
Supervised
Data with labels or class
information
Aim
Classification
Regression
Forecasting
The two main approaches in Machine Learning

10. What can Machine Learning do?

11. Exploring a fraud example dataset
Synthetic financial dataset for fraud detection from Kaggle1
6,362,620 transactions over 30 days
Fraudulent (Class = 1) & regular (Class = 0) transactions.
[1] https://www.kaggle.com/ntnu-testimon/paysim1

12. Exploring a fraud example dataset
Synthetic financial dataset for fraud detection from Kaggle1
6,362,620 transactions over 30 days
Fraudulent (Class = 1) & regular (Class = 0) transactions.
[1] https://www.kaggle.com/ntnu-testimon/paysim1

13. We can already learn a lot just by looking at the
data

14. Fraud Detection with unsupervised
dimensionality reduction
Identify hidden patterns

15. Principal Component
Analysis (PCA)
Linear relationships between
features
Reflects the majority of
variation in our data
Shows sample dissimilarity
Fraud Detection with unsupervised
dimensionality reduction
Identify hidden patterns

16. t-distributed Stochastic
Neighbor Embedding (t-
SNE)
Complex relationships between
features
Shows also similarity by
clustering samples
Fraud Detection with unsupervised
dimensionality reduction
Identify hidden patterns

17. Fraud Detection with supervised Machine
Learning
Train a classification model on the class label e.g. with Gradient Boosting1 or
Random Forest2
[1] https://shirinsplayground.netlify.com/2018/11/ml_basics_gbm/
[2] https://shirinsplayground.netlify.com/2018/10/ml_basics_rf/

18. Why Fraud Detection is particularly tricky
Fraud is rare => highly unbalanced class distribution
This makes using normal approaches and algorithms difficult
Preprocessing/balancing brings other problems
Prediction accuracy is misleading (more on that later)

19. Why Fraud Detection is particularly tricky
Fraud is rare => highly unbalanced class distribution
This makes using normal approaches and algorithms difficult
Preprocessing/balancing brings other problems
Prediction accuracy is misleading (more on that later)
Fraud is hard to detect => high number of mislabelled data
Fraud methods change often and unpredictably
Simple fraud is easy to detect with rules
Unknown/new fraud methods require Machine Learning

20. Fraud Detection with unsupervised Anomaly
Detection
If we assume that fraud is sufficiently different from regular transactions,
unsupervised learning will flag them as anomalies

21. Fraud Detection with unsupervised Anomaly
Detection
If we assume that fraud is sufficiently different from regular transactions,
unsupervised learning will flag them as anomalies
Anomaly detection with Deep Learning autoencoders

22. What are Neural Nets
A particular type of algorithm for Machine Learning
Used in supervised and unsupervised learning

23. What are Neural Nets
A particular type of algorithm for Machine Learning
Used in supervised and unsupervised learning
Modeled after the human brain

24. How Neural Nets learn classification tasks
Input: instances + features (data)
Output: class label
Neural Nets learn by optimizing weights (and biases)
Activation functions
Minimizing loss functions
Backpropagation
Optimization, e.g. with Gradient Descent

25. How Neural Nets learn classification tasks
Input: instances + features (data)
Output: class label
Neural Nets learn by optimizing weights (and biases)
Activation functions
Minimizing loss functions
Backpropagation
Optimization, e.g. with Gradient Descent
More info: https://shirinsplayground.netlify.com/2018/11/neural_nets_explained/

26. The most important things to know about Deep
Learning

27. Deep Learning vs. Machine Learning

28. Deep Learning autoencoders
Anomaly detection
Semi-supervised: input = output
Minimizing reconstruction error or loss
Outlier samples will have a larger reconstruction error1
[1] https://shiring.github.io/machine_learning/2017/05/01/fraud &
https://shiring.github.io/machine_learning/2017/05/02/fraud_2

29. Deep Learning autoencoders
Anomaly detection
Semi-supervised: input = output
Minimizing reconstruction error or loss
Outlier samples will have a larger reconstruction error1
[1] https://shiring.github.io/machine_learning/2017/05/01/fraud &
https://shiring.github.io/machine_learning/2017/05/02/fraud_2

30. Pre-training supervised models with
autoencoders
Using the weights from the autoencoder model for initialising the training of
a supervised classification model

31. A few words on performance evaluation
Accuracy is easy to understand but not necessarily correct!

32. A few words on performance evaluation
Accuracy is easy to understand but not necessarily correct!
Performance is evaluated on an (independent) test set

33. A few words on performance evaluation
Accuracy is easy to understand but not necessarily correct!
Performance is evaluated on an (independent) test set
We basically want to know:
How many predictions were correct? (Accuracy)
BUT: With < 1% of fraud cases, a model that never identifies instances as fraud
would still achieve a > 99% accuracy.

34. Precision
How many predicted fraud cases in
the test data were correctly
classified?
So, what's a better way to measure
performance?

35. Precision
How many predicted fraud cases in
the test data were correctly
classified?
Sensitivity or recall
How many fraud cases in the test
data were correctly classified as
fraud?
So, what's a better way to measure
performance?

36. Precision
How many predicted fraud cases in
the test data were correctly
classified?
Sensitivity or recall
How many fraud cases in the test
data were correctly classified as
fraud?
So, what's a better way to measure
performance?
F1-Score
Harmonic average of precision and recall:
F1 = (
recall −
1 + precision −
1
2
) = 2 ∗
precision ∗ recall
precision + recall

37. A Shiny demo
https://shiring.shinyapps.io/fraud_example_dashboard/

38. Customer Churn

39. What is Customer Churn?
Loss of revenue due to customers cancelling contract, not returning, etc.
Affects e-commerce, telecommunication, internet service providers and other
businesses dealing with customers

40. What is Customer Churn?
Loss of revenue due to customers cancelling contract, not returning, etc.
Affects e-commerce, telecommunication, internet service providers and other
businesses dealing with customers
Goal
Identify customers that are likely to churn
And try to prevent churn with targeted actions, like ads

41. What is Customer Churn?
Loss of revenue due to customers cancelling contract, not returning, etc.
Affects e-commerce, telecommunication, internet service providers and other
businesses dealing with customers
Goal
Identify customers that are likely to churn
And try to prevent churn with targeted actions, like ads
=> Reduce the proportion of churned customers!

42. How can Machine Learning help prevent churn?
Learn from historical data about customer behavior

43. How can Machine Learning help prevent churn?
Learn from historical data about customer behavior
Generate and use predictive models that put customers into segments:
1. Customers with low risk of churn; generally don't need particular attention;
attentions could even be harmful and costs time and money;

44. How can Machine Learning help prevent churn?
Learn from historical data about customer behavior
Generate and use predictive models that put customers into segments:
1. Customers with low risk of churn; generally don't need particular attention;
attentions could even be harmful and costs time and money;
2. Customers with high risk of churn; (targeted) advertisement or other actions
are advised;

45. How can Machine Learning help prevent churn?
Learn from historical data about customer behavior
Generate and use predictive models that put customers into segments:
1. Customers with low risk of churn; generally don't need particular attention;
attentions could even be harmful and costs time and money;
2. Customers with high risk of churn; (targeted) advertisement or other actions
are advised;
3. Customers with particularly high 'value'; investment into preventing churn will
be especially important to maximize revenue;

46. Predicting customer churn
Traditional statistical methods, like logistic regression
Machine Learning approaches, e.g. with Random Forests, Gradient Boosting or
Neural Nets

47. Predicting customer churn
Traditional statistical methods, like logistic regression
Machine Learning approaches, e.g. with Random Forests, Gradient Boosting or
Neural Nets
Supervised learning
Classification: segment customers into groups, e.g. "Churn" vs. "No Churn"
Regression: predict churn probability for each customer

48. A customer churn case-study1
Telco Customer Churn dataset from IBM Watson2
[1] https://shirinsplayground.netlify.com/2018/12/customer_churn_code/
[2] https://www.ibm.com/communities/analytics/watson-analytics-blog/predictive-
insights-in-the-telco-customer-churn-data-set/

49. A customer churn case-study1
Telco Customer Churn dataset from IBM Watson2
7,043 customers
19 feature
1 response variable: Churn ("No Churn": 5174, "Churn": 1869)
[1] https://shirinsplayground.netlify.com/2018/12/customer_churn_code/
[2] https://www.ibm.com/communities/analytics/watson-analytics-blog/predictive-
insights-in-the-telco-customer-churn-data-set/

50. A customer churn case-study1
Telco Customer Churn dataset from IBM Watson2
7,043 customers
19 feature
1 response variable: Churn ("No Churn": 5174, "Churn": 1869)
[1] https://shirinsplayground.netlify.com/2018/12/customer_churn_code/
[2] https://www.ibm.com/communities/analytics/watson-analytics-blog/predictive-
insights-in-the-telco-customer-churn-data-set/

52. Modeling customer churn with Keras &
TensorFlow
Open-Source Deep Learning API
Train and prototype neural nets fast & easy
Modular layer structure makes it very flexible
Mathematical operations done by backend, e.g. TensorFlow (default)
https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html

53. Training with Keras & TensorFlow
1. Initialising sequential model
2. Adding layers
3. Compiling model
4. Fitting model

54. Training with Keras & TensorFlow
1. Initialising sequential model
2. Adding layers
3. Compiling model
4. Fitting model

55. H2O.ai
Open-Source Machine Learning
API
Optimized for distributed and
parallel training
Stacked Ensembles mit H2O

56. H2O.ai
Open-Source Machine Learning
API
Optimized for distributed and
parallel training
Stacked Ensembles mit H2O
Stacked Ensembles
Combined models tend to perform better than individual models!

57. Precision
How many predicted fraud cases in
the test data were correctly
classified?
Sensitivity or recall
How many fraud cases in the test
data were correctly classified as
fraud?
Evaluation of customer churn models
Rinse and repeat
F1-Score
Harmonic average of precision and recall

58. Evaluation of customer churn models
Area under the ROC curve
Receiver Operator Characteristic
Proportion false positives vs. true positives

59. Thresholds influence prediction accuracy
In binary classification: default = 0.5
Best threshold depends on task and costs of wrong predictions!

60. Thresholds influence prediction accuracy
In binary classification: default = 0.5
Best threshold depends on task and costs of wrong predictions!

61. Thresholds influence prediction accuracy

62. Confusion matrices
Table of wrong and correct predictions
actual predict_0 predict_1 error rate
0 667 107 0.14 107/774
1 86 194 0.31 86/280
Total 753 301 0.18 193/1054

63. Confusion matrices
Table of wrong and correct predictions
actual predict_0 predict_1 error rate
0 667 107 0.14 107/774
1 86 194 0.31 86/280
Total 753 301 0.18 193/1054
But what does that really tell us?
Not that much without a cost/revenue calculation!

64. Cost/revenue calculation
Let’s assume that

65. Cost/revenue calculation
Let’s assume that
1. A marketing campaign + employee time will cost the company 1000€ per year
for every customer that is included in the campaign.

66. Cost/revenue calculation
Let’s assume that
1. A marketing campaign + employee time will cost the company 1000€ per year
for every customer that is included in the campaign.
2. The annual average revenue per customer is 2000€ (in more complex
scenarios customers could be further divided into revenue groups to calculate
how “valuable” they are and how harmful loosing them would be)

67. Cost/revenue calculation
Let’s assume that
1. A marketing campaign + employee time will cost the company 1000€ per year
for every customer that is included in the campaign.
2. The annual average revenue per customer is 2000€ (in more complex
scenarios customers could be further divided into revenue groups to calculate
how “valuable” they are and how harmful loosing them would be)
3. Investing into unnecessary marketing doesn’t cause churn by itself (i.e. a
customer who isn’t going to churn isn’t reacting negatively to the add campaign -
which could happen in more complex scenarios).

68. Cost/revenue calculation
Let’s assume that
1. A marketing campaign + employee time will cost the company 1000€ per year
for every customer that is included in the campaign.
2. The annual average revenue per customer is 2000€ (in more complex
scenarios customers could be further divided into revenue groups to calculate
how “valuable” they are and how harmful loosing them would be)
3. Investing into unnecessary marketing doesn’t cause churn by itself (i.e. a
customer who isn’t going to churn isn’t reacting negatively to the add campaign -
which could happen in more complex scenarios).
4. Without a customer churn model the company would target half of their
customer (by chance) for ad-campaigns

69. Cost/revenue calculation
Let’s assume that
1. A marketing campaign + employee time will cost the company 1000€ per year
for every customer that is included in the campaign.
2. The annual average revenue per customer is 2000€ (in more complex
scenarios customers could be further divided into revenue groups to calculate
how “valuable” they are and how harmful loosing them would be)
3. Investing into unnecessary marketing doesn’t cause churn by itself (i.e. a
customer who isn’t going to churn isn’t reacting negatively to the add campaign -
which could happen in more complex scenarios).
4. Without a customer churn model the company would target half of their
customer (by chance) for ad-campaigns
5. Without a customer churn model the company would lose about 25% of their
customers to churn

70. Cost/revenue calculation
This would mean that compared to no intervention we would have

71. Cost/revenue calculation
This would mean that compared to no intervention we would have
prop_p0_true == customers who were correctly predicted to not churn did not
cost anything (no marketing money was spent): +/-0€

72. Cost/revenue calculation
This would mean that compared to no intervention we would have
prop_p0_true == customers who were correctly predicted to not churn did not
cost anything (no marketing money was spent): +/-0€
prop_p0_false == customers that did not churn who are predicted to churn will
be an empty investment: +/-0€ - 1500€

73. Cost/revenue calculation
This would mean that compared to no intervention we would have
prop_p0_true == customers who were correctly predicted to not churn did not
cost anything (no marketing money was spent): +/-0€
prop_p0_false == customers that did not churn who are predicted to churn will
be an empty investment: +/-0€ - 1500€
prop_p1_false == customer that were predicted to stay but churned: -2000€

74. Cost/revenue calculation
This would mean that compared to no intervention we would have
prop_p0_true == customers who were correctly predicted to not churn did not
cost anything (no marketing money was spent): +/-0€
prop_p0_false == customers that did not churn who are predicted to churn will
be an empty investment: +/-0€ - 1500€
prop_p1_false == customer that were predicted to stay but churned: -2000€
prop_p1_true == customers that were correctly predicted to churn:
Let’s say 100% of those could be kept by investing into marketing: +2000€
-1500€
Let’s say 50% could be kept by investing into marketing: +2000€ * 0.5 -1500€
https://shirinsplayground.netlify.com/2018/12/customer_churn_code/

75. Cost/revenue calculation
How much revenue can we gain from predicting customer churn (assuming
conversion rate of 0.7)?

76. Explaining model predictions
https://shirinsplayground.netlify.com/2018/12/customer_churn_code/

77. Predictive Maintenance

78. Predictive Maintenance
Industry 4.0 & IoT
Uses advanced analytics and Machine Learning to optimize machine costs
and output
Monitor machines, e.g with sensors

79. Goal: Predict machine failure
before it happens to
Save costs
Reduce downtime
Optimize asset availability,
productivity and output
quality
Extend machine life
Lower safety risks
Predictive Maintenance
Industry 4.0 & IoT
Uses advanced analytics and Machine Learning to optimize machine costs
and output
Monitor machines, e.g with sensors

80. Goal: Predict machine failure
before it happens to
Save costs
Reduce downtime
Optimize asset availability,
productivity and output
quality
Extend machine life
Lower safety risks
compared to
Preventive Maintenance
Repair at Failure
Predictive Maintenance
Industry 4.0 & IoT
Uses advanced analytics and Machine Learning to optimize machine costs
and output
Monitor machines, e.g with sensors

81. Wind turbines
Motor Engines
Cars
Planes
Industrial machinery
Examples for Predictive Maintenance

82. Predictive Maintenance and Big Data
Relies on large amounts of data collected over long periods of time

83. Predictive Maintenance and Big Data
Relies on large amounts of data collected over long periods of time
Databases
Collect and store data in an efficient and scalable way, e.g. Hadoop, AWS, etc.
(cloud vs locally)

84. Predictive Maintenance and Big Data
Relies on large amounts of data collected over long periods of time
Databases
Collect and store data in an efficient and scalable way, e.g. Hadoop, AWS, etc.
(cloud vs locally)
Modeling
Model scalability and distributed computing, e.g. Apache Spark, AWS, Google
Cloud, H2O, etc.

85. Predictive Maintenance and Big Data
Relies on large amounts of data collected over long periods of time
Databases
Collect and store data in an efficient and scalable way, e.g. Hadoop, AWS, etc.
(cloud vs locally)
Modeling
Model scalability and distributed computing, e.g. Apache Spark, AWS, Google
Cloud, H2O, etc.
Prediction
(Near) real-time analysis! Streaming...

86. Why can Predictive Maintenance be tricky?
Data quality
Raw data usually needs preprocessing & feature extraction

87. Why can Predictive Maintenance be tricky?
Data quality
Raw data usually needs preprocessing & feature extraction
We don't always have (enough or proper) data on machine failure and repairs
(like log files)

88. Why can Predictive Maintenance be tricky?
Data quality
Raw data usually needs preprocessing & feature extraction
We don't always have (enough or proper) data on machine failure and repairs
(like log files)
Feature engineering can become complex & require domain expertise

89. Why can Predictive Maintenance be tricky?
Data quality
Raw data usually needs preprocessing & feature extraction
We don't always have (enough or proper) data on machine failure and repairs
(like log files)
Feature engineering can become complex & require domain expertise
Production may change / vary

90. Why can Predictive Maintenance be tricky?
Data quality
Raw data usually needs preprocessing & feature extraction
We don't always have (enough or proper) data on machine failure and repairs
(like log files)
Feature engineering can become complex & require domain expertise
Production may change / vary
We often don't know the reason for machine failure

91. Why can Predictive Maintenance be tricky?
Data quality
Raw data usually needs preprocessing & feature extraction
We don't always have (enough or proper) data on machine failure and repairs
(like log files)
Feature engineering can become complex & require domain expertise
Production may change / vary
We often don't know the reason for machine failure
Production environment is usually not robust

92. Why can Predictive Maintenance be tricky?
Data quality
Raw data usually needs preprocessing & feature extraction
We don't always have (enough or proper) data on machine failure and repairs
(like log files)
Feature engineering can become complex & require domain expertise
Production may change / vary
We often don't know the reason for machine failure
Production environment is usually not robust
Fail score is predicted for an interval of time in the future
Probabilistic models (Bayesian)

93. Predicting machine failure
An example analysis
Data set for predictive maintenance
Machine failure of 1000 instances
Labels: broken (1: 397 instances, or 39.7%) & not broken (0: 603 instances,
or 60.3%)
6 features

94. Time-series forecasting
Classification
Regression
Anomaly detection
Survival Analysis
Predicting machine failure
An example analysis
Data set for predictive maintenance
Machine failure of 1000 instances
Labels: broken (1: 397 instances, or 39.7%) & not broken (0: 603 instances,
or 60.3%)
6 features

95. Time-series forecasting
Classification
Regression
Anomaly detection
Survival Analysis
Predicting machine failure
An example analysis
Data set for predictive maintenance
Machine failure of 1000 instances
Labels: broken (1: 397 instances, or 39.7%) & not broken (0: 603 instances,
or 60.3%)
6 features

96. Same techniques as used before
in fraud detection & customer
churn
Classification: Predict whether
machine will likely break in a
given period of time
Regression: Calculate fail score
(probability of breakage in given
period of time)
Predictive Maintenance with Machine Learning

97. Same techniques as used before
in fraud detection & customer
churn
Classification: Predict whether
machine will likely break in a
given period of time
Regression: Calculate fail score
(probability of breakage in given
period of time)
Predictive Maintenance with Machine Learning

98. Predictive Maintenance with Survival Analysis
Cox Proportional Hazards regression model

99. Predictive Maintenance with Time-Series
Forecasting
Facebook's prophet algorithm1
logistic growth curve trend
yearly & weekly seasonal components + holidays
https://research.fb.com/prophet-forecasting-at-scale/

100. Predictive Maintenance with Time-Series
Forecasting
Facebook's prophet algorithm1
logistic growth curve trend
yearly & weekly seasonal components + holidays
Moving averages, ARIMA, ARFIMA, etc.
https://research.fb.com/prophet-forecasting-at-scale/

101. Predictive Maintenance with Time-Series
Forecasting
Facebook's prophet algorithm1
logistic growth curve trend
yearly & weekly seasonal components + holidays
Moving averages, ARIMA, ARFIMA, etc.
https://research.fb.com/prophet-forecasting-at-scale/

102. Evaluating Predictive Maintenance
Cost/benefit calculation
Similar to the examples before:
Probability of detection = recall
Probability of false alarm = fallout

103. ROC curve:
Evaluating Predictive Maintenance
Cost/benefit calculation
Similar to the examples before:
Probability of detection = recall
Probability of false alarm = fallout

104. ROC curve:
Optimize model to
=> avoid false alarms but maximize
true alarms
iterative Monte-Carlo-
Simulation to compare
costs/benefits with old strategies
Evaluating Predictive Maintenance
Cost/benefit calculation
Similar to the examples before:
Probability of detection = recall
Probability of false alarm = fallout

105. A Shiny demo
https://shiring.shinyapps.io/demo_dashboard_pred_maint_fast/

106. Thank you and keep in touch :-)
shirin.glander@gmail.com
shirin-glander.de
Twitter: @ShirinGlander
Github: @ShirinG
codecentric.ai
PS: Werkstudenten, Praktikanten, Bachelor-/Masterarbeiten, etc. welcome!