Data preprocessing involves cleaning, transforming, reducing, and preparing data for machine learning models. The key goals of preprocessing are to ensure data is in the right format for analysis and modeling. Common techniques include data cleaning such as removing duplicates and dealing with missing values and outliers, as well as data transformation like scaling, normalization, and feature extraction. Proper preprocessing unlocks the power of data for analysis and machine learning.

1. Data Preprocessing: Unlocking the
Power of Manufacturing Data
A Guide to Data Cleaning and Transformation
1

2. 2
What is Data Pre-
processing????
What is it’s
importance???
How do I pre-process a
data???
Role of it in Data
Analysis??
Why is it required??

3. Data Pre-processing
• Involves cleaning, transformation, reduction and preparing data for ML models

4. Goal of Data Pre-processing
• Ensures data is in right format suitable for exploration, analysis and modeling

5. Common Techniques in Pre-processing
• Data Cleaning includes
• Removing duplicates,
• Dealing missing values and outliers
• Correcting data entry errors

6. Common Techniques in Pre-processing
• Data Transformation
Converting data to a format suitable for analysis.
It includes,
• Scaling the data,
• Normalizing the data
• Transformation for normal distribution of data

7. Common Techniques in Pre-processing
• Feature Extraction
Important step to identify & select important features from the data and
extracting them.
It includes,
• Selecting most relevant features,
• Combining multiple into one feature.
• Creating new set of features from existing ones.

8. Common Techniques in Pre-processing
• Data Splitting
Step to separate the entire data into training, validation and test set.
 Training set  To fit the model
 Validation set  To tune hyperparameters
 Test set  To evaluate final performance of the model

9. Data Cleaning – Handling Outliers
• What are Outliers???

10. Data Cleaning – Handling Outliers
• Methods to handle outliers:
 IQR method:
o Involves calculating Interquartile range (IQR)  Difference between 75
percentile and 25 percentile data.
o Outliers = 1.5*(IQR below 1st quartile and above 3rd quartile)
o Outliers identified can either be removed or replaced. How??

11. Data Cleaning – Handling Outliers
• Methods to handle outliers:
 Z score method:
o Involves standardizing data by subtracting mean and dividing standard
deviation.
z=
𝑥−𝜇
𝜎
o Outliers = 3 to 4 times standard deviation from mean value
o Outliers identified can either be removed or replaced. How??

12. Data Transformation – Feature Scaling
• What is Feature scaling???

13. Data Transformation – Feature Scaling
• What is Feature scaling???
Any Example on why scaling
is essential??

14. Data Transformation – Feature Scaling
• Methods of Feature Scaling???

15. Data Transformation – Feature Scaling
• Methods of Feature Scaling???
This scales features by dividing actual value with maximum value of each feature.
𝑥𝑠𝑐𝑎𝑙𝑒𝑑 =
𝑥
max(𝑎𝑏𝑠 𝑥 )
This scales features to range between 0 and 1.
𝑥𝑠𝑐𝑎𝑙𝑒𝑑 =
𝑥 − min(𝑥)
max 𝑥 − min(𝑥)

16. Data Transformation – Feature Scaling
• Methods of Feature Scaling???
This scales features to get a mean 0 and st deviation as 1.
𝑥𝑠𝑐𝑎𝑙𝑒𝑑 =
𝑥 − 𝑚𝑒𝑎𝑛(𝑥)
𝑠𝑡𝑑 𝑥
This scales features to get a mean 0 and variance as 1.
𝑥𝑠𝑐𝑎𝑙𝑒𝑑 =
𝑥 − 𝑚𝑒𝑎𝑛(𝑥)
𝑣𝑎𝑟 𝑥

17. Data Cleaning – Handling Missing values
• What are missing values???

18. Data Cleaning – Handling Missing values
• What are types of missing values???
• Missing completely at random (MCAR):
o Occurs when data missing  Not related to any other variables in a dataset.
o Data missing @ completely random  No systematic pattern
1. Understanding missing data  Important.
2. If data missing randomy still represents population
3. If missing systematically biased for analysis
• Missing at random (MAR):
o Occurs when data missing  Not completely random. But can be explained by other variables.
o MAR provides asymptotically unbiased estimates.
• Missing NOT at random (MNAR):
o Occurs when data missing  Not random. can be explained by other variables.

19. Examples – MCAR, MAR, MNAR
• Test papers of 50 students in a class
Professor while bringing 50 papers loses 3 papers due to sudden wind  MCAR.
Classic case of each student losing his sheet with same probability
Professor getting 49 papers out of 50 students as one student is absent  MAR.
Probability of missing data is due to observed data  Attendance of class  No unobserved
data effect
Professor getting 49 papers out of 50 students as one student is absent because he didn’t
understand topic of exam  MNAR.
Probability of missing data is due to un-observed data  Struggling with exam topics

20. Data Cleaning – Handling Missing values
• How to deal with missing values???

21. Data Cleaning – Handling Missing values
• How to deal with missing values???

22. Data Cleaning – Handling Missing values
• How to deal with missing values???

23. Data Cleaning – Handling Missing values
• How to deal with missing values???

24. Data Cleaning – Handling Missing values
• How to deal with missing values???

25. Data Cleaning – Handling Missing values
• How to deal with missing values???

26. Data Cleaning – Handling Missing values
• How to deal with missing values???