Data Preprocessing- Data transformation, Scaling, Normalization, Standardization.

1. Data Science
Data Preprocessing
(Data Transformation)

2. Data
Preprocessing
Data
Integration
Data
Transformation
Data
Reduction
or
dimension
reduction
Data
Cleaning
Scaling, Discretization
Categorical Encoding
Handling missing
values Outliers,
duplicates

3. Data Transformation
Data transformation is the process of converting data from one format or
structure into another format or structure for analysis. It is a fundamental
aspect of most data integration and data management tasks such as data
wrangling, data warehousing, data integration and application integration.
The primary goal of data transformation is to make the data more suitable
for the analysis tasks at hand, such as predictive modeling or exploratory
data analysis.
Common techniques used in data transformation include normalization,
standardization, and discretization.

4. Transformation Techniques
1. Scaling
Scaling is the process of transforming the features of a dataset so that they
fall within a specific range. Scaling is useful when we want to compare two
different variables on equal grounds.
The goal of scaling is to ensure that all variables contribute equally to the
analysis, particularly when using algorithms that are sensitive to the scale of
the input features.

5. Methods of Scaling
Scaling
Normalization
(Min-Max Scaling)
Standardization
(Z-Score Scaling)

6. Normalization
Normalization (Min-Max Scaling): Normalization rescales the values of a
feature to fit within a specific range, usually between 0 and 1. The formula
for normalization is:
Age: 25,35,45 Salary: 30000, 50000, 70000
For Age X normalized
Age=25 (25-25) / (45-25) 0
Age=35 (35-25)/(45-25) 0.5
Age=45 (45-25)/ (45-25) 1
Salary Y normalized
30000 0/40000 0
50000 20000/40000 0.5
70000 40000/40000 1

7. Standardization
Standardization (z-score Scaling): Standardization rescales the values of a
feature so that they have a mean of 0 and a standard deviation of 1. The
formula for standardization is:
Age: 25,35,45
μ (Mean Age) = 35 μ (S.D Age) = 8.16
For Age X standardized
Age=25 (25-35) / 8.16 -1.22
Age=35 (35-35)/8.16 0
Age=45 (45-35)/ 8.16 1.22

8. Transformation Techniques
2. Discretization
• This is a process of converting continuous data into a set of data intervals.
Continuous attribute values are substituted by small interval labels. This
makes the data easier to study and analyze. If a data mining task handles a
continuous attribute, then its discrete values can be replaced by constant
quality attributes. This improves the efficiency of the task.
• This method is also called a data reduction mechanism as it transforms a
large dataset into a set of categorical data.
• For example, 25,30,35,40,45,50,55,60,65,70 The values for the age
attribute can be replaced by the interval labels such as (25-40 : Young,
41-60 : Adult, 61-70 : Senior)

9. Transformation Techniques
3. Data Aggregation
• Data collection or aggregation is the method of storing and presenting
data in a summary format. The data may be obtained from multiple data
sources to integrate these data sources into a data analysis description.
This is a crucial step since the accuracy of data analysis insights is highly
dependent on the quantity and quality of the data used.
• For example, Sales, data may be aggregated to compute monthly &
annual total amounts.

10. Transformation Techniques
4. Encoding categorical variables
It involves transforming categorical variables into a numerical format
suitable for machine learning format. There are several methods for
encoding categorical variables, two common approaches are:
• One-hot encoding
• Label encoding.

11. One-Hot Encoding
• One-hot encoding transforms each categorical variable into a binary
vector where each category is represented by a binary bit. Each category
is represented by a binary bit, with a 1 indicating the presence of the
category and a 0 indicating absence.
• This method creates additional columns, one for each unique category,
which can lead to a high-dimensional dataset. It's suitable for categorical
variables with a relatively small number of unique categories.
• Example:
• Original categorical variable: { "Red", "Blue", "Green" }
• One-hot encoded variables:
• "Red" : [1, 0, 0]
• "Blue" : [0, 1, 0]
• "Green" : [0, 0, 1]

12. Label Encoding
• Label encoding assigns a unique numerical label to each category. It
replaces each category with its corresponding numerical label.
• This method does not create additional columns but can introduce
ordinality among categories, which may not always be desirable.
• It's suitable for categorical variables with ordinal relationships between
categories.
• Example:
• Original categorical variable: { "Red", "Blue", "Green" }
• Label encoded variables:
• "Red" : 0
• "Blue" : 1
• "Green" : 2

13. Thanks for Watching!