This document discusses various techniques for data preprocessing including data cleaning, integration, transformation, reduction, and discretization. Data cleaning involves filling in missing values, smoothing noisy data, identifying outliers, and resolving inconsistencies. Data integration combines data from multiple sources by integrating schemas and resolving value conflicts. Data transformation techniques include normalization, aggregation, generalization, and smoothing. Data reduction aims to reduce the volume of data while maintaining similar analytical results. This includes data cube aggregation, dimensionality reduction by removing unimportant attributes, data compression, and discretization which converts continuous attributes to categorical bins.

1. Data Preprocessing
BY:
K.KOTTAISAMY

2. Data Preprocessing
Data cleaning
Data integration
Data transformation
Data reduction
Data discretization

3. Major tasks in Data Preprocessing
Data cleaning
Fill in missing values, smooth noisy data, identify or remove
outliers, and resolve inconsistencies
Data integration
Integration of multiple databases, data cubes, or files
Data transformation
Normalization and aggregation

4. Data reduction
Reduced representation in volume but produces the same or
similar analytical results
Data discretization
Part of data reduction but with particular
importance, especially for numerical data

5. Data Preprocessing

6. Data Cleaning
Data in the Real World Is Dirty:
Lots of potentially incorrect data, e.g., instrument
faulty, human or computer error, transmission error
incomplete: lacking attribute values, containing only
aggregate data
e.g., Occupation = “ ” (missing data)
noisy: containing noise, errors, or outliers
e.g., Salary = “−10” (an error)

7. inconsistent: containing discrepancies in codes or
names, e.g.,
Age = “42”, Birthday = “03/07/2010”
Was rating “1, 2, 3”, now rating “A, B, C”
discrepancy between duplicate records
Intentional (e.g., disguised missing data)
Jan. 1 as everyone’s birthday?

8. Data Integration
Data integration:
combines data from multiple sources
Schema integration
integrate metadata from different sources
Detecting and resolving data value conflicts
for the same real world entity, attribute values from
different sources are different,
e.g., different scales, metric vs. British units
Removing duplicates and redundant data

9. Data Transformation
Smoothing:
remove noise from data
Aggregation: summarization, data cube construction
Generalization: concept hierarchy climbing
Normalization: scaled to fall within a small, specified
range
min-max normalization
z-score normalization
normalization by decimal scaling

10. Data Transformation: Normalization
Min-max normalization: to [new_minA, new_maxA]
v'
v minA
(new _ maxA new _ minA) new _ minA
maxA minA
Ex.
Let income range $12,000 to $98,000 normalized to
[0.0, 1.0]. Then $73,000 is mapped to
73,600
98,000
12 ,000
(1.0 0)
12 ,000
0
0.716

11. Data Transformation: Normalization
Z-score normalization (μ: mean, σ: standard
deviation):
v'
v
A
A
Ex. Let μ = 54,000, σ = 16,000. Then
73,600 54 ,000
16 ,000
1.225

12. Data Transformation: Normalization
Normalization by decimal scaling
v
v'
10 j
Where j is the smallest integer such that Max(|ν’|) < 1

13. Data reduction
Why data reduction?
A database/data warehouse may store terabytes of data
Complex data analysis/mining may take a very long time to
run on the complete data set
Data reduction
Obtain a reduced representation of the data set that is much
smaller in volume but yet produce the same analytical results

14. Data reduction strategies
Data cube aggregation
Dimensionality reduction — e.g., remove unimportant
attributes
Data Compression
Discretization and concept hierarchy generation

15. Data cube aggregation
The lowest level of a data cube (base cuboid)
The aggregated data for an individual entity of
interest
E.g., a customer in a phone calling data warehouse
Multiple levels of aggregation in data cubes
Further reduce the size of data to deal with

16. Dimensionality Reduction
Feature selection (attribute subset selection):
Select a minimum set of attributes that is sufficient for
the data mining task.
Heuristic methods
step-wise forward selection
step-wise backward elimination
combining forward selection and backward elimination

17. Data Discretization
Three types of attributes:
Nominal — values from an unordered set
Ordinal — values from an ordered set
Continuous — real numbers
Discretization:
Some classification algorithms only accept
categorical attributes.
Reduce data size by discretization
Prepare for further analysis