This document discusses techniques for preprocessing data including data cleaning, transformation, integration, and reduction. Data cleaning removes noise and inconsistencies by filling missing values and identifying outliers. Data integration merges data from multiple sources while dealing with issues like different naming conventions. Data transformation techniques include normalization, aggregation, generalization, attribute selection, and dimensionality reduction to reduce data size and handle inconsistencies. Preprocessing is needed to clean noisy, inconsistent data and handle incomplete data from various sources.

1. Data Preprocessing
V. Saranya
AP/CSE
Sri Vidya College of Engineering & Technology, Virudhunagar

2. Preprocessing
• Databases are so noisy
– By missing values, inconsistent data, huge size.
– From multiple resources
– Low quality data
– Low quality mining results

3. Techniques
1. Data Cleaning
2. Data transformation
3. Data integration
4. Data reduction

4. Data Cleaning
Remove noise
Correct inconsistencies.
By filling missing values,
smoothing noisy data
Identify outliers

5. Data Integration
• Merge data from
multiple resources.
• (DW)
Issue:
• Different names in
different databases
may cause
inconsistencies.

6. Data Transformation
• Normalization
– Ex:
-2,32,100  -0.02,
.32, .00

7. Data Reduction
• Reduce data size by aggregating or
clustering.

8. Data reduction includes
1. Data aggregation:
– Building data
cube

9. 2. Data Generalization
- concept hierarchy

10. 3. Attribute subset selection:
 removing irrelevant attributes through
correlation analysis

11. 4. Dimensionality Reduction:
 minimize the dimensionality

12. 5. Numerosity reduction;
 replacing the data by alternative.

13. Need for preprocessing
• Incomplete, noisy and inconsistent data may
be in large volume of databases.

14. Reason for incomplete data
• Attributes may not be available
• Misunderstanding
• Data may be deleted.
• Missing data
• Fault data
• Errors in data transformation

15. Data Discretization
• Form of data reduction