introduction of data preprocessing,cleaning,data integration and transformation,no quality data,major tasks of data preprocessing,

1. DATA PREPROCESSING

2. INTRODUCTION
 Data preprocessing is a data mining technique that
involves transformation raw data into an understandable
format.
 Real world data is often
 Inconsistent
 Insufficient
 Lacking in certain behaviors or trends, and is likely to
contain many error.

3. Cleaning
 It is the process of detecting and correcting corrupt or
inaccurate record from a record set, table, or database
and refers to identifying
 Incomplete
 Incorrect
 Inaccurate
 Irrelevant
 part of the data and replaying

4. Data Integration and Transformation
 In computing, data transformation is the process of
converting data from one format or structure into
another format or structure.
 It is fundamental aspect of most data integration and
data management tasks such as
 Data wrangling,
 Data integration
 Application integration.

5. Data Reduction
 Data reduction is the transformation of numerical
or alphabetical digital information derived
empirically or experimentally into a corrected ordered
and simplified form.
 The basic concept is the reduction of multitudinous
amount of data down to the meaningful parts.

6. Cube
 A data cube is generally used to easily interpret data. It
is especially useful clean representation data together
with dimensions as a certain measures of business
requirement.
 A cube’s every dimension represents certain
characteristic of the data base, for example daily,
monthly, or yearly sales.

7. Attributes
 Attributes subset selection is a technique which
process.
 Data reduction reduces the size of data so that it can
be used for analysis purposes more efficiently.
 Need of attribute subset selection.
 The data set may have a large number of attributes.
 Data mining bayesian classification advertisement.
 Bayesian classification is based on bayes theorem.

8. BAYESIAN CLASSIFIER
 Bayesian classifier are the statistial classifiers.
 Bayesian classifier can predicts class membership
probabilities such as the probability that given
tuple belong to a particular class.

10. Data in the real world is dirty
 Incomplete : lacking attribute values, lacking certain
attributes of interest, or containing only aggregate
data.
 Noisy: containing errors or outliers.
 Inconsistent: containing discrepancies in codes or
names.
 Broad categories
 Intrinsic, contextual, representational and
accessibility.

11. No quality data, no quality mining
 Quality decisions must be based on quality data.
 Data warehouse needs consistent integration of
quality data.
• A multi-dimensional measure of data quality
A well-accepted multi-dimensional view:
 Accuracy, completeness, consistency, timeliness,
believability, value added, interpretability,
accessibility.

12. Data Cleaning
 Inconsistent data:
 Manual correction using external references
 Semi-automatic using various tools
- To detect violation of known functional
dependencies and data constraints.
- To correct redundant data

13. Data Reduction
 Manage Data Reduction
 Data reduction: reduced representation, while still
retaining critical information
1. Data cube aggregation
2. Dimensionality reduction
3. Data compression
4. Numerosity reduction
5. Discretization and concept hierarchy generation

14. Data Cleaning
 Tasks of Data Cleaning
a. Fill in missing values
b. Identify outliers and smooth noisy data
c. Correct inconsistent data