1.
2
Data Preprocessing
 Data Preprocessing: An Overview
 Data Quality
 Major Tasks in Data Preprocessing
 Data Cleaning
 Data Integration
 Data Reduction
 Data Transformation and Data Discretization
1. Data collection
parameters,
features, variables
• Errors will propagate
If you have error in very beginning (1st step)
1st step  2nd  3rd  4th

2.
Data Quality: Why Preprocess the Data?
 Measures for data quality: A multidimensional view
 Accuracy: correct or wrong, accurate or not
 Completeness: not recorded, unavailable, …
 Consistency: some modified but some not, dangling,
…
 Timeliness: timely update?
 Believability: how trustable the data are correct?
 Interpretability: how easily the data can be
understood?

3.
Major Tasks in Data Preprocessing
 Data cleaning
 Fill in missing values, smooth noisy data, identify or remove
outliers, and resolve inconsistencies
 Data integration
 Integration of multiple databases, data cubes, or files
 Data reduction
 Dimensionality reduction
 Numerosity reduction
 Data compression
 Data transformation and data discretization
 Normalization
 Concept hierarchy generation

4.
5
Histogram Analysis
 Divide data into buckets and
store average (sum) for each
bucket
 Partitioning rules:
 Equal-width: equal bucket
range
 Equal-frequency (or equal-
depth) 0
5
10
15
20
25
30
35
40
10000 30000 50000 70000 90000
Interval of 10000

5.
6
Correlation Analysis (Nominal Data)
 Χ2 (chi-square) test
 The larger the Χ2 value, the more likely the variables
are related
 The cells that contribute the most to the Χ2 value are
those whose actual count is very different from the
expected count
 Correlation does not imply causality
 # of hospitals and # of car-theft in a city are correlated
 Both are causally linked to the third variable: population



Expected
Expected
Observed 2
2 )
(

Features,
attributes or
variable

6.
7
Chi-Square Calculation: An Example
 Χ2 (chi-square) calculation (numbers in parenthesis are
expected counts calculated based on the data distribution
in the two categories)
 It shows that like_science_fiction and play_chess are
correlated in the group
93
.
507
840
)
840
1000
(
360
)
360
200
(
210
)
210
50
(
90
)
90
250
( 2
2
2
2
2










Play chess Not play chess Sum (row)
Like science fiction 250(90) 200(360) 450
Not like science fiction 50(210) 1000(840) 1050
Sum(col.) 300 1200 1500

7.
8
Correlation Analysis (Numeric Data)
 Correlation coefficient (also called Pearson’s product
moment coefficient)
where n is the number of tuples, and are the respective
means of A and B, σA and σB are the respective standard deviation
of A and B, and Σ(aibi) is the sum of the AB cross-product.
 If rA,B > 0, A and B are positively correlated (A’s values
increase as B’s). The higher, the stronger correlation.
 rA,B = 0: independent; rAB < 0: negatively correlated
B
A
n
i i
i
B
A
n
i i
i
B
A
n
B
A
n
b
a
n
B
b
A
a
r



 )
1
(
)
(
)
1
(
)
)(
( 1
1
,








 

A B

8.
Visually Evaluating Correlation
Scatter plots
showing the
similarity from
–1 to 1.
Age
Height
f1
f2

9.
Covariance (Numeric Data)
 Covariance is similar to correlation
where n is the number of tuples, and are the respective mean or
expected values of A and B, σA and σB are the respective standard
deviation of A and B.
 Positive covariance: If CovA,B > 0, then A and B both tend to be larger
than their expected values.
 Negative covariance: If CovA,B < 0 then if A is larger than its expected
value, B is likely to be smaller than its expected value.
 Independence: CovA,B = 0 but the converse is not true:
 Some pairs of random variables may have a covariance of 0 but are not
independent. Only under some additional assumptions (e.g., the data follow
multivariate normal distributions) does a covariance of 0 imply independence
A B
Correlation coefficient:

10.
Co-Variance: An Example
 It can be simplified in computation as
 Suppose two stocks A and B have the following values in one week:
(2, 5), (3, 8), (5, 10), (4, 11), (6, 14).
 Question: If the stocks are affected by the same industry trends, will
their prices rise or fall together?
 E(A) = (2 + 3 + 5 + 4 + 6)/ 5 = 20/5 = 4
 E(B) = (5 + 8 + 10 + 11 + 14) /5 = 48/5 = 9.6
 Cov(A,B) = (2×5+3×8+5×10+4×11+6×14)/5 − 4 × 9.6 = 4
 Thus, A and B rise together since Cov(A, B) > 0.

11.
The Normal Distribution
X
f(X)


Changing μ shifts the
distribution left or right.
Changing σ increases or
decreases the spread.
- Population
- Samples
- Mean
- Standard Deviation

12.
The Normal Distribution:
as mathematical function (pdf)
2
)
(
2
1
2
1
)
( 







x
e
x
f
Note constants:
=3.14159
e=2.71828
This is a bell shaped
curve with different
centers and spreads
depending on  and 

13.
**The beauty of the normal curve:
No matter what  and  are, the area between -1 and
+1 is about 68%; the area between -2 and +2 is
about 95%; and the area between -3 and +3 is
about 99.7%. Almost all values fall within 3 standard
deviations.

14.
68-95-99.7 Rule
68% of
the data
95% of the data
99.7% of the data

15.
Example
 Suppose SAT scores roughly follows a normal
distribution in the U.S. population of college-
bound students (with range restricted to 200-800),
and the average math SAT is 500 with a standard
deviation of 50, then:
 68% of students will have scores between 450
and 550
 95% will be between 400 and 600
 99.7% will be between 350 and 650

16.
Basic Formulas for Probabilities
• Product Rule : probability P(AB) of a conjunction of two events A and B:
•Sum Rule: probability of a disjunction of two events A and B:
•Theorem of Total Probability : if events A1, …., An are mutually exclusive with
)
(
)
|
(
)
(
)
|
(
)
,
( A
P
A
B
P
B
P
B
A
P
B
A
P 

)
(
)
(
)
(
)
( AB
P
B
P
A
P
B
A
P 



)
(
)
|
(
)
(
1
i
n
i
i A
P
A
B
P
B
P 



17.
Basic Approach
Bayes Rule:
)
(
)
(
)
|
(
)
|
(
D
P
h
P
h
D
P
D
h
P 
 P(h) = prior probability of hypothesis h
 P(D) = prior probability of training data D
 P(h|D) = probability of h given D (posterior density )
 P(D|h) = probability of D given h (likelihood of D given h)
The Goal of Bayesian Learning: the most probable hypothesis given
the training data (Maximum A Posteriori hypothesis )
map
h
)
(
)
|
(
max
)
(
)
(
)
|
(
max
)
|
(
max
h
P
h
D
P
D
P
h
P
h
D
P
D
h
P
h
H
h
H
h
H
h
map






Null hypothesis
Alternate hypothesis
In which class I have to put my
sample?
Prediction (classification )
Class = [0 1]

18.
An Example
Does patient have cancer or not?
A patient takes a lab test and the result comes back positive.
The test returns a correct positive result in only 98% of the
cases in which the disease is actually present, and a correct
negative result in only 97% of the cases in which the disease is
not present. Furthermore, .008 of the entire population have this
cancer.
)
(
)
(
)
|
(
)
|
(
)
(
)
(
)
|
(
)
|
(
97
.
)
|
(
,
03
.
)
|
(
02
.
)
|
(
,
98
.
)
|
(
992
.
)
(
,
008
.
)
(
























P
cancer
P
cancer
P
cancer
P
P
cancer
P
cancer
P
cancer
P
cancer
P
cancer
P
cancer
P
cancer
P
cancer
P
cancer
P

19.
MAP Learner
For each hypothesis h in H, calculate the posterior probability
)
(
)
(
)
|
(
)
|
(
D
P
h
P
h
D
P
D
h
P 
Output the hypothesis hmap with the highest posterior probability
)
|
(
max D
h
P
h
H
h
map


Comments:
Computational intensive
Providing a standard for judging the performance of learning
algorithms
Choosing P(h) and P(D|h) reflects our prior
knowledge about the learning task

20.
Bayes Optimal Classifier
 Question: Given new instance x, what is its most probable
classification?
 Hmap(x) is not the most probable classification!
Example: Let P(h1|D) = .4, P(h2|D) = .3, P(h3 |D) =.3
Given new data x, we have h1(x)=+, h2(x) = -, h3(x) = -
What is the most probable classification of x ?
Bayes optimal classification:
)
|
(
)
|
(
max D
h
P
h
v
P i
H
hj
i
j
V
vj



Example:
P(h1| D) =.4, P(-|h1)=0, P(+|h1)=1
P(h2|D) =.3, P(-|h2)=1, P(+|h2)=0
P(h3|D)=.3, P(-|h3)=1, P(+|h3)=0 6
.
)
|
(
)
|
(
4
.
)
|
(
)
|
(








D
h
P
h
P
D
h
P
h
P
i
H
hi
i
i
H
hi
i

21.
Naïve Bayes Learner
Assume target function f: X-> V, where each instance x described by attributes
<a1, a2, …., an>. Most probable value of f(x) is:
)
(
)
|
....
,
(
max
)
....
,
(
)
(
)
|
....
,
(
max
)
....
,
|
(
max
2
1
2
1
2
1
2
1
j
j
n
V
vj
n
j
j
n
V
vj
n
j
V
vj
v
P
v
a
a
a
P
a
a
a
P
v
P
v
a
a
a
P
a
a
a
v
P
v






Naïve Bayes assumption:
)
|
(
)
|
....
,
( 2
1 j
i
i
j
n v
a
P
v
a
a
a
P 

(attributes are conditionally independent)
a1 (#
persons)
a2
(temp)
…
.
an label

22.
Bayesian classification
 The classification problem may be formalized
using a-posteriori probabilities:
 P(C|X) = prob. that the sample tuple
X=<x1,…,xk> is of class C.
 E.g. P(class=N | outlook=sunny,windy=true,…)
 Idea: assign to sample X the class label C such
that P(C|X) is maximal

23.
Estimating a-posteriori probabilities
 Bayes theorem:
P(C|X) = P(X|C)·P(C) / P(X)
 P(X) is constant for all classes
 P(C) = relative freq of class C samples
 C such that P(C|X) is maximum =
C such that P(X|C)·P(C) is maximum
 Problem: computing P(X|C) is unfeasible!

24.
Naïve Bayesian Classification
 Naïve assumption: attribute independence
P(x1,…,xk|C) = P(x1|C)·…·P(xk|C)
 If i-th attribute is categorical:
P(xi|C) is estimated as the relative freq of
samples having value xi as i-th attribute in class
C
 If i-th attribute is continuous:
P(xi|C) is estimated thru a Gaussian density
function
 Computationally easy in both cases

25.
Naive Bayesian Classifier (II)
 Given a training set, we can compute the
probabilities
O utlook P N H um idity P N
sunny 2/9 3/5 high 3/9 4/5
overcast 4/9 0 norm al 6/9 1/5
rain 3/9 2/5
Tem preature W indy
hot 2/9 2/5 true 3/9 3/5
m ild 4/9 2/5 false 6/9 2/5
cool 3/9 1/5

26.
Play-tennis example: estimating P(xi|C)
Outlook Temperature Humidity Windy Class
sunny hot high false N
sunny hot high true N
overcast hot high false P
rain mild high false P
rain cool normal false P
rain cool normal true N
overcast cool normal true P
sunny mild high false N
sunny cool normal false P
rain mild normal false P
sunny mild normal true P
overcast mild high true P
overcast hot normal false P
rain mild high true N
outlook
P(sunny|p) = 2/9 P(sunny|n) = 3/5
P(overcast|p) = 4/9 P(overcast|n) = 0
P(rain|p) = 3/9 P(rain|n) = 2/5
temperature
P(hot|p) = 2/9 P(hot|n) = 2/5
P(mild|p) = 4/9 P(mild|n) = 2/5
P(cool|p) = 3/9 P(cool|n) = 1/5
humidity
P(high|p) = 3/9 P(high|n) = 4/5
P(normal|p) = 6/9 P(normal|n) = 2/5
windy
P(true|p) = 3/9 P(true|n) = 3/5
P(false|p) = 6/9 P(false|n) = 2/5
P(p) = 9/14
P(n) = 5/14

27.
Example : Naïve Bayes
Predict playing tennis in the day with the condition <sunny, cool, high, strong> (P(v|
o=sunny, t= cool, h=high w=strong)) using the following training data:
Day Outlook Temperature Humidity Wind Play Tennis
1 Sunny Hot High Weak No
2 Sunny Hot High Strong No
3 Overcast Hot High Weak Yes
4 Rain Mild High Weak Yes
5 Rain Cool Normal Weak Yes
6 Rain Cool Normal Strong No
7 Overcast Cool Normal Strong Yes
8 Sunny Mild High Weak No
9 Sunny Cool Normal Weak Yes
10 Rain Mild Normal Weak Yes
11 Sunny Mild Normal Strong Yes
12 Overcast Mild High Strong Yes
13 Overcast Hot Normal Weak Yes
14 Rain Mild High Strong No
we have :
021
.
)
|
(
)
|
(
)
|
(
)
|
(
)
(
005
.
)
|
(
)
|
(
)
|
(
)
|
(
)
(


n
strong
p
n
high
p
n
cool
p
n
sun
p
n
p
y
strong
p
y
high
p
y
cool
p
y
sun
p
y
p
tennise
playing
of
days
wind
strong
with
tennise
playing
of
days
#
#

28.
The independence hypothesis…
 … makes computation possible
 … yields optimal classifiers when satisfied
 … but is seldom satisfied in practice, as
attributes (variables) are often correlated.
 Attempts to overcome this limitation:
 Bayesian networks, that combine Bayesian reasoning
with causal relationships between attributes
 Decision trees, that reason on one attribute at the
time, considering most important attributes first

29.
Naïve Bayes Algorithm
Naïve_Bayes_Learn (examples)
for each target value vj
estimate P(vj)
for each attribute value ai of each attribute a
estimate P(ai | vj )
Classify_New_Instance (x)
)
|
(
)
(
max j
x
a
i
V
vj
j v
a
P
v
P
v
i




Typical estimation of P(ai | vj)
m
n
mp
n
v
a
P c
j
i



)
|
( Where
n: examples with v=v; p is prior estimate for P(ai|vj)
nc: examples with a=ai, m is the weight to prior

30.
K-Nearest-Neighbors Algorithm
 K nearest neighbors (KNN) is a simple algorithm that stores all
available cases and classifies new cases based on a similarity
measure (distance function)
 KNN has been used in statistical estimation and pattern recognition
since 1970’s.

31.
K-Nearest-Neighbors Algorithm
 A case is classified by a majority voting of its neighbors, with the
case being assigned to the class most common among its K nearest
neighbors measured by a distance function.
 If K=1, then the case is simply assigned to the class of its nearest
neighbor

32.
Distance Function Measurements

33.
Hamming Distance
 For category variables, Hamming distance can be used.

34.
K-Nearest-Neighbors

35.
What is the most possible label for c?
c

36.
What is the most possible label for c?
 Solution: Looking for the nearest K neighbors of c.
 Take the majority label as c’s label
 Let’s suppose k = 3:

37.
What is the most possible label for c?
c

38.
What is the most possible label for c?
 The 3 nearest points to c are: a, a and o.
 Therefore, the most possible label for c is a.

39.
Nearest Neighbour Rule
Non-parametric pattern
classification.
Consider a two class problem
where each sample consists of
two measurements (x,y).
k = 1
k = 3
For a given query point q,
assign the class of the
nearest neighbour.
Compute the k nearest
neighbours and assign the
class by majority vote.

40.
Nearest Neighbour Issues
 Expensive
 To determine the nearest neighbour of a query point q, must compute the distance to all N training
examples
+ Pre-sort training examples into fast data structures (kd-trees)
+ Compute only an approximate distance (LSH)
+ Remove redundant data (condensing)
 Storage Requirements
 Must store all training data P
+ Remove redundant data (condensing)
- Pre-sorting often increases the storage requirements
 High Dimensional Data
 “Curse of Dimensionality”
 Required amount of training data increases exponentially with dimension
 Computational cost also increases dramatically
 Partitioning techniques degrade to linear search in high dimension

41.
Decision theory
 Decision theory is the study of making decisions that have
a significant impact
 Decision-making is distinguished into:
 Decision-making under certainty
 Decision-making under non-certainty
 Decision-making under risk
 Decision-making under uncertainty

42.
Probability theory
 Most decisions have to be taken in the presence of uncertainty
 Probability theory quantifies uncertainty regarding the
occurrence of events or states of the world
 Basic elements of probability theory:
 Random variables describe aspects of the world whose state
is initially unknown
 Each random variable has a domain of values that it can take
on (discrete, boolean, continuous)
 An atomic event is a complete specification of the state of the
world, i.e. an assignment of values to variables of which the
world is composed

43.
Probability Theory..
 Probability space
 The sample space S={e1 ,e2 ,…,en } which is
a set of atomic events
 Probability measure P which assigns a real
number between 0 and 1 to the members of
the sample space
 Axioms
 All probabilities are between 0 and 1
 The sum of probabilities for the atomic events
of a probability space must sum up to 1
 The certain event S (the sample space itself)

44.
Prior
 Priori Probabilities or Prior reflects our prior knowledge of
how likely an event occurs.
 In the absence of any other information, a random
variable is assigned a degree of belief called unconditional
or prior probability

45.
Class Conditional probability
 When we have information concerning
previously unknown random variables
then we use posterior or conditional
probabilities: P(a|b) the probability of a
given event a that we know b
 Alternatively this can be written (the
product rule):
P(a b)=P(a|b)P(b)

)
(
)
(
)
|
(
b
P
b
a
P
b
a
P



46.
Bayes’ rule
 The product rule can be written as:
 P(a b)=P(a|b)P(b)
 P(a b)=P(b|a)P(a)
 By equating the right-hand sides:
 This is known as Bayes’ rule


)
(
)
(
)
|
(
)
|
(
a
P
b
P
b
a
P
a
b
P 

47.
Bayesian Decision Theory
 Bayesian Decision Theory is a fundamental
statistical approach that quantifies the
tradeoffs between various decisions using
probabilities and costs that accompany such
decisions.
 Example: Patient has trouble breathing
– Decision: Asthma versus Lung cancer
– Decide lung cancer when person has
asthma
 Cost: moderately high (e.g., order
unnecessary tests, scare patient)
– Decide asthma when person has lung

48.
Decision Rules
 Progression of decision rules:
 – (1) Decide based on prior probabilities
 – (2) Decide based on posterior probabilities
 – (3) Decide based on risk

49.
Fish Sorting Example Revisited

50.
Decision based on prior probabilities

51.
Question
 Consider a two-class problem, { c1 and c2 } where the prior
probabilities of the two classes are given by
 P ( c1 ) = ⋅7 and P ( c2 ) = ⋅3
 Design a classification rule for a pattern based only on prior
probabilities
 Calculation of Error Probability – P ( error )

52.
Solution

53.
Decision based on class conditional probabilities

54.
Posterior Probabilities

55.
Bayes Formula
 Suppose the priors P(wj) and conditional densities p(x|wj) are
known,
( | ) ( )
( | )
( )
j j
j
p x P
P x
p x
 
 
posterior
likelihood
prior
evidence

56.
Making a Decision

57.
Probability of Error
Average probability of
error
P(error
)
Bayes decision rule minimizes this error because

58.
 The dotted line at x0 is a threshold partitioning the feature
 space into two regions,R1 and R2. According to the Bayes decision
rule,for all values
 of x in R1 the classifier decides 1 and for all values in R2 it decides
2. However,
 it is obvious from the figure that decision errors are unavoidable.
Example of the two regions R1 and R2 formed by the Bayesian
classifier for the case of two equiprobable classes.
The dotted line at x0 is a threshold partitioning the feature space into two
regions,R1 and R2. According to the Bayes decision rule, for all values of x in
R1 the classifier decides 1 and for all values in R2 it decides 2. However, it is
obvious from the figure that decision errors are unavoidable.

59.
total probability,Pe,of committing a decision error
 which is equal to the total shaded area under the curves in Figure

60.
Minimizing the Classification Error Probability
 Show that the Bayesian classifier is optimal with respect to
minimizing the classification error probability.

61.
Generalized Bayesian Decision Theory

62.
Bayesian Decision Theory…

63.
Bayesian Decision Theory…