Exploring the Noise Resilience of Classification Algorithms

Exploring the Noise Resilience
Combined Sturges Algorithm
Akrita Agarwal
Advisor: Dr. Anca Ralescu
November 7, 2015
Akrita Agarwal Exploring the Noise Resilience Combined Sturges AlgorithmNovember 7, 2015 1 / 39

Motivation

Motivation
A study on Noise?
Real-world datasets are noisy
Recordings under normal environmental conditions
Equipment Measurement Error
Most algorithms ignore Noise.
Not a lot of research done on Noise.
Aim : Explore the robustness of algorithms to Noise.
Which algorithm is least aﬀected by noisy Datasets?

Classiﬁcation

Classification
Classification : Assigning a new observation to a set of known
categories
Companies store large amounts of data.
Effective Classifier can assist in making good predictions and informed
business decisions.
E.g. Whether to recommend Prime products to the non-prime
customers, based on behavior

Classification Algorithms
Two broad kinds of Classifiers are -
Frequency based classifiers: use the frequency of datapoints in the
dataset to determine the class membership of a given test point,
Geometry based classifiers leverage the geometrical aspects of a
dataset such as the distance.

Naive Bayes
The Naive Bayes Classiﬁer
Frequency based classiﬁer
Computes the probability of a test data point to be in each class
class probability extracted from training data.
Pros
Intuitive to understand and build.
Easily trained, even with a small dataset
It’s fast
Cons
Assumes conditional independence of the data
ignores the underlying geometry of the data.

k Nearest Neighbors
The k Nearest Neighbors Classiﬁer
Geometry based classiﬁer
Assigns the class to test data point by determining the majority class
of k nearest points
Pros
Easy to implement and understand
Classes don’t have to be linearly separable
Cons
Tends to ignore the importance of an attribute; uses all
only indirectly takes into account the frequency of the data

Combined Sturges Classiﬁer

Combined Sturges
The Combined Sturges(CS)Classiﬁer
Explicitly uses geometry + frequency
Data represented as Frequency distribution on class.
Classiﬁcation Score is computed for each class.
Test point assigned to class with highest Score.
Continuous data values are binned.
No. of bins = 1 + log2n
Sturges, 1926 - Choice of a Class Interval

Combined Sturges
Dummy Dataset
Table: Dummy Dataset
A1 A2 Class
3 2 1
1 2 1
4 2 0
3 2 1
1 1 0
2 2 1
3 3 0
4 1 0

Combined Sturges
Dummy Dataset
Table: Dummy Dataset
A1 A2 Class
3 2 1
1 2 1
4 2 0
3 2 1
1 1 0
2 2 1
3 3 0
4 1 0
Table: Frequency Distribution on Classes 0 & 1
A1 f (A1) A2 f (A2)
1 0.25 1 0.50
3 0.25 2 0.25
4 0.50 3 0.25
A1 f (A1) A2 f (A2)
1 0.25 2 0.75
2 0.25 3 0.25
3 0.50

Combined Sturges
Test Point : T1
3 4

Combined Sturges
1 Geometric Criterion
Test Point : T1
3 4
minimum Distance
Classiﬁcation
Criteria :
Geometric
Classiﬁcation
Score: Highest
Posterior
Probability
Table: Nearest distance of T1 to Classes
A1 f (A1) A2 f (A2)
1 0.25 1 0.50
3 0.25 2 0.25
4 0.50 3 0.25
A1 f (A1) A2 f (A2)
1 0.25 2 0.75
2 0.25 3 0.25
3 0.50

Combined Sturges
Classiﬁcation Score, S(c) c ∈ 0, 1
S(0)
A1 = P(Class0) × f (A1)
A2 = P(Class0) × f (A2)
average(A1, A2) =average(0.5 ×0.25, 0.5 × 0.25) = 0.125
S(1)
A1 = P(Class1) × f (A1)
A2 = P(Class1) × f (A2)
average(A1, A2) = average(0.5 × 0.50, 0.5 × 0.25) = 0.187
S(0) < S(1)
Class 1

Combined Sturges
1 Statistical Criterion
Test Point : T1
3 4
maximum
Frequency
Classiﬁcation
criteria : Statistical
Classiﬁcation
Score: Minimum
Distance
Table: Maximum Frequency in Classes
A1 f (A1) A2 f (A2)
1 0.25 1 0.50
3 0.25 2 0.25
4 0.50 3 0.25
A1 f (A1) A2 f (A2)
1 0.25 2 0.75
2 0.25 3 0.25
3 0.50

Combined Sturges
Classiﬁcation Score
S(0)
A1 = (4 − 3) = 1
A2 = (4 − 1) = 3
average(A1, A2) = average(1, 3) = 2
S(1)
A1 = (3 − 3) = 0
A2 = (4 − 2) = 2
average(A1, A2) = average(0, 2) = 1
S(0) > S(1)
Class 1

Combined Sturges
1 Combined Criterion
Test Point : T1
3 4
d =
(T1 − A1).f (A1)
Expected Distance
ED = EDc
A1.EDc
A2
min Expected
Distance, ED
Table: Aggregate Expected Distance, ED
A1 f (A1) d.f A2 f (A2) d.f
1 0.25 0.50 1 0.50 1.50
3 0.25 0 2 0.25 0.50
4 0.50 0.50 3 0.25 0.25
ED0
A1 1.00 ED0
A2 2.25
A1 f (A1) d.f A2 f (A2) d.f
1 0.25 0.50 2 0.75 1.50
2 0.25 0.25 3 0.25 0.25
3 0.50 0
ED1
A1 0.75 ED1
A2 1.75

Combined Sturges
Classiﬁcation Penalty
S(0)
ED = 1.00 × 2.25 = 2.25
S(0) = ED × (1 − P(Class0)) = 1.125
S(1)
ED = 0.75 × 1.75 = 1.31
S(1) = ED × (1 − P(Class1)) = 0.655
S(0) > S(1)
Class 1

The Noise Model

The Noise Model
Dealing with Noise
Brodley & Fried, 1999 - detect and reduce noise
Kubica & Moore, 2003 - identify Noise using a probabilistic model
and remove it.
Elias Kalapanidas, 2003 - Developed a Noise Model based on data
properties.

The Noise Model
Additive Noise, x = x + δx
δx = σxj × zi,j
σxj , standard deviation of attribute j,
zi,j = CDF(pi,j )
xi,j =
xi,j if pi,j ≥ n
xi,j if pi,j < n
(1)
Based on Noise level n ∈ {0, 0.15, 0.30, 0.50, 0.80}

The Noise Model
Attribute-level Noise
Table: Original Dataset
A1 A2 Class
3 2 1
1 2 1
4 2 0
3 2 1
1 1 0
2 2 1
3 3 0
4 1 0
Table: 40% (n = 0.4) Noisy Dataset
A1 A2 Class
8.5 0.55 1
8.9 2 1
4 0.7 0
3 2 1
4.7 1 0
2 2 1
3 3 0
1.6 0.02 0

Datasets

Datasets
Artiﬁcial datasets
Multivariate Normal
x1 = random Normal vector, t = random Normal vector
x2 = 0.8x1 + 0.6t
x3 = 0.6x1 + 0.8t
x4 = t
Linear Function with Non-normal inputs
x2 = (x1)2
+ 0.5t

Datasets
2 Artiﬁcial datasets
Diﬀerent Imbalanced-Ratio
3 Real Datasets
Table: Comparison of physical properties of Datasets.
Dataset
No. of
Samples
No. of
Classes
No. of
Attributes
Attribute
Value
Imbalance
Ratio
Haberman 306 2 3 Integer 2.78
A1 200 3 4 Real 6.66
A2 200 3 4 Real 39
Iris 150 3 4 Real 2
Pima
Diabetes
768 2 8
Integer,
Real
1.87

Process Flow
1 Create Artiﬁcial Datasets
2 Implement the Noise model on all Datasets
3 Apply the three algorithms
4 Compare the results

Results

Results
Performance Measures
Confusion Matrix
Table: Confusion matrix for 2 classes.
Predicted Outcome
Positive Negative
Actual values
Positive TP FN
Negative FP TN
Accuracy Acc = TP+TN
TP+TN+FP+FN
Precision P = TP
TP+FP
Recall R = TP
TP+FN
F-measure Fα = PR
αP+(1−α)R

Results
Non-Noisy Datasets
Artiﬁcial Datasets -
knn does the best - 91.2% & 93.7%
Good improvement in CS from 65% - 76%
Table: Non-Noisy Artiﬁcial Datasets - Performance of all algorithms
Dataset Algorithm Accuracy Precision Recall F-measure
CS 65.0 63.5 70.1 66.6
knn 91.2 92.8 87.4 89.8A1
Naive Bayes 60.2 61.6 60.14 64.1
CS 76.0 68.4 71.62 69.7
knn 93.7 94.7 91.9 93.2A2
Naive Bayes 63.1 61.1 65.2 63.5

Results
Real Datasets -
Iris : knn does best. Followed by Naive Bayes.
Haberman : CS does best. Naive Bayes is really bad.
Pima-Diabetes : CS is best. Naive Bayes follows.
Table: Non-Noisy Real Datasets - Performance of all algorithms
Dataset Algorithm Accuracy Precision Recall F-Measure
CS 94.3 95.1 94.3 94.7
knn 96.7 96.8 96.7 96.8Iris
Naive Bayes 96.2 93.7 95 94.3
CS 75.2 67.2 61.6 64.2
knn 73.4 63.2 54.8 58.5Haberman
Naive Bayes 0.5 41.9 47.6 47.3
CS 73.7 74.9 65.1 69.6
knn 64.5 65.6 66.9 66.3
Pima -
Diabetes
Naive Bayes 70.3 59.2 56.7 57.9

Results
Noisy Datasets : A1
knn does best.
For both knn and CS, No change with noise
Naive Bayes does bad.
Table: Noisy A1 dataset - Performance of all algorithms
Algorithm Noise % Accuracy Precision Recall F-Measure
0 65 63.5 70.1 66.6
15 64.8 63.4 96.7 96.8CS
50 65.5 63.2 95 94.3
0 87.5 87.2 61.6 61.6
15 87.3 88.1 54.8 58.5knn
50 86.7 88.5 47.6 47.3
0 ≈ 0 ≈ 0 ≈ 0 ≈ 0
15 ≈ 0 ≈ 0 ≈ 0 ≈ 0Naive Bayes
50 ≈ 0 ≈ 0 ≈ 0 ≈ 0

Results
Noisy Datasets : A2
knn does best, but goes from 92.6% - 86.3%
For CS, no change with noise
From A1 to A2, CS : 65% - 76%
Table: Noisy A2 dataset - Performance of all algorithms
0 76.0 68.4 71.6 69.7
15 76.8 64.7 73.1 68.4CS
50 76.4 66.9 71.7 68.5
0 92.6 86.9 85.5 86.2
15 91.1 84.2 84.2 83.5knn
50 86.3 83.0 78.2 77.9
0 ≈ 0 ≈ 0 ≈ 0 ≈ 0
15 ≈ 0 ≈ 0 ≈ 0 ≈ 0Naive Bayes
50 ≈ 0 ≈ 0 ≈ 0 ≈ 0

Results
Noisy Datasets : Iris
knn does best at 0% Noise (96.7%) , then CS 94.5%
CS does best at 50% Noise - 73.1%, then knn - 63.8%
Table: Noisy Iris dataset - Performance of all algorithms
0 94.5 94.9 94.5 94.7
15 86.2 87.6 86.2 86.9CS
50 73.1 74.9 73.1 73.9
0 96.7 96.8 96.7 96.8
15 83.6 84.6 83.6 84.1knn
50 63.8 63.2 63.8 63.5
0 93.3 92.3 91.9 92.1
15 92.3 91.5 91.2 91.4Naive Bayes
50 0.7 18.3 0.7 NaN

Results
Noisy Datasets : Haberman
CS does best at 74.7%
Naive Bayes performs badly at ≈ 43%
Table: Noisy Haberman dataset - Performance of all algorithms
0 74.7 66.7 61.4 63.9
15 66.1 62.2 61.9 62.0CS
50 74.5 66.6 63 64.7
0 74.1 65.7 55.1 59.7
15 72.0 56.2 52.3 54.0knn
50 70.5 51.8 50.6 51.0
0 41.0 47.1 46.5 46.8
15 43.3 46.2 45.3 45.7Naive Bayes
50 41.4 34.7 32.4 31.8

Results
Noisy Datasets : Pima-Diabetes
CS does best, followed by knn
Naive Bayes bad with Noise : 70% - 55.7% - 0%
Table: Noisy Pima-Diabetes dataset - Performance of all algorithms
0 72.8 72.8 64.2 68.2
15 70.8 68.3 65.8 67CS
50 67.0 64.9 55.9 60.0
0 63.5 64.6 65.9 65.2
15 60.8 61.2 62.3 61.7knn
50 55.0 55.6 56.1 55.8
0 70.3 59.2 56.7 57.9
15 55.7 49.4 46.0 NaNNaive Bayes
50 0 0 0 NaN

Results
Results Summary
Table: Best Algorithm for diﬀerent Noise Levels
Dataset 0% Noise 15% Noise 50% Noise
A1 knn knn knn
A2 knn knn knn
Haberman CS knn CS
Iris knn Naive Bayes CS
Pima -
Diabetes
CS CS CS

Conclusion
No algorithm is best.
In general knn has better accuracy but CS is more robust to noise.
Naive Bayes does much worse for noise, than others.
Also:
CS performs well for Imbalanced Datasets.

Future Work
Test with more datasets.
Test for performance on imbalanced datasets.
Only additive Noise model was used, try with other variations.
Compare with more algorithms.

Questions
Questions?

Exploring the Noise Resilience of Classification Algorithms

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (6)

Similar to Exploring the Noise Resilience of Classification Algorithms

Similar to Exploring the Noise Resilience of Classification Algorithms (20)

Recently uploaded

Recently uploaded (20)

Exploring the Noise Resilience of Classification Algorithms