The document discusses the K-nearest neighbor (KNN) algorithm, a non-parametric lazy learning classification method. KNN stores all training examples and classifies new examples based on their similarity to stored examples, by finding the k most similar examples and basing the classification on those neighbors. The document provides examples of how KNN can be used for spam filtering by classifying emails based on their word counts and distances to stored labeled examples. The classification result may depend on the choice of k for the number of neighbors.

1. K-NEAREST NEIGHBOR
ALGORITHM
Presented by Hien Nguyen

2. WHY DO WE CARE?
Amazon Prime Movie Adaptive Text Retrieval
Spam Email filtering Online course recommendation

3. WHAT IS K-NN ALGORITHM?
KNN is a non parametric lazy
learning algorithm that stores
all available cases and classifies
new cases based on a similarity
measure
3
Features Comparator
Library
Recommendati
on

4. K-NN CLASSIFICATION
4
Sir/Mada
m
Occurrenc
es
Word
Cout
Class
10 100 Spam
5 200 Good
25 100 Spam
30 400 Spam
20 300 Spam
0 400 Good
20 500 Good
30 600 Spam
10 600 Good
15 700 Spam
8 700 Spm
2 700 Good

5. K-NN CLASSIFICATION
2
21
2
21 )()( yyxxD 

6. K-NN CLASSIFICATIONSir/Madam
Occurrence
s
Word Cout Class Distanc
e
10 100 Spam 100
5 200 Good 5
25 100 Spam 101.11
30 400 Spam 200.99
20 300 Spam 100.49
0 400 Good 200.24
20 500 Good 300.16
30 600 Spam 400.49
10 600 Good 400
15 700 Spam 500.02
5
8 700 Spm 500.00
4

7. K-NN CLASSIFICATIONSir/Madam
Occurrence
s
Word Cout Class Distanc
e
10 100 Spam 100
5 200 Good 5
25 100 Spam 101.11
30 400 Spam 200.99
20 300 Spam 100.49
0 400 Good 200.24
20 500 Good 300.16
30 600 Spam 400.49
10 600 Good 400
15 700 Spam 500.02
5
8 700 Spm 500.00
4
K=1 => M: Good email

8. K-NN CLASSIFICATIONSir/Madam
Occurrence
s
Word Cout Class Distanc
e
10 100 Spam 100
5 200 Good 5
25 100 Spam 101.11
30 400 Spam 200.99
20 300 Spam 100.49
0 400 Spam 200.24
20 500 Good 300.16
30 600 Spam 400.49
10 600 Good 400
15 700 Spam 500.02
5
8 700 Spm 500.00
4
K=3 => M: Spam

9. K-NN CLASSIFICATIONSir/Madam
Occurrence
s
Word Cout Class Distanc
e
10 100 Spam 100
5 200 Good 5
25 100 Spam 101.11
30 400 Spam 200.99
20 300 Spam 100.49
0 400 Good 200.24
20 500 Good 300.16
30 600 Spam 400.49
10 600 Good 400
15 700 Spam 500.02
5
8 700 Spm 500.00
4
K=5 => M: Spam

10. KNN – VOTING
Majority voting:
 all votes are equal. Count how many of the k neighbours have that class.
Return the class with the most votes.
Inverse distance-weighted voting:
 Closer neighbours get higher votes. While there are better-motivated methods, the
simplest version is to take a neighbour’s vote to be the inverse of its distance to the
new instance:
 Then we sum the votes and return the class with the highest vote
10

11. SUMMARY
KNN is conceptually simple, yet able to solve complex problems
Can work with relatively little information
Learning is simple (no learning at all!)
Memory and CPU cost
Feature selection problem
Sensitive to representation
11WWW.ISMARTSOFT.COM

12. PRACTICE
Creature A Creature B
3-NN ? ?
5-NN ? ?

13. ACKNOWLEDGEMENT
Two examples of spam email and game are from MIT Open Course
Midterm and Final exam pages.