K Nearest Neighbour, Distance Metrics, Scaling, Strengths and Weakness

1. K Nearest Neighbours
Varun Jain

2. Show me your
Friends and I’ll
show you your
future

3. The Idea

6. Dothraki or a Westerosian
Five Neighbours
https://medium.com/@pratikkotian04/an-introduction-to-knn-k-nearest-neighbours-the-game-of-thrones-way-f2738cf563ab

7. 60
Age
40
20
0 10
Number of malignant nodes
20
Features. No of malignant nodes / Age
Labels. Healthy / Complications / Lost

8. 0 10
Number of malignant nodes
20
60
Age 40
New Example
known: #nodes,age
Predict:
healthy | comp. | lose
2 Features. No of malignant nodes / Age
3 Labels. Healthy / Complications / Lost

9. 60
20
0 10
Number of malignant nodes
20
K Nearest Neighbors
Features. No of malignant nodes / Age
Labels. Healthy / Lost
Age 40
New Example
known:
#nodes,age
Predict:
healthy | lose

10. K Nearest Neighbors K=1
Neighbor count: 0 1
60
20
0 10
Number of malignant nodes
20
Age 40
New Example
known:
#nodes,age
Predict:
healthy | lose

11. 20
0 10
Number of malignant nodes
20
K Nearest Neighbors K=2
Neighbor count: 1 1
60
Age 40
New Example

12. 20
0 10
Number of malignant nodes
20
K Nearest Neighbors K=3
Neighbor count: 2 1
60
Age 40
New Example

13. 20
0 10
Number of malignant nodes
20
K Nearest Neighbors K=4
Neighbor count: 3 1
60
Age 40
New Example

14. 20
0 10
Number of malignant nodes
20
Age 40
New Example
K Nearest Neighbors K=5
Neighbor count: 4 1
60

15. 20
0 10
Number of malignant nodes
20
Age 40
N New Example
K Nearest Neighbors K=6
Neighbor count: 4 2
60

16. Distance Measurement

17. Distance Measurement
K nearest neighbors is a simple algorithm that stores all available cases and classifies new cases
based on a similarity measure (e.g., distance functions).
As in Knn fits all the data in the memory at the time of training and an object is classified by
a majority vote of its neighbors, with the object being assigned to the class most common among
its k nearest neighbors.

18. 60
Age40
KNN Decision Boundary K=1
20
0 10
Number of malignant nodes
20

19. 60
Age40
KNN Decision Boundary K=5
20
0 10
Number of malignant nodes
20

20. K=1 K=5 K=25
K=1 K=20
https://bdewilde.github.io/blog/blogger/2012/10/26/classification-of-hand-written-digits-3/

21. 60
Features. No of malignant nodes / Age
Labels. Healthy / Lost
20
0 10
Number of malignant nodes
20
Age 40

22. 0 10
Number of malignant nodes
20
60
20
Multiclass KNN Decision Boundary
K=5
Age 40

23. 60
20
012345
Scaling is crucial!
Number of surgeries
Age 40

24. Scaling is crucial!
60
31
28
22
20
012345
Number of surgeries
25
Age 40

25. Scaling is crucial!
60
31
28
25
Nearest
Neighbors!
22
20
012345
Number of surgeries
Age 40

26. 60
Scaling is crucial!
20
0 1 2 3 4 5
Number of surgeries
Age 40

27. 60
Scaling is crucial!
20
0 1 2 3 4 5
Number of surgeries
Age 40

28. K Nearest Neighbors
● Lazy
● “fits” fast, predicts slow
● Assuming that our data is d-dimensional, then the
straightforward implementation is O(dn) time.
● Higher memory (saves entire training set)
● Various implementations, including weighted KNN.
● Can work very poorly in high dimensional feature space
(curse of dimensionality)
● Requires preprocessing (Scaling etc.)