This document discusses k-nearest neighbor (k-NN) machine learning algorithms. It explains that k-NN is an instance-based, lazy learning method that stores all training data and classifies new examples based on their similarity to stored examples. The key steps are: (1) calculate the distance between a new example and all stored examples, (2) find the k nearest neighbors, (3) assign the new example the most common class of its k nearest neighbors. Important considerations include the distance metric, value of k, and voting scheme for classification.

1. CS 9633 Machine Learning k-nearest neighbor Chapter 8: Instance-based learning Adapted from notes by Tom Mitchell http://www-2.cs.cmu.edu/~tom/mlbook-chapter-slides.html

2. A classification of learning algorithms Eager learning algorithms Neural networks Decision trees Bayesian classifiers Lazy learning algorithms K-nearest neighbor Case based reasoning

3. General Idea of Instance-based Learning Learning: store all the data instances Performance: when a new query instance is encountered retrieve a similar set of related instances from memory use to classify the new query

4. Pros and Cons of Instance Based Learning Pros Can construct a different approximation to the target function for each distinct query instance to be classified Can use more complex, symbolic representations Cons Cost of classification can be high Uses all attributes (do not learn which are most important)

5. k-nearest neighbor (knn) learning Most basic type of instance learning Assumes all instances are points in n-dimensional space A distance measure is needed to determine the “closeness” of instances Classify an instance by finding its nearest neighbors and picking the most popular class among the neighbors

6. Important Decisions Distance measure Value of k (usually odd) Voting mechanism Memory indexing

7. Euclidean Distance Typically used for real valued attributes Instance x (often called a feature vector) Distance between two instances x i and x j

8. Discrete Valued Target Function Training algorithm: For each training example <x, f(x), add the example to the list training_examples Classification algorithm: Given a query instance x q to be classified Let x 1 …x k be the k training examples nearest to x q Return

9. Continuous valued target function Algorithm computes the mean value of the k nearest training examples rather than the most common value Replace fine line in previous algorithm with

10. Training dataset Poor Married Medium Very High Jack Good Married Very High Very Low Igor Doubtful Married Medium Low Harry Doubtful Unmarried High Low George Poor Married Very low High Fred Poor Married Low High Ellen Poor Married Low Very high Dale Poor Unmarried Very low Medium Candy Doubtful Married High Low Ben Good Married High High Abel Risk Marital Status Income Debt Customer ID

11. k-nn K = 3 Distance Score for an attribute is 1 for a match and 0 otherwise Distance is sum of scores for each attribute Voting scheme: proportionate voting in case of ties

12. Test Set ? Unmarried High Low Steve ? Married Very low Very low Trey ? Divorced Low High Unace ? Married Low High Vasco ? Unmarried Very low Very low Xu ? Married High Low Yong ? Married Medium High Zeb Risk Marital Status Income Debt Customer ID

13. Algorithm questions What is the space complexity for the model? What is the time complexity for learning the model? What is the time complexity for classification of an instance?