Nearest Neighbors: Learn from Similar Examples
You want to estimate a home's price by looking at similar homes nearby.
Nearest Neighbors: Learn from Similar Examples explained visually
You want to estimate a home's price by looking at similar homes nearby. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
You want to estimate a home's price by looking at similar homes nearby.
Define similarity
You want to estimate a home's price by looking at similar homes nearby. Nearest-neighbor methods use that basic idea in feature space. They find training examples close to a new example, then combine their targets. For classification, neighbors can vote; for regression, their values can be averaged.
Choose the neighborhood size
The meaning of close depends on the features, scaling, and distance function. If one feature spans thousands and another spans only zero to one, the first may dominate an ordinary Euclidean distance. Scaling can help, but it must be fitted on training data. Irrelevant features can also make similarity less meaningful, particularly as the number of dimensions grows.
Recognize scaling and dimensionality
The number of neighbors controls flexibility. A very small neighborhood can follow local noise, while a larger neighborhood smooths predictions and may blur real boundaries. Choose the value with validation or cross-validation. Remember the deployment cost: prediction may require comparing against many stored training examples. A method that is simple to describe can still be expensive at query time, and storing examples may have privacy implications for the application.
A worked example
For a binary classification example, five nearby training records contain four examples of class A and one of class B. An unweighted majority vote predicts class A. That vote is only useful if the chosen distance identifies genuinely relevant neighbors.
Worked example
For a binary classification example, five nearby training records contain four examples of class A and one of class B. An unweighted majority vote predicts class A. That vote is only useful if the chosen distance identifies genuinely relevant neighbors.
Quick check: test your understanding
Try it yourself: Why can combining age in years and income in dollars cause a distance problem without scaling?
Check the reasoning
Dollar differences can be numerically much larger and dominate the distance, even when age is equally important to the task.
Does choosing one neighbor guarantee the best generalization?
No. It can make predictions highly sensitive to noise or mislabeled individual examples.
How would you solve the practice case?
Dollar differences can be numerically much larger and dominate the distance, even when age is equally important to the task.
What does the worked example show?
For a binary classification example, five nearby training records contain four examples of class A and one of class B. An unweighted majority vote predicts class A. That vote is only useful if the chosen distance identifies genuinely relevant neighbors.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 7 transcript214 words · Video transcript · Matches the published narration and captionsOpen transcript
You want to estimate a home's price by looking at similar homes nearby. Nearest-neighbor methods use that basic idea in feature space. They find training examples close to a new example, then combine their targets. For classification, neighbors can vote; for regression, their values can be averaged.
The meaning of close depends on the features, scaling, and distance function. If one feature spans thousands and another spans only zero to one, the first may dominate an ordinary Euclidean distance. Scaling can help, but it must be fitted on training data. Irrelevant features can also make similarity less meaningful, particularly as the number of dimensions grows.
The number of neighbors controls flexibility. A very small neighborhood can follow local noise, while a larger neighborhood smooths predictions and may blur real boundaries. Choose the value with validation or cross-validation. Remember the deployment cost: prediction may require comparing against many stored training examples. A method that is simple to describe can still be expensive at query time, and storing examples may have privacy implications for the application.
For a binary classification example, five nearby training records contain four examples of class A and one of class B. An unweighted majority vote predicts class A. That vote is only useful if the chosen distance identifies genuinely relevant neighbors.