PCA: Compress Features into Directions
Several measurements may tell us nearly the same thing.
PCA: Compress Features into Directions explained visually
Several measurements may tell us nearly the same thing. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Several measurements may tell us nearly the same thing.
Find directions of variation
Several measurements may tell us nearly the same thing. Principal component analysis, or principal component analysis, finds directions that summarize variation in numerical data. It can provide a lower-dimensional representation, reduce redundancy, or support visualization. It does not use target labels when choosing those directions.
Reduce dimensions carefully
The first principal component captures the largest variance under the principal component analysis formulation, and subsequent components capture remaining variance while being orthogonal to earlier ones. Centering is essential, and scaling may be appropriate when features use different units. Fit these transformations on training data. A direction with high variance is not necessarily the direction most useful for prediction.
Keep preprocessing separate
Reducing the number of components discards information. Choose the dimension using the intended goal, not only a visually convenient two-dimensional plot. Explained variance describes retained variation, not retained predictive accuracy. If principal component analysis feeds a supervised model, evaluate the complete pipeline. Also remember that principal components are combinations of original features, which can make individual predictions harder to explain in the original units.
A worked example
Suppose two measurements both describe nearly the same size characteristic. Their values may lie close to a diagonal line in a scatter plot. One principal component along that line can summarize much of their variation, while the perpendicular direction contains comparatively little variation.
Worked example
Suppose two measurements both describe nearly the same size characteristic. Their values may lie close to a diagonal line in a scatter plot. One principal component along that line can summarize much of their variation, while the perpendicular direction contains comparatively little variation.
Quick check: test your understanding
Try it yourself: Could a low-variance feature still be important for predicting a rare event?
Check the reasoning
Yes. Predictive relevance and variance are different concepts. A subtle feature may distinguish the rare event even if it contributes little overall variation.
Does retaining 95 percent of variance guarantee retaining 95 percent of prediction accuracy?
No. Those percentages measure different things.
How would you solve the practice case?
Yes. Predictive relevance and variance are different concepts. A subtle feature may distinguish the rare event even if it contributes little overall variation.
What does the worked example show?
Suppose two measurements both describe nearly the same size characteristic. Their values may lie close to a diagonal line in a scatter plot. One principal component along that line can summarize much of their variation, while the perpendicular direction contains comparatively little variation.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 14 transcript211 words · Video transcript · Matches the published narration and captionsOpen transcript
Several measurements may tell us nearly the same thing. Principal component analysis, or principal component analysis, finds directions that summarize variation in numerical data. It can provide a lower-dimensional representation, reduce redundancy, or support visualization. It does not use target labels when choosing those directions.
The first principal component captures the largest variance under the principal component analysis formulation, and subsequent components capture remaining variance while being orthogonal to earlier ones. Centering is essential, and scaling may be appropriate when features use different units. Fit these transformations on training data. A direction with high variance is not necessarily the direction most useful for prediction.
Reducing the number of components discards information. Choose the dimension using the intended goal, not only a visually convenient two-dimensional plot. Explained variance describes retained variation, not retained predictive accuracy. If principal component analysis feeds a supervised model, evaluate the complete pipeline. Also remember that principal components are combinations of original features, which can make individual predictions harder to explain in the original units.
Suppose two measurements both describe nearly the same size characteristic. Their values may lie close to a diagonal line in a scatter plot. One principal component along that line can summarize much of their variation, while the perpendicular direction contains comparatively little variation.