Evaluate Imbalanced Classification Honestly
Only one out of every hundred manufactured parts is defective.
Evaluate Imbalanced Classification Honestly explained visually
Only one out of every hundred manufactured parts is defective. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Only one out of every hundred manufactured parts is defective.
Accuracy can hide failure
Only one out of every hundred manufactured parts is defective. A model that always says good achieves ninety-nine percent accuracy while finding no defects. Class imbalance makes overall accuracy easy to misinterpret. Evaluation should describe the errors that matter to the intended decision.
Separate error types
A confusion matrix separates true positives, false positives, true negatives, and false negatives. Precision describes the correctness of positive predictions, while recall describes how many actual positives were found. Their tradeoff changes with the decision threshold. Precision-recall curves are often informative when positives are rare, but they still need to be interpreted in the context of the observed class prevalence and application.
Choose a usable operating point
Training choices such as class weighting or resampling can change what the model learns. Apply resampling only within the training portion of each split, never before creating the evaluation set. Assess calibration and the workload caused by false alarms. A technically improved recall may overwhelm reviewers if it produces too many alerts. Choose an operating point with validation data and evaluate the complete review process on held-out examples.
A worked example
Among one,zero parts, ten are defective. An always-good predictor correctly labels nine hundred and ninety parts but has zero recall for defects. If another model finds eight defects and raises twelve false alarms, its precision is eight divided by twenty, or forty percent, and its recall is eighty percent.
Worked example
Among 1,000 parts, ten are defective. An always-good predictor correctly labels 990 parts but has zero recall for defects. If another model finds eight defects and raises twelve false alarms, its precision is eight divided by twenty, or 40 percent, and its recall is 80 percent.
Quick check: test your understanding
Try it yourself: Why might a factory accept lower precision when missed defects are expensive, and what practical limit must it still consider?
Check the reasoning
Higher recall may prevent costly misses, but the number of false alarms must remain manageable for inspection staff and production.
Should synthetic oversampling be applied before train-test splitting?
No. It can leak related or generated information into the evaluation set.
How would you solve the practice case?
Higher recall may prevent costly misses, but the number of false alarms must remain manageable for inspection staff and production.
What does the worked example show?
Among 1,000 parts, ten are defective. An always-good predictor correctly labels 990 parts but has zero recall for defects. If another model finds eight defects and raises twelve false alarms, its precision is eight divided by twenty, or 40 percent, and its recall is 80 percent.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 12 transcript222 words · Video transcript · Matches the published narration and captionsOpen transcript
Only one out of every hundred manufactured parts is defective. A model that always says good achieves ninety-nine percent accuracy while finding no defects. Class imbalance makes overall accuracy easy to misinterpret. Evaluation should describe the errors that matter to the intended decision.
A confusion matrix separates true positives, false positives, true negatives, and false negatives. Precision describes the correctness of positive predictions, while recall describes how many actual positives were found. Their tradeoff changes with the decision threshold. Precision-recall curves are often informative when positives are rare, but they still need to be interpreted in the context of the observed class prevalence and application.
Training choices such as class weighting or resampling can change what the model learns. Apply resampling only within the training portion of each split, never before creating the evaluation set. Assess calibration and the workload caused by false alarms. A technically improved recall may overwhelm reviewers if it produces too many alerts. Choose an operating point with validation data and evaluate the complete review process on held-out examples.
Among one,zero parts, ten are defective. An always-good predictor correctly labels nine hundred and ninety parts but has zero recall for defects. If another model finds eight defects and raises twelve false alarms, its precision is eight divided by twenty, or forty percent, and its recall is eighty percent.