Explore Data Before Fitting a Model
A model can fit a spreadsheet full of mistakes without telling you which mistakes matter.
Explore Data Before Fitting a Model explained visually
A model can fit a spreadsheet full of mistakes without telling you which mistakes matter. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
A model can fit a spreadsheet full of mistakes without telling you which mistakes matter.
Check the table
A model can fit a spreadsheet full of mistakes without telling you which mistakes matter. Exploratory data analysis is the first opportunity to understand what the rows and columns actually represent. Check units, ranges, missing values, duplicates, and the process that produced each field before choosing a learning algorithm.
Look at distributions
Summaries and plots reveal different aspects of the data. A mean describes a central value, but a histogram may reveal two very different groups. A scatter plot can show nonlinear relationships or unusual observations that a correlation coefficient hides. Compare groups relevant to the application and investigate whether missingness follows a pattern. Missing values are sometimes evidence about how the data were collected.
Investigate suspicious patterns
Unusual values should be investigated rather than automatically deleted. A temperature of five hundred may be a unit error, a broken sensor, or a real extreme event. The correct response depends on the source. Keep a record of cleaning decisions and avoid exploring the reserved test set in ways that shape the model. Use development data to learn the data-generation process, and preserve independent evidence for later evaluation.
A worked example
Suppose a distance column contains values around five, ten, and fifteen, plus a group around five,zero. The second group might use meters while the first uses kilometers. A model could treat that unit inconsistency as a meaningful pattern unless the data are checked first.
Worked example
Suppose a distance column contains values around 5, 10, and 15, plus a group around 5,000. The second group might use meters while the first uses kilometers. A model could treat that unit inconsistency as a meaningful pattern unless the data are checked first.
Quick check: test your understanding
Try it yourself: List three checks you would perform on a new numerical column before using it as a feature.
Check the reasoning
Check its definition and unit, inspect its range and distribution, and examine missing or exceptional values against the source records.
Should every outlier be removed?
No. Some outliers are genuine observations that the application must handle.
How would you solve the practice case?
Check its definition and unit, inspect its range and distribution, and examine missing or exceptional values against the source records.
What does the worked example show?
Suppose a distance column contains values around 5, 10, and 15, plus a group around 5,000. The second group might use meters while the first uses kilometers. A model could treat that unit inconsistency as a meaningful pattern unless the data are checked first.
Further reading
Machine Learning — Tom M. Mitchell.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 3 transcript224 words · Video transcript · Matches the published narration and captionsOpen transcript
A model can fit a spreadsheet full of mistakes without telling you which mistakes matter. Exploratory data analysis is the first opportunity to understand what the rows and columns actually represent. Check units, ranges, missing values, duplicates, and the process that produced each field before choosing a learning algorithm.
Summaries and plots reveal different aspects of the data. A mean describes a central value, but a histogram may reveal two very different groups. A scatter plot can show nonlinear relationships or unusual observations that a correlation coefficient hides. Compare groups relevant to the application and investigate whether missingness follows a pattern. Missing values are sometimes evidence about how the data were collected.
Unusual values should be investigated rather than automatically deleted. A temperature of five hundred may be a unit error, a broken sensor, or a real extreme event. The correct response depends on the source. Keep a record of cleaning decisions and avoid exploring the reserved test set in ways that shape the model. Use development data to learn the data-generation process, and preserve independent evidence for later evaluation.
Suppose a distance column contains values around five, ten, and fifteen, plus a group around five,zero. The second group might use meters while the first uses kilometers. A model could treat that unit inconsistency as a meaningful pattern unless the data are checked first.