Plan the survey and clean the data (free preview) | StemPy Lab
Free previewCourse: Standard
Plan the survey and clean the data
Real data has mistakes: a typing error, a value that cannot be true, a blank. Decide what values are sensible, find the ones outside, check them with the source, fix them in a copy and keep the original. One bad value can move the mean a long way, while the median hardly moves.
This is the first lesson of Averages: Class Survey Data Project, free for everyone to read.
The survey that gave a strange answer
A class asked twelve students: how many hours did you sleep last night? The answers went into a list. Then somebody found the mean.
How far does one mistake pull?
You should find that the mean moves a little for every hour you add to the wrong value. The median stays at 8 hours unless the wrong value drops below 8. A value beyond the dashed line at 16 hours cannot be true.
Plan before you ask
A good survey starts with one clear question, such as "How many hours did you sleep last night?" Then decide three things.
Who you ask. A fair sample is chosen at random, as in Course 1.1.
What you write down. A number in hours, not a word such as "a lot".
How you record it. One row for each student, one column for each fact.
Cleaning the data
Cleaning means finding the values that cannot be true and dealing with them. Follow these steps in this order.
Decide what is sensible. For hours of sleep in one night, a value below 0 or above 16 cannot be true.
Find the values outside that range. A short program can look at every value for you.
Check with the source. Ask the student. Never guess.
Fix a copy. If the student says 8, put 8 in a copy of the list. If you cannot check, leave the value out and say so in your report.
Keep the original. Somebody may want to see what the survey really collected.
Practise with new numbers
Common mistakes
Deleting a strange value without checking. It may be true. Ask first.
Guessing the right value. Only the person who answered knows it.
Cleaning the original list. Keep it, and fix a copy.
Hiding a change. If you left a value out, say so in the report.
Remember these
Decide what is sensible before you look at the data.
Check a strange value with the source. Do not guess.
Fix a copy and keep the original.
One bad value moves the mean much more than the median.
Check yourself
Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.
A survey of shoe sizes has the value 75. Is it sensible? What do you do?
Why do we keep the original data after cleaning?
Ten students gave a mean of 13 hours of sleep, but one value was typed as 60 instead of 6. What is the correct mean?
Which value of 4,5,6,5 and 500 minutes of travel would you check first?
Liked this lesson? There is more
Averages: Class Survey Data Project is a Standard course. It adds 2 more lessons, practice and tests on top of the free chapter.
8Problems3Python Labs1Question Set1Mock Test
Compare groups and write the report: To compare two groups, find the mean and the median of each, and say how many were asked, because small groups can mislead. To combine two group means, add the totals and divide by all the students, never average the two means. A good report has the question, the method, the numbers, a graph and one honest limit.
Coding the survey: one table, many groups: A real survey is one table with a row for each student and a column for each answer. A short program can split the rows into groups by a label, such as the way to school, and give each group its count, mean and median. It can also check every value against a sensible range in one pass. Small groups swing: one unusual student moves a small group's mean much more than a large group's.