Course 2.1Lesson 5 of 5

Project: a safe data reader

In this project we write a safe data reader. Lines of the form name,mark come in from a messy source, with a mark in words, a missing mark, a mark out of range and an empty line. The program keeps the good lines in a dictionary, and reports each bad line with its line number and the reason, instead of stopping. We plan the cases and the expected result by hand, build the reader with try, except and a raised error, test the average of the good data and the list of problems, and extend the reader with a count of the kinds of problem and a limit on the number of problems.

about 25 min1 simulation2 practice questions

Lesson 5 of 5Show all lessons

Try the code and the simulations beside the lesson.

What you will be able to do

After this lesson you can

read messy data without crashing,
keep the good lines and report the bad ones with a reason,
combine try, except, raise and continue in one loop, and
test a program with good and bad input.

The goal

Data from the real world is messy. A teacher types the marks of a class into a file, one line for each student, in the form name,mark. But people make mistakes: a mark in words, a missing mark, a mark of 140, an empty line. A program that stops at the first bad line is no use. A program that skips the bad lines silently is worse, because the teacher does not know that anything went wrong.

We want a reader that

keeps every good line in a dictionary of name: mark,
reports every bad line, with its line number and the reason,
never stops, and
still gives the average of the good marks.

Step 1: the cases, and the expected result

The lines are:

Line	Text	Verdict
1	`Asha,72`	good
2	`Ravi,85`	good
3	`Meera,sixty`	bad: the mark is not a whole number
4	`Kabir`	bad: no comma, so the line does not split into two
5	`Ana,140`	bad: out of range
6	`Dev,-3`	bad: out of range
7	`Tara,90`	good
8	(empty)	bad: no comma

The expected result: five problems, the lines 3, 4, 5, 6 and 8. The good dictionary has Asha 72, Ravi 85 and Tara 90. The average is 247 divided by 3, which is about 82.33.

Step 2: the program

Run it, and compare with your table.

Look at the loop, line by line.

- enumerate(lines, start=1) gives the line number and the line. The numbers start at 1, as people count.

- name, text = line.split(",") splits the line at the comma. If the line has no comma, or more than one, the unpacking fails with a ValueError (not enough values to unpack, or too many values). This is the case of line 4 and line 8, and we did not write a single if for them.

- mark = int(text) fails with a ValueError for sixty.

- if mark < 0 or mark > 100: raise ValueError(...) makes our own error for the numbers out of range, with a message. So all the three kinds of problem end in the same except ValueError part. One part, one place to report.

- In the except, error holds the message. We store a line that names the line number and the reason, and continue goes on to the next line. The bad line is not added.

- good[name] = mark is outside the try, after the continue, so it runs only for good lines.

The messages from Python itself (invalid literal for int() with base 10: 'sixty') are a little technical, but they say what happened. A tidier reader could write its own sentence for each case, with raise ValueError("the mark is not a number").

Step 3: test

Use the table as your test. All eight lines have a known verdict. Then test the edges.

Test	Expected
a mark of exactly 0	good
a mark of exactly 100	good
a mark of 101	bad
a name with spaces, such as `Mary Ann,80`	good (the split is at the comma only)
two commas, `Ravi,85,90`	bad
spaces around the number, `Ravi, 85`	good: `int(" 85")` works
no good line at all	the average would divide by zero

The last row is a real fault in the program: if every line is bad, good is empty, and sum(...) / len(good) stops with a ZeroDivisionError. A safe reader must check if good: before the average.

The if good: at the end is the check that the reader needs when every line is bad.

Step 4: extend it

Pick one or two, and do them without looking at a solution. Write the expected answers first.

Protect the average. Show No good lines instead of dividing by zero.
Count the kinds of problem. Use the counting dictionary: how many lines had a bad number, and how many had a bad form?
A limit. Stop reading after 5 problems, and say that the file looks wrong.
A duplicate name. If a name appears twice, report the second line as a problem.
A function. Put the reader in a function read_marks(lines) that returns the dictionary and the list of problems as a tuple.

Common mistakes

Putting good[name] = mark inside the try. Then you do not know which line failed.
Forgetting continue. The bad line is added with an old value of mark.
Using a bare except:. A typing mistake in the reader is hidden.
Dividing by len(good) with no good lines.
Reporting the problem without the line number. The teacher cannot find it.

Practice

Start with two quick questions. Each New numbers gives you a fresh one, so you can practise as long as you like.

Then the labs and the problems:

Warm-up. Open the lab Project: a safe data reader. The checker gives your function lines that are different from the example.
Build. Do Step 4 for one extension.
Stretch. Open the problem Skip the bad values and the problem An empty list, and see how the ideas appear in the reader.

Remember these

Real data is messy. Keep the good, and report the bad with a reason.
One try around the risky lines, one except for the kinds that you expect, continue to go on.
Raise your own error for a rule that Python does not check, such as a range.
Test the empty case, the edges, and the case where everything is bad.