Course 4.1Lesson 5 of 5

Project: a text toolkit

In this project we build a text toolkit: small functions for messy text that each do one job, and a table of tests that runs them all and shows PASS or FAIL. The functions give the tidy words of a text, count the words, find the most common word with a rule for ties, write a name in title case, and shift a text for a Caesar code. We plan the expected answers by hand, build the functions from the tools of the course, run the table of tests, and extend the toolkit with a palindrome test, a word frequency report and a code breaker.

about 25 min1 simulation2 practice questions

Lesson 5 of 5Show all lessons

Try the code and the simulations beside the lesson.

What you will be able to do

After this lesson you can

put the text tools of the course into functions that do one job each,
decide what to do with ties and with empty text,
test the functions with a table of cases, and
extend the toolkit with a tool of your own.

The goal

You will clean, count and change text often. Instead of writing the same loops again, make a toolkit: small functions, each with one job, that you can use in any program. And a table of tests, so that you know that each one is right.

The tools:

tidy_words(text): the words of a text, in small letters, with no punctuation.
word_counts(text): a dictionary of word and count.
most_common(text): the most common word. When two words tie, the one that comes first in the alphabet.
title_case(name): a name with a capital at the start of each word, and one space between the words.
shift_text(text, shift): the Caesar shift.

Step 1: the expected answers

Work them out by hand.

Tool	Input	Expected
`tidy_words`	`"The Cat, the HAT!"`	`["the", "cat", "the", "hat"]`
`word_counts`	`"a b a"`	`{"a": 2, "b": 1}`
`most_common`	`"the cat and the hat and the bat"`	`"the"`
`most_common` (a tie)	`"b a"`	`"a"`
`title_case`	`" aSHA rao "`	`"Asha Rao"`
`shift_text`	`"abc xyz"` with 3	`"def abc"`
`tidy_words` (empty)	`""`	`[]`

Step 2: the program

Run it. The last line says how many tests passed. Study the design.

- tidy_words builds a clean text letter by letter: keep a letter if it is a letter or a space, and drop everything else. This removes every kind of mark, without a list of marks. Then split() makes the words.

- word_counts uses tidy_words, so it reuses the first tool. Small tools make bigger tools. The counting is the dictionary pattern.

- most_common goes through the words in sorted order and keeps the best so far. Because the test is > (strictly more), a later word with the same count does not replace the earlier one, and the earlier one in the alphabet wins the tie. The rule is written in a comment. A tie rule that is not written down is a bug waiting to happen.

- title_case uses split() with no argument, which also removes the extra spaces, then capitalize() for each part, and join. The capitalize() method makes the first letter big and the rest small, so aSHA becomes Asha.

- shift_text is the Caesar function of the last lesson.

- The table is a list of tests. Each has a name, the function, a tuple of arguments, and the expected answer. The loop calls function(*arguments), with the star that unpacks the tuple.

Step 3: break something, and watch

A table of tests is only useful if it catches mistakes. Try these.

Change if best is None or counts[word] > counts[best] to >=. The tie test fails, because the later word now wins.
Remove letter == " " from tidy_words. All the words stick together, and several tests fail.
Remove % 26. The test with xyz fails.

After each, put the code back. This is the habit: write the tests, then change the code without fear.

This is why the toolkit cleans the text first. Every count after that treats A and a as one letter.

Step 4: extend it

Pick one or two, and do them without looking at a solution. Add each to the table, with the expected answer worked out by hand first.

A palindrome test. is_palindrome(text), ignoring capitals, spaces and marks.
The top three. top_words(text, 3): the three most common words, with their counts, the most common first.
The longest word. With a rule for ties.
A code breaker. crack(secret): guess the shift from the most common letter, as in the next course.
Initials. initials("asha kumari rao") gives "AKR".

Common mistakes

Mixing the cleaning into every function. One tidy_words, used by the others.
A tie rule that nobody wrote down.
Testing only a normal sentence. Test the empty text, one word, extra spaces and punctuation.
**Forgetting `*arguments`.**
Writing the expected answers from the output. Work them out by hand.

Practice

Start with two quick questions. Each New numbers gives you a fresh one, so you can practise as long as you like.

Then the labs and the problems:

Warm-up. Open the lab Cut and glue, if you have not done it, and the lab Find and count.
Build. Open the lab Project: a text toolkit. The checker tries your functions on texts that are different from the example.
Stretch. Open the lab Secret messages and the lab The word counter and write them as tools of your toolkit.