Course 4.1Lesson 2 of 5

Clean the text and count the words

Real text is messy: The, the and THE are three different strings, and a word with a full stop is not the same as the word without. Before counting, clean the text in the same steps: make every letter small with lower, remove the punctuation with replace, cut the text into words with split, and then count with a dictionary or ask a question about the list. The key of the dictionary is the word and the value is how many times it was seen. max with key equals len gives the longest word, and max with key equals counts.get gives the most common word. A list comprehension can pick out the letters that pass a test.

about 30 min1 simulation2 practice questions

Lesson 2 of 5Show all lessons

Try the code and the simulations beside the lesson.

What you will be able to do

After this lesson you can

clean a text in four steps,
count the words with a dictionary,
find the longest word and the most common word, and
pick out letters or words that pass a test.

Real text is messy

The words The, the and THE are the same word, but Python sees three different strings. A word with a full stop, cat., is not the same as cat. So before we count, we clean.

Five boxes in a column show how a messy sentence becomes word counts. The text as it arrives is The cat, the HAT. With dot lower, every letter is made small: the cat, the hat. With dot replace, the punctuation is removed: the cat the hat. With dot split, the text is cut into the list the, cat, the, hat. Last, a dictionary counts each word: the is 2, cat is 1, hat is 1.

The steps are always the same:

Lower: make every letter small.
Replace: remove the punctuation.
Split: cut the text into words.
Count: use a dictionary, or ask a question about the list.

Predict first. After cleaning, how many words does the text have? Which word is the longest, and how many words have exactly 3 letters? Then run it.

The cleaned text has 10 words. The longest word is kitten, with 6 letters, and 7 words have 3 letters: the three times, and sat, ran, did and nap.

Count the words

You counted letters with a dictionary in the first course of this book. Counting words is the same idea: the key is a word, and the value is how many times we saw it.

Most of the shortest words carry little meaning. Leaving them out is a first step to finding what a text is about.

Where the method breaks

The cleaning above handles a simple text. Real texts have more trouble.

Apostrophes. don't becomes dont if you remove all the marks, and it's becomes its. For a count, that is often fine. For a careful job, keep the apostrophe.
Hyphens. well-known is one word or two? Decide, and be consistent.
Numbers and symbols. The cleaning above leaves digits as they are.
A better filter. Keep only the letters and spaces: build the new text letter by letter, and keep a letter if letter.isalpha() or it is a space. This removes every kind of mark without a list. You will use it in the project.

Common mistakes

Counting before cleaning. The and the are two words.
Removing the full stop and getting two words stuck together. replace(".", "") on "end.Start" gives endStart. Replace the mark by a space if words can touch it.
Forgetting that max(words) without a key gives the last word alphabetically.
Not keeping the result of lower(). Write tidy = text.lower().
Using the length of the text (len(text)) when you mean the number of words (len(words)).

Practice

Start with two quick questions. Each New numbers gives you a fresh one, so you can practise as long as you like.

Then the labs and the problems:

Warm-up. Open the problem The most common word and the problem Vowels in a sentence.
Build. Open the lab The word counter and the lab Count and clean. The checker uses texts with capital letters and punctuation.
Stretch. Open the problem The longest word and find what happens when two words are equally long.

Remember these

Clean first: lower(), then replace() for every punctuation mark.
Then split() for the words, and a dictionary to count them.
max(words, key=len) is the longest word. max(counts, key=counts.get) is the most common.
A list comprehension collects the items that pass a test.