Course 2.2Lesson 5 of 5

Project: a data pipeline

In this project we build a data pipeline from generators. Each stage is a small generator that takes a stream of values and gives a new stream: one gives the non-empty lines of a text, the next gives the words of each line, the next cleans each word to small letters without punctuation, and the next drops the boring words. The last stage, which is a normal function, counts what arrives. Because every stage is lazy, a value passes through all the stages one at a time, and the whole text is never held in a big list, so the same pipeline can handle a file of any size. We plan the stages and the expected counts by hand, build and test each stage alone, and extend the pipeline with a stage for bigrams and a top-ten report.

about 25 min1 simulation

Lesson 5 of 5Show all lessons

Try the code and the simulations beside the lesson.

What you will be able to do

After this lesson you can

design a program as a pipeline of small stages,
write each stage as a generator that takes a stream and gives a stream,
test each stage alone, and
explain why a pipeline can handle data of any size.

The goal

We want the most common words of a text. You solved this with a list and a dictionary in the Essentials course. Now imagine that the text is a file of a million lines, too big to hold in a list. We need a way to handle it one piece at a time.

The idea is a pipeline, like a factory line: the text goes in at one end, and each stage does one job on each piece that passes. The pieces never pile up.

Step 1: plan the stages

Stage	In	Out	Job
`lines`	a text	lines	skip the empty lines
`words`	lines	words	cut each line into words
`clean`	words	words	small letters, no punctuation, drop empty
`without`	words	words	drop the boring words
`count`	words	a dictionary	count (a normal function: it needs all the words)

The first four are generators. Each takes a stream and gives a stream, so they can be joined: the output of one is the input of the next.

Step 2: the expected answer

The text has three lines:

text

The cat sat on the mat.
The dog barked at the cat!
A cat is not a dog, and a dog is not a cat.

Work out by hand. After cleaning and dropping the boring words (the, a, is, not, and, on, at), these words are left: cat, sat, mat, dog, barked, cat, cat, dog, dog, cat. So cat appears 4 times, dog 3 times, and sat, mat and barked once each.

Step 3: the program

Run it, and compare with your count.

Look at the design.

- Every stage is a generator: it has for ... in stream to take values and yield to give them. None of them makes a list.

- The stages are joined by nesting the calls: without(clean(words(lines(text))), boring). Read it from the inside out: lines, then words, then clean, then without. Each call just gives back a generator. Nothing runs until count asks for values.

- When count runs its for word in stream, it asks the last stage for a word. That stage asks the one before it, and so on, back to the first. One word travels through all the stages, and is counted, before the next word is asked for. At any moment, only one word is in the pipeline.

- clean uses strip(".,!?") to remove punctuation at the two ends, and yields only words that are not empty.

- count is the only stage that must see everything, so it is a normal function that returns a dictionary.

- The top three are found with sorted(counts, key=counts.get, reverse=True)[:3].

Because the pipeline is lazy, you could replace lines(text) with a generator that reads a file, one line at a time, and the same stages would handle a file of any size, with the memory of one line.

Step 4: test each stage alone

The great gain of small stages is that you can test each one by itself, with a list as the input.

Stage	Input	Expected
`clean`	`["The", "CAT!", "...", "dog,"]`	`["the", "cat", "dog"]`
`without`	`["a", "cat", "the"]` with boring `{"a", "the"}`	`["cat"]`
`words`	`["a b", "c"]`	`["a", "b", "c"]`
`lines`	`"a\n\nb\n"`	`["a", "b"]`
`count`	`["a", "b", "a"]`	`{"a": 2, "b": 1}`
the whole pipeline	an empty text	`{}`

Wrap each generator in list(...) to see its values. A list is a perfectly good stream: a stage does not care whether its input comes from a list, a file or another generator. This is the power of iterables.

Each stage is small and can be tested alone. The pipeline is read from the inside out: evens runs first.

Extend it

Pick one or two, and do them without looking at a solution. Write the expected answers first.

A top-ten report. Show the ten most common words with their counts.
A stage for pairs of words. pairs(stream) gives each pair of neighbouring words, so that you can count common pairs.
Read a real file. Write a generator file_lines(name) that yields the lines of a file one at a time, and use it as the first stage.
A stage that limits. first(stream, n) gives only the first n values, then stops.
A running total. A generator that yields the running count of words as they pass.

Common mistakes

Making a list inside a stage. The pipeline stops being lazy, and a big file fills the memory.
Using the stream twice. A generator is used up after one pass. Make a new pipeline for a second count.
Forgetting yield and using return. The stage gives one value and ends.
Joining the stages in the wrong order, such as words before lines.
Testing only the whole pipeline. Test each stage, and the empty input.

Practice

Start with two quick questions. Each New numbers gives you a fresh one, so you can practise as long as you like.

Then the labs and the problems:

Warm-up. Open the lab Project: the pipeline. The checker tests your stages alone and together, and with an empty text.
Build. Do Step 4 for one extension.
Stretch. Open the lab Write a generator again and write first(stream, n).

Remember these

A pipeline is a row of stages. Each is a generator that takes a stream and gives a stream.
Nothing runs until the last stage asks. Then one value travels through all the stages at a time.
Small stages can be tested alone with a list as input.
A lazy pipeline handles data of any size, with the memory of one piece.