Generators: functions that yield

What you will be able to do

After this lesson you can

• write a function with yield,
• follow it as it pauses and continues,
• write an endless generator and take only the first few values, and
• use a generator expression instead of a list.

yield

The class in the last lesson needed two methods and a lot of care. Python has a much shorter way to write an iterator: a generator. A generator is an ordinary function with one new word in it: yield.

Predict first. What type is g? Does count_to(3) print anything by itself? Then run it.

• Calling count_to(3) does not run the body. It gives a generator object. Nothing is shown.
• next(g) runs the body until the first yield. It gives out the value after yield (1) and pauses, with all its variables kept: i is 1.
• The next next(g) continues from where it paused: i = i + 1, the loop goes round, and the next yield gives 2.
• When the body ends, Python raises StopIteration, as for any iterator.
• A for loop, list and sum do all this for you: list(count_to(5)) is [1, 2, 3, 4, 5], and sum(count_to(100)) is 5050.

A generator is an iterator. All the rules of the last lesson hold: it is lazy, and it can be used once.

Seeing it pause

A generator is a function that can be stopped in the middle, and started again with its variables intact. A normal function can only run from the start to the end.

Endless generators

Because a generator makes a value only when asked, it can go on for ever.

fibonacci() is endless too. The loop at the end asks for values until one is over 1000, and then stops with break. The generator never has to know how many values you want. It is the caller who decides when to stop.

This is a strong idea: the source of the values and the use of the values are separate. The generator says how to make the next value, and the loop decides how many to take.

Generator expressions

A comprehension with round brackets makes a generator, and not a list.

A list of squares would need all its values at once. The generator holds only where it is, so it can go on as long as you ask.

When to use which

Common mistakes

• Using a generator twice. The second time it is empty.
• list(endless_generator()). It never ends. Use islice.
• Forgetting that calling the function runs nothing. The body runs on the first next.
• Using return value and expecting it to be one of the values. In a generator, return only ends it. Use yield.
• Printing the generator and seeing <generator object ...>. Convert with list to see the values, if it is finite.

Practice

Start with two quick questions. Each New numbers gives you a fresh one, so you can practise as long as you like.

Then the labs and the problems:

1. Warm-up. Open the lab Write a generator. You write two generators, and the checker takes values from them.
2. Build. Open the problem Follow the generator and the problem Endless and first.
3. Stretch. Open the problem Primes forever and write a generator of prime numbers.

Remember these

• A function with yield is a generator. Calling it gives an iterator. next runs it to the next yield.
• It pauses with its variables kept, and continues when asked again.
• A generator can be endless. islice takes the first few, and the loop decides when to stop.
• A generator expression, (... for ...), does not build a list. It can be used once.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. What does calling a generator function do?

2. What does yield do, compared with return?

3. Why is sum(n * n for n in range(10 ** 7)) better than sum([n * n for n in range(10 ** 7)])?