Iterators: what a for loop really does

What you will be able to do

After this lesson you can

• explain what a for loop does with iter and next,
• tell an iterable from an iterator,
• write a class that can be used in a for loop, and
• explain why an iterator can be used only once.

What a for loop really does

You have written for n in numbers: hundreds of times. But how does Python go from the list to the items, one at a time? It does two things, and you can do them by hand.

Predict first. What does the second next give, and what happens at the fourth? Then run it.

1. iter(numbers) asks the list for an iterator: a small object that remembers where it is in the list.
2. next(it) asks the iterator for the next value. The first call gives 10, the second 20, the third 30.
3. When nothing is left, the iterator raises StopIteration. This is not an error in the usual sense. It is the signal that the loop is over.

The for loop does exactly this: it calls iter once, then next again and again, and catches StopIteration to end quietly. The try and except in the program shows what the loop hides.

Iterable, and iterator

Two words, and the difference matters.

The program shows the difference. list(it) gives all the items the first time, and an empty list the second time: the iterator is used up. But list(numbers) gives the items again and again, because each call gets a fresh iterator from the list.

This matters in real programs. If you loop over an iterator once, and then try again, you get nothing, and no error. A file object and a generator behave this way too.

Your own iterator

To make your own kind of thing that works in a for loop, write a class with two special methods.

This is a lot of machinery for a countdown. In the next lesson, you will see that generators do the same job in a few lines. But understanding the machinery shows you what a generator is.

An iterator can only go forward. Once it is used up, list(it) gives an empty list: start a new iterator to go again.

Lazy and eager

A list holds all its items in memory at once. A range(10 ** 9) does not: it only remembers the start, the stop and the step, and makes each number when asked. We say that range is lazy. An iterator is often lazy: it makes a value only when it is asked for one, and then forgets it.

Why does that matter? A list of a billion numbers needs gigabytes of memory, and a computer would run out. A lazy iterator needs a few bytes, and gives its values one at a time. For data that is big, or endless, lazy is the only way.

Common mistakes

• Looping twice over an iterator and getting nothing the second time.
• Forgetting raise StopIteration in __next__. The loop never ends.
• Forgetting to move on in __next__. The same value for ever.
• Writing __next__ without __iter__. The for loop first asks for iter.
• Calling next on a list. A list is an iterable, not an iterator. Use next(iter(my_list)).

Practice

Start with two quick questions. Each New numbers gives you a fresh one, so you can practise as long as you like.

Then the labs and the problems:

1. Warm-up. Open the problem Iterable or iterator? and sort six examples.
2. Build. Open the lab Build an iterator class. You write a class with __iter__ and __next__, and the checker uses it in loops.
3. Stretch. Open the problem Used up and predict what a second loop gives.

Remember these

• A for loop calls iter once, and next again and again, until StopIteration.
• An iterable can give an iterator. An iterator gives the next value, and can be used once.
• A class with __iter__ and __next__ can be used in for, list and sum.
• Lazy iterators make values only when asked, so they can be huge or endless.

Check yourself

Try each one on your own first. If you are stuck, open a hint. Then check with Show the answer.

1. What does next(it) do when the iterator has no more values?

2. What does list(it) give if you call it twice on the same iterator?

3. Which special method gives the next value?