Comprehensions, Iterators and Generators

Name: Comprehensions, Iterators and Generators
Author: StemPy Lab

A for loop that builds a list is so common that Python has a short way to write it. In this course you write list, dictionary and set comprehensions, use the helpers zip, enumerate, any, all and sorted with a key, and see what a for loop really does underneath, with iterables and iterators. You write generators with yield, which give their values one at a time and so can handle very long or even endless sequences, and you join generators into a pipeline. A word-frequency pipeline is the project at the end. The practice runs your code on real data and tells you which results are wrong.

5 lessons, about 2 h 35 min5 simulations11 problems10 practice questionsPython in every lesson

Created by StemPy Lab

Course outline (5 lessons)

Lesson 1about 35 min
List comprehensions
A list comprehension builds a list in one line: the expression to collect, then for item in iterable, then an optional if that keeps only some items. It is a loop that appends, written short. The if with an else that chooses a value goes before the for. Two for parts make nested loops, with the first as the outer loop, and a comprehension inside a comprehension makes a table or a transposed table. A comprehension is for building a list, one idea at a time. If it needs several conditions and branches, or is used only for its side effects, a plain loop is clearer.
1 simulation2 practice questions3 Python programs3 problems1 Python lab1 question set
Read the lesson
Lesson 2about 30 min
Dictionaries, sets, and the helpers zip, enumerate, any and all
A dictionary comprehension writes key colon value then for, and a set comprehension uses curly brackets with one expression. Both can filter with if, and they build a dictionary or a set in one line. zip pairs the items of two lists, enumerate gives a position with each item, any is True if at least one item passes a test and all is True if every item passes, and for an empty collection any is False and all is True. sorted, min and max take a key function, which can be written as a short function with no name by using lambda. Use a named function when it is more than one short expression.
1 simulation2 practice questions2 Python programs3 problems1 Python lab
Read the lesson
Lesson 3about 30 min
Iterators: what a for loop really does
A for loop works in two steps. It asks the iterable for an iterator with iter, and then calls next on the iterator again and again until a StopIteration is raised. An iterable can give an iterator, such as a list, a text, a dictionary, a range or a file. An iterator remembers where it is and can be used only once, while a list can be looped over many times. To make your own iterator, write a class with the methods __iter__, which returns the object, and __next__, which gives the next value or raises StopIteration. Understanding this explains why a generator or a file can be looped over only once, and why range uses so little memory.
1 simulation2 practice questions2 Python programs2 problems1 Python lab
Read the lesson
Lesson 4about 35 min
Generators: functions that yield
A generator is a function with yield in it. Calling it does not run the body but gives a generator object, an iterator. Each next runs the body until the next yield, gives that value, and pauses with all its variables kept, then continues from there when asked again. When the body ends, StopIteration is raised. A generator is a short way to write an iterator, and it can be endless, because values are made only when asked. itertools.islice takes the first few values of an endless generator. A generator expression is a comprehension in round brackets that does not build a list, so sum of a generator uses little memory, and any and all stop at the first answer. A generator can be used only once.
1 simulation2 practice questions4 Python programs3 problems1 Python lab
Read the lesson
Lesson 5about 25 min
Project: a data pipeline
In this project we build a data pipeline from generators. Each stage is a small generator that takes a stream of values and gives a new stream: one gives the non-empty lines of a text, the next gives the words of each line, the next cleans each word to small letters without punctuation, and the next drops the boring words. The last stage, which is a normal function, counts what arrives. Because every stage is lazy, a value passes through all the stages one at a time, and the whole text is never held in a big list, so the same pipeline can handle a file of any size. We plan the stages and the expected counts by hand, build and test each stage alone, and extend the pipeline with a stage for bigrams and a top-ten report.
1 simulation2 practice questions1 Python program1 Python lab1 note
Read the lesson

Test Your Learning

Comprehensions, Iterators and Generators: Predict the Output

10 questions · 15 min

Not started

Your teacher

StemPy Lab

← Data In and Out

← Files and Data FormatsNext →

More onPython Going Further › Comprehensions, Iterators and Generators

Discussion (0)

Loading…

Comprehensions, Iterators and Generators

Course outline (5 lessons)

List comprehensions

Dictionaries, sets, and the helpers zip, enumerate, any and all

Iterators: what a for loop really does

Generators: functions that yield

Project: a data pipeline

Test Your Learning

Your teacher

Discussion (0)