Files, Comprehensions and Generators: Quick Revision

A brief note on Files, Comprehensions and Generators. Read it before a test. Try each question at the end before you open its answer.

Text files

• with open(name, mode) as f: opens a file, and always closes it at the end of the block.

• write does not add a newline. Put \n at the end of each line yourself.
• A loop over a file gives its lines, each ending with \n. Use line.strip() before you compare.
• A missing file stops with a FileNotFoundError. Catch that error by name with try and except.

Lines as records, CSV and JSON

• A line can be a record. Strip it, split it, and change the types: everything read from a file is text.
• Expect bad lines. Keep the good ones, and report the bad ones with their line numbers.
• To change a record, change the data and write the whole file again with "w".
• CSV is a table of text. csv.reader gives each row as a list; csv.DictReader gives a dictionary, with the header as keys. Open it with newline="".
• JSON saves whole structures. json.dump and json.load use files; json.dumps and json.loads use texts.
• JSON keeps dictionaries with text keys, lists, texts, numbers, True, False and None. A tuple comes back as a list, a number key comes back as text, and a set cannot be saved.
• The usual pattern: load with a default, change, and save the whole data again.

Comprehensions and helpers

• [expression for item in iterable if condition] builds a new list in one line.
• A filter if goes at the end. A choice, a if condition else b, goes at the front.
• {k: v for ...} builds a dictionary, and {e for ...} builds a set.
• zip pairs two lists and stops at the shorter one. enumerate gives a position with each item.
• any is True when one item passes, and all when every item passes. For an empty list, any is False and all is True.
• lambda x: expression is a short function with no name, handy as a key.

Iterators and generators

• A for loop calls iter once, then next again and again, until StopIteration.
• An iterator gives its values once. A second loop over it gets nothing.
• A function with yield is a generator. Calling it runs nothing yet; each next runs it to the next yield, where it pauses with its variables kept.
• A generator expression, (x * x for x in data), makes values one at a time, without building a list.
• A generator can be endless. itertools.islice takes the first few.
• A pipeline is a row of generators, each taking a stream and giving a stream. It holds one piece of data at a time.

A worked example

What does this program show?

1. g = every_other(5) makes a generator, but runs none of its body. Nothing is printed yet.
2. The first next runs the body from the top: it prints start, reaches yield 5, and pauses. 5 is printed.
3. The second next continues after the yield: n becomes 3, and it yields 3.
4. list(g) asks for every value left. It yields 1; then n becomes $-1$, the loop ends, done is printed, and the generator stops.

The program shows start, 5, 3, done and [1], one on each line. The [1] comes last, because list must finish before print runs.

Practice with new numbers

A program to try

The program writes a small CSV file of rainfall, reads it through a generator that skips bad rows, and saves a summary as JSON.

Mistakes to avoid

• "w" when you meant to add. The old text is gone. Use "a".
• Sums with text. row["mm"] + 5 stops with TypeError: can only concatenate str (not "int") to str. Convert with float() first.
• Saving a set as JSON. It stops with TypeError: Object of type set is not JSON serializable. Save a list.
• else at the end of a comprehension. [x for x in xs if x > 0 else 0] is a SyntaxError. A choice goes at the front: [x if x > 0 else 0 for x in xs].
• Using a generator twice. The second pass is empty. Make a new generator.
• list() of an endless generator. It never ends. Use islice.

Check yourself

1. A diary program opens diary.txt with "w" each morning and writes that day's line. After it has run on three mornings, how many lines does the file hold?

2. What is {len(w) for w in ["ant", "bee", "wasp"]}, and why does it hold fewer values than the list of words?

3. A program runs pairs = zip(["Ira", "Om", "Tia"], [88, 92]). What does list(pairs) give?

4. A generator g yields 2, 5 and 9, and then ends. A program runs total = sum(g) and then count = len(list(g)). What are total and count?

5. A log file is 2 GB. Which is the better way to clean its lines, [line.strip() for line in f] or (line.strip() for line in f)? Why?