Generators

yield makes a generator

A function with yield in it does not run when you call it. It hands back a generator, and the body runs a piece at a time as values are asked for:

Python

def counter():
    print("starting")
    yield 1
    print("between")
    yield 2
    print("finishing")

gen = counter()
print(type(gen).__name__)
print(next(gen))
print(next(gen))

Output

generator
starting
1
between
2

Note "starting" did not print until the first next(). The function is paused between yields, keeping its variables exactly where they were.

Usually you just loop

Python

def countdown(n):
    while n > 0:
        yield n
        n -= 1

for value in countdown(3):
    print(value)

print(list(countdown(3)))
print(sum(countdown(4)))

Output

3
2
1
[3, 2, 1]
10

Why bother: memory

A list holds every value at once. A generator holds one. For a large input that is the difference between working and not:

Python

def squares_list(n):
    return [i * i for i in range(n)]

def squares_gen(n):
    for i in range(n):
        yield i * i

print(sum(squares_list(1000)))
print(sum(squares_gen(1000)))
print(sum(i * i for i in range(1000)))

Output

332833500
332833500
332833500

Same answer three ways. The last two never build the list.

Why bother: it can be endless

Python

def naturals():
    n = 1
    while True:
        yield n
        n += 1

def take(source, count):
    for index, value in enumerate(source):
        if index >= count:
            return
        yield value

print(list(take(naturals(), 5)))

Output

[1, 2, 3, 4, 5]

naturals() never ends, and asking for five costs five. A list version would never return.

Generator expressions

The comprehension form, with round brackets:

Python

squares = (n * n for n in range(5))
print(type(squares).__name__)
print(list(squares))

Output

generator
[0, 1, 4, 9, 16]

Inside a call the extra brackets are unnecessary:

Python

print(sum(n * n for n in range(5)))
print(max(len(w) for w in ["a", "abc", "ab"]))
print(any(n > 3 for n in [1, 2, 5]))

Output

30
3
True

any and all stop at the first decisive value, so pairing them with a generator means the rest is never computed.

They are used up

This is the one thing that catches everyone:

Python

squares = (n * n for n in range(4))
print(sum(squares))
print(sum(squares))
print(list(squares))

Output

14
0
[]

The first sum consumed it. zip, map, filter and enumerate behave the same way. Wrap in list() if you need it twice.

yield from

Delegates to another generator, which keeps nested iteration flat:

Python

def letters():
    yield "a"
    yield "b"

def both():
    yield from letters()
    yield from [1, 2]

print(list(both()))

Output

['a', 'b', 1, 2]

Reading a large file

The pattern generators exist for. Nothing here holds more than one line:

Python

with open("log.txt", "w", encoding="utf-8") as f:
    f.write("ok 1\nerror 2\nok 3\nerror 4\n")

def lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.strip()

def only(source, word):
    for line in source:
        if line.startswith(word):
            yield line

print(list(only(lines("log.txt"), "error")))

Output

['error 2', 'error 4']

Each stage takes a generator and returns one. The file is read once, lazily, however large it is.

yield makes a generator

Usually you just loop

Why bother: memory

Why bother: it can be endless

Generator expressions

They are used up

yield from

Reading a large file

Test yourself

When does the body of a generator function first run?

Why does the second sum() over the same generator give 0?

Closures