Generators and yield

A Function With yield Does Not Return a Value — It Returns a Generator

Module 27 ended with a promise: everything you built by hand with DateRangeIterator — a class implementing __iter__ and __next__, tracking its own state between calls — Python can give you almost for free, with a single keyword. That keyword is yield.

A generator function is any function whose body contains at least one yield statement. Calling it does not run the function body at all — it immediately returns a generator object, which is a real, genuine iterator, automatically satisfying the entire iterator protocol from Module 27 without you writing a single __next__ method.

def count_up_to(limit):
    print("Starting the count!")
    n = 1
    while n <= limit:
        yield n
        n += 1

gen = count_up_to(3)
print(gen)   # <generator object count_up_to at 0x...> — no "Starting the count!" printed yet!
print(type(gen))   # <class 'generator'>

Nothing inside count_up_to has executed yet — not even the print("Starting the count!") on the very first line. Calling a generator function only creates the generator object; the body starts running only once you begin pulling values from it, which is the entire subject of Part 02.

The Mechanism That Trips Up Almost Everyone at First

Here is the part that takes real effort to internalize, because nothing else in Python behaves quite like it. When you call next() on a generator, the function body starts running from the top — and runs until it hits a yield statement. At that exact point, execution pauses, the yielded value is handed back to whoever called next(), and — critically — every local variable and the exact line the function was on are frozen in place, preserved completely. The next call to next() does not restart the function; it resumes execution exactly where it left off, right after that yield.

gen = count_up_to(3)

print(next(gen))
# Starting the count!     <- the function body finally starts running
# 1                        <- runs until "yield n" with n=1, then PAUSES here

print(next(gen))
# 2                        <- RESUMES right after "yield", runs "n += 1", loops, hits yield again

print(next(gen))
# 3                        <- same thing again

print(next(gen))
# StopIteration            <- the while loop condition is now False, function falls off the end

This is fundamentally different from a normal function call, where every call starts fresh from line one with no memory of any previous call. A generator function's local state — every variable, its current position in a loop, everything — survives between calls to next(), held in suspended animation. This is, under the hood, exactly how the Python interpreter implements __next__ for you automatically: it is genuinely running your function's bytecode, pausing it mid-execution, and resuming it later, something ordinary function calls simply cannot do.

Once the function body reaches its natural end — falls off the bottom, or hits an explicit return with no value — the generator raises StopIteration, exactly like any other exhausted iterator from Module 27. A return statement inside a generator does not send back a normal return value the way it would in a regular function; it simply ends the generator.

The Lazy Cousin of the List Comprehension

Module 12 covered list comprehensions in depth: [x**2 for x in range(10)] builds the entire list immediately, in memory, all at once. A generator expression uses nearly identical syntax — parentheses instead of square brackets — but produces values lazily, one at a time, exactly like a generator function does.

squares_list = [x**2 for x in range(1_000_000)]     # built ENTIRELY, right now, in memory
squares_gen  = (x**2 for x in range(1_000_000))     # produces nothing yet — lazy

print(type(squares_list))   # <class 'list'>
print(type(squares_gen))    # <class 'generator'>

print(next(squares_gen))    # 0   — the FIRST value, computed only now
print(next(squares_gen))    # 1
print(next(squares_gen))    # 4

Every rule from Module 27 about iterators applies directly: a generator expression is a single-pass iterator, exhausted after one full loop, and cannot be rewound or reused. It also supports every comprehension feature you already know — an if filter clause, nested loops, and multiple for clauses — the syntax carries over completely.

# Only even squares, computed lazily
even_squares = (x**2 for x in range(20) if x % 2 == 0)
print(list(even_squares))   # [0, 4, 16, 36, 64, 100, 144, 196, 256, 324]

Why This Actually Matters — A Concrete Memory Comparison

This is not an abstract, academic distinction. Imagine processing a 4 GB production log file, extracting every line that contains the string "ERROR". A list comprehension approach loads the entire file into memory before you can even start looking at the first error line. A generator processes the file one line at a time, holding only the current line in memory, no matter how large the file is.

def get_error_lines_list(filepath):
    with open(filepath) as f:
        return [line for line in f if "ERROR" in line]
        # Every single line of the file is read AND kept in memory before this
        # function even returns — for a 4 GB file, that is roughly 4 GB of RAM.

def get_error_lines_gen(filepath):
    with open(filepath) as f:
        for line in f:
            if "ERROR" in line:
                yield line
        # At any given moment, only the current line exists in memory.
        # A 4 GB file and a 4 MB file consume roughly the SAME peak memory here.

for error_line in get_error_lines_gen("app.log"):
    process(error_line)   # each line is handled and then can be garbage collected

The trade-off is real, not free: the generator version is typically slightly slower for small inputs, because of the pause/resume overhead on every single value, and it cannot be indexed, sliced, or looped over twice. But for anything genuinely large — log files, database result sets, API pagination, huge CSVs — the memory savings are not a minor optimization; they are frequently the difference between a script that runs and a script that gets killed by the operating system for exhausting available memory.

Delegating to a Sub-Generator

yield from hands off iteration to another iterable or generator entirely, yielding every value it produces in turn — without writing a manual for ... yield loop around it. It is syntactic sugar, but genuinely useful sugar that shows up constantly once generators start composing with each other.

def chain_manual(*iterables):
    for iterable in iterables:
        for item in iterable:
            yield item

list(chain_manual([1, 2], "ab", (True, False)))
# [1, 2, 'a', 'b', True, False]

def chain_delegated(*iterables):
    for iterable in iterables:
        yield from iterable

list(chain_delegated([1, 2], "ab", (True, False)))
# [1, 2, 'a', 'b', True, False] — identical result

This is genuinely more than a shorthand for a nested loop once you have generators calling other generators — a common shape when a large task is naturally broken into smaller sub-tasks, each expressed as its own generator function.

def read_section(name, rows):
    for row in rows:
        yield f"[{name}] {row}"

def read_full_report():
    yield from read_section("summary", ["total: 1200", "errors: 3"])
    yield from read_section("details", ["row A", "row B"])

for line in read_full_report():
    print(line)
# [summary] total: 1200
# [summary] errors: 3
# [details] row A
# [details] row B

Lazily Reading a Huge CSV File

Modules 15 and 16 covered reading files and working with CSV data. Here is where generators make that combination genuinely production-grade: a function that reads a large CSV file and yields one parsed row at a time, using the csv module, never holding the whole file's rows in memory at once.

import csv

def read_high_value_orders(filepath, minimum_total):
    with open(filepath, newline="") as f:
        reader = csv.DictReader(f)
        for row in reader:
            total = float(row["total"])
            if total >= minimum_total:
                yield {
                    "order_id": row["order_id"],
                    "customer": row["customer"],
                    "total": total,
                }

# Nothing is read from disk until you actually start pulling values:
for order in read_high_value_orders("orders_2026.csv", minimum_total=500):
    send_to_fulfillment_priority_queue(order)
    # Each row is read from disk, parsed, filtered, and processed — one at a time.
    # A 10-million-row CSV and a 10-row CSV use roughly the same peak memory here.

Notice that this reads exactly like a normal function using a normal for loop — the laziness is invisible at the call site, entirely a consequence of the single yield keyword inside. This is a genuinely important property of generators: the calling code does not need to know or care whether it is looping over a list or a generator — the for order in ... syntax is identical either way, exactly because generators fully satisfy the iterator protocol from Module 27.

Generators Are Not Always the Right Tool

Reaching for a generator by default, everywhere, is its own mistake. A list is the right choice whenever you genuinely need any of the things a single-pass iterator cannot give you:

data_gen = (x for x in range(10))

len(data_gen)          # TypeError — generators have no length; there is no way to
                        # know how many items remain without consuming them

data_gen[3]              # TypeError — generators cannot be indexed or sliced;
                          # there is no random access, only "the next value"

for x in data_gen: ...   # first pass — fully consumes it
for x in data_gen: ...   # second pass — produces NOTHING; already exhausted

If you need to know how many items there are before processing them, need to access items out of order, or need to loop over the same data more than once, materialize it into a list (or another concrete collection) up front. The honest rule of thumb: use a generator when data is large, processed once, and processed in order — use a list when you need to inspect, index, measure, or revisit it.

The Out-of-Memory Kill at a Raleigh Healthcare Data Company

def load_appointments(filepath):
    with open(filepath, newline="") as f:
        reader = csv.DictReader(f)
        return [row for row in reader]   # the ENTIRE file, all at once, in memory

def find_anomalies(filepath):
    appointments = load_appointments(filepath)   # 40M+ rows, now — several GB
    return [a for a in appointments if is_suspicious(a)]

def load_appointments(filepath):
    with open(filepath, newline="") as f:
        reader = csv.DictReader(f)
        yield from reader   # yield from Part 05 — delegates row by row

def find_anomalies(filepath):
    for appointment in load_appointments(filepath):
        if is_suspicious(appointment):
            yield appointment   # still lazy — the CALLER decides whether to
                                  # materialize this into a list or stream it further

Four Misconceptions About Generators

5 Interview Questions — With Complete Answers

Generator Mistakes That Cause Genuinely Confusing Bugs

Errors You Will Hit With Generators — And Exactly Why