Python Performance — Profiling and Optimisation

python -m cProfile -s cumulative my_script.py

import cProfile

def process_all_records(records):
    return [transform(r) for r in records]

cProfile.run("process_all_records(records)")

         1000004 function calls in 2.145 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.012    0.012    2.145    2.145 script.py:4(process_all_records)
   500000    1.203    0.000    1.980    0.000 script.py:8(is_duplicate)
   500000    0.777    0.000    0.777    0.000 {method 'append' of 'list' objects}
        1    0.153    0.153    0.153    0.153 script.py:2(load_records)

def is_duplicate(record, seen):
    return record["id"] in seen           # membership check on a LIST

def deduplicate(records):
    seen = []
    result = []
    for record in records:
        if not is_duplicate(record, seen):
            result.append(record)
            seen.append(record["id"])
    return result

def deduplicate(records):
    seen = set()          # membership check is O(1) on average, not O(n)
    result = []
    for record in records:
        if record["id"] not in seen:
            result.append(record)
            seen.add(record["id"])
    return result

# Same logic, same result — O(n) overall instead of O(n²).
# On 500,000 records, this is the difference between roughly 5 seconds and
# a genuinely unusable multi-minute runtime.

import timeit

def concat_with_plus():
    result = ""
    for i in range(1000):
        result += str(i)
    return result

def concat_with_join():
    return "".join(str(i) for i in range(1000))

print(timeit.timeit(concat_with_plus, number=1000))    # e.g. 0.412 seconds total
print(timeit.timeit(concat_with_join, number=1000))     # e.g. 0.187 seconds total — clearly faster

1. Wrong data structure?
   -> list membership checks in a loop, linear search for something a dict/set
      would find in O(1) — usually the highest-leverage, lowest-risk fix.

2. Wrong algorithm?
   -> nested loops that could be restructured (e.g. sorting once instead of
      repeatedly scanning), redundant repeated work that could be computed once.

3. Repeated expensive work with the same inputs?
   -> functools.lru_cache (covered in the Decorators module) or a manual cache,
      IF the function is pure (same input always -> same output) and called
      repeatedly with overlapping inputs.

4. Genuinely CPU-bound work at the limits of what pure Python can do?
   -> reach for NumPy/pandas (next module) for vectorised numeric work, or
      multiprocessing (covered earlier in this phase) for true parallelism.

5. Is it actually a problem worth fixing at all?
   -> a function that runs once at startup taking 200ms extra is very often
      not worth any engineering time, no matter how "inefficient" it looks.

A nightly job deduplicating that day's transaction records ran in 5 minutes when it was written, against a modest dataset. A year of organic business growth later, it takes over 3 hours and increasingly threatens to miss its overnight processing window entirely. Two engineers separately assume, without profiling, that the database write step must be the bottleneck and spend a day investigating batch-write tuning with no meaningful improvement.

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   2000000    847.2    0.000   847.2    0.000 dedupe.py:12(is_duplicate)
        1      0.4      0.4     3.1      3.1 dedupe.py:31(write_to_database)

The actual bottleneck, and why it had been invisible for a year

is_duplicate — a small, unremarkable-looking helper checking membership against a plain Python list — accounted for over 99% of total runtime, not the database write step everyone had assumed. The bug had existed since the code was first written; it was simply invisible when the dataset was small enough that O(n²) still finished in seconds. As the business grew and transaction volume grew with it, the same unchanged code silently crossed from "fine" to "the single largest operational risk in the nightly pipeline," with no code change ever having introduced the regression — only data volume did.

seen = set()   # was: seen = []
# ...
if record["id"] not in seen:   # now O(1) instead of O(n)

The job's runtime dropped from over 3 hours back to under 2 minutes — faster than its original 5-minute runtime a year earlier, since the fixed version now scales linearly instead of quadratically. The team's retrospective conclusion: "an hour with a profiler would have found this on day one of the slowdown; a full day of tuning the wrong subsystem found nothing, because nobody had actually measured where the time was going."