The Standard Library Toolkit — pathlib, collections, itertools, subprocess

Go beyond imports: navigate files safely, choose the right collection, compose iterators lazily, and run external programs with controlled inputs and errors.

Est. 50 min Updated Suggest a fix
// Part 01 — Build on the Standard Library

Useful Tools Are Already Installed with Python

Python ships with a broad standard library. Before adding a dependency, check whether a well-supported module already handles the task. This lesson deepens four everyday tools: pathlib for paths, collections for specialized containers, itertools for lazy iteration, and subprocess for launching external programs.

You have already opened files and used modules and virtual environments. Here the goal is not to memorize every function; it is to recognize the shape of the problem, select an appropriate tool, and understand its safety and resource behavior.

🎯 Pro Tip
A good script remains portable: build paths from a known directory, stream large inputs when possible, and treat external process output and exit status as data that must be checked.
// Part 02 — pathlib

Compose Paths Instead of Concatenating Strings

Path models a filesystem path with operations such as joining, checking, reading, and writing. The / operator joins path components using the platform's path rules. A relative path is interpreted from the process working directory, so production code often anchors it to a known location rather than assuming how the script was launched.

Anchor data relative to the script
from pathlib import Path

PROJECT_DIR = Path(__file__).resolve().parent
DATA_DIR = PROJECT_DIR / "data"
input_file = DATA_DIR / "orders.csv"

if not input_file.is_file():
    raise FileNotFoundError(f"Expected input file: {input_file}")

for path in sorted(DATA_DIR.glob("*.csv")):
    print(path.name, path.stat().st_size)

resolve() produces an absolute path, and parent navigates to its containing directory. For generated output, make the parent directory explicitly with mkdir(parents=True, exist_ok=True). Avoid deriving a path from untrusted input without checking where the resolved path ends up.

Create an output directory and write text
output_dir = PROJECT_DIR / "output" / "daily"
output_dir.mkdir(parents=True, exist_ok=True)
report = output_dir / "summary.txt"
report.write_text("processed 24 orders\n", encoding="utf-8")

text = report.read_text(encoding="utf-8")
print(text)
⚠️ Important
A filename supplied by a user can contain path components. If a program accepts one, validate the resolved destination against the intended base directory; joining paths alone does not enforce containment.
// Part 03 — collections

Use a Container That Matches the Operation

The built-in list and dictionary cover many cases, but collections adds containers for common patterns. Counter counts hashable values, defaultdict initializes missing values, and deque supports efficient appends and pops at both ends. Choose based on the operations you need, not on novelty.

Count event types and group IDs
from collections import Counter, defaultdict, deque

events = ["placed", "delivered", "placed", "cancelled", "placed"]
counts = Counter(events)
print(counts.most_common(2))

orders_by_city: dict[str, list[str]] = defaultdict(list)
orders_by_city["Hyderabad"].append("ORD-1")
orders_by_city["Visakhapatnam"].append("ORD-2")

recent_ids = deque(["ORD-5", "ORD-6"], maxlen=3)
recent_ids.append("ORD-7")
print(list(recent_ids))

A bounded deque discards items from the opposite end as new items arrive, making it useful for a rolling window or recent-event buffer. Do not use it as durable storage: when older entries fall off, they are gone.

// Part 04 — itertools

Compose Lazy Iterators Without Unnecessary Lists

itertools provides building blocks for efficient iteration. Many return iterators, so they generate values on demand rather than materializing an entire result. This matters when processing a large input, but remember that an iterator is usually consumed once.

Chain sources, take a sample, form combinations
from itertools import chain, combinations, islice

morning = ["ORD-1", "ORD-2"]
evening = ["ORD-3", "ORD-4"]
all_orders = chain(morning, evening)
first_three = list(islice(all_orders, 3))
print(first_three)

pairs = combinations(["Asha", "Ravi", "Meera"], 2)
print(list(pairs))

itertools.groupby groups adjacent equal keys, not every matching value in an arbitrary unsorted sequence. Sort by the same key first if you need all records for a category together.

Sort before using groupby
from itertools import groupby

rows = [
    {"city": "Hyderabad", "order": "ORD-1"},
    {"city": "Chennai", "order": "ORD-2"},
    {"city": "Hyderabad", "order": "ORD-3"},
]
rows.sort(key=lambda row: row["city"])
for city, group in groupby(rows, key=lambda row: row["city"]):
    print(city, [row["order"] for row in group])
💡 Note
Do not convert every iterator to a list automatically. Materialize only when you need repeated traversal, indexing, or a concrete collection and the size is reasonable.
// Part 05 — subprocess

Run Programs with an Explicit Contract

subprocess.run() starts another process and waits for it. Pass an argument list, set a timeout, and decide how to handle a non-zero exit code. With check=True, Python raises CalledProcessError if the command fails. Capture output when the parent program needs to inspect it.

Call a known executable with separate arguments
import subprocess

result = subprocess.run(
    ["python", "--version"],
    check=True,
    capture_output=True,
    text=True,
    timeout=10,
)
print(result.stdout or result.stderr)

For a known script, prefer a list of arguments and the current interpreter path from sys.executable. Do not build a shell command by interpolating user-controlled strings. If you have a command-specific reason to use a shell, treat it as a security-sensitive design decision and validate inputs carefully.

Run this project with its active interpreter
import subprocess
import sys

result = subprocess.run(
    [sys.executable, "scripts/reconcile.py", "--date", "2026-11-09"],
    check=False,
    capture_output=True,
    text=True,
    timeout=30,
)
if result.returncode != 0:
    print("Reconciliation failed:", result.stderr)
else:
    print(result.stdout)
⚠️ Important
A timeout does not make an unsafe command safe. Avoid shell=True for ordinary calls; use an argument sequence, restrict what executable can run, and never assume external output is trustworthy.
// Part 06 — Choosing the Tool

A Small Decision Table

TaskToolRemember
Join, inspect, or search filesystem pathspathlib.PathAnchor relative paths deliberately
Count values / group by a key / keep recent itemsCounter / defaultdict / dequeKnow whether missing keys mutate the container
Combine or sample a stream lazilyitertoolsIterators are one-pass unless recreated
Launch a child programsubprocess.runPass args, check status, add a timeout

These modules fit together in a batch workflow: discover input files with pathlib, stream rows, count or group with collections, combine records with itertools, then optionally call a separate, controlled executable. Keep each transformation small enough to test independently.

// Part 07 — Common Mistakes

Small Convenience Features Have Real Semantics

  • Assuming a relative path starts beside the script: it starts from the process working directory unless anchored. Build from __file__ or configured paths.
  • Using a mutable dictionary default: defaultdict(list) creates a list on missing-key access. Use a normal dict when missing reads should not mutate.
  • Using groupby on unsorted records: only adjacent keys are grouped. Sort by the same key first or use a different aggregation strategy.
  • Forgetting iterators are consumed: create a fresh iterator when traversing again; materialize intentionally if reuse is required.
  • Ignoring a subprocess failure or hanging process: inspect the return code, use check=True when exceptions are appropriate, and set a timeout.
// Part 08 — Practice

Practice: Build a Safe Batch Summary

  1. Use pathlib to find CSV files in a configured input directory and write a report into a created output directory.
  2. Count delivery states with Counter and group order IDs by city with defaultdict(list).
  3. Use islice to inspect only the first few rows of a large iterator without reading the whole file.
  4. Group sorted records by city with groupby; then intentionally remove the sort and explain the changed output.
  5. Run a harmless child command with subprocess.run, capture its output, and demonstrate how a timeout or nonzero status is handled.

Review question: which parts need to remain lazy, which results should be persisted, and what should happen when an expected input file is missing?

🎯 Key Takeaways

  • ✓pathlib.Path makes path composition portable, but relative paths still depend on the process working directory.
  • ✓Counter, defaultdict, and deque each encode a useful collection pattern; choose based on required operations.
  • ✓itertools composes lazy, often one-pass iterators; do not materialize large sequences without a reason.
  • ✓groupby groups adjacent keys, so sort by the grouping key first when aggregating arbitrary records.
  • ✓Use subprocess.run with an argument list, an explicit status policy, captured output when needed, and a timeout.
  • ✓The standard library can remove dependencies, but it does not remove the need for careful input, path, and failure handling.

What comes next

Lesson 20 begins object-oriented Python with classes and instances.

Lesson 20 → Classes and Objects — The Basics

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...