The Standard Library Toolkit — pathlib, collections, itertools, subprocess
Go beyond imports: navigate files safely, choose the right collection, compose iterators lazily, and run external programs with controlled inputs and errors.
Useful Tools Are Already Installed with Python
Python ships with a broad standard library. Before adding a dependency, check whether a well-supported module already handles the task. This lesson deepens four everyday tools: pathlib for paths, collections for specialized containers, itertools for lazy iteration, and subprocess for launching external programs.
You have already opened files and used modules and virtual environments. Here the goal is not to memorize every function; it is to recognize the shape of the problem, select an appropriate tool, and understand its safety and resource behavior.
Compose Paths Instead of Concatenating Strings
Path models a filesystem path with operations such as joining, checking, reading, and writing. The / operator joins path components using the platform's path rules. A relative path is interpreted from the process working directory, so production code often anchors it to a known location rather than assuming how the script was launched.
from pathlib import Path
PROJECT_DIR = Path(__file__).resolve().parent
DATA_DIR = PROJECT_DIR / "data"
input_file = DATA_DIR / "orders.csv"
if not input_file.is_file():
raise FileNotFoundError(f"Expected input file: {input_file}")
for path in sorted(DATA_DIR.glob("*.csv")):
print(path.name, path.stat().st_size)resolve() produces an absolute path, and parent navigates to its containing directory. For generated output, make the parent directory explicitly with mkdir(parents=True, exist_ok=True). Avoid deriving a path from untrusted input without checking where the resolved path ends up.
output_dir = PROJECT_DIR / "output" / "daily"
output_dir.mkdir(parents=True, exist_ok=True)
report = output_dir / "summary.txt"
report.write_text("processed 24 orders\n", encoding="utf-8")
text = report.read_text(encoding="utf-8")
print(text)Use a Container That Matches the Operation
The built-in list and dictionary cover many cases, but collections adds containers for common patterns. Counter counts hashable values, defaultdict initializes missing values, and deque supports efficient appends and pops at both ends. Choose based on the operations you need, not on novelty.
from collections import Counter, defaultdict, deque
events = ["placed", "delivered", "placed", "cancelled", "placed"]
counts = Counter(events)
print(counts.most_common(2))
orders_by_city: dict[str, list[str]] = defaultdict(list)
orders_by_city["Hyderabad"].append("ORD-1")
orders_by_city["Visakhapatnam"].append("ORD-2")
recent_ids = deque(["ORD-5", "ORD-6"], maxlen=3)
recent_ids.append("ORD-7")
print(list(recent_ids))A bounded deque discards items from the opposite end as new items arrive, making it useful for a rolling window or recent-event buffer. Do not use it as durable storage: when older entries fall off, they are gone.
Compose Lazy Iterators Without Unnecessary Lists
itertools provides building blocks for efficient iteration. Many return iterators, so they generate values on demand rather than materializing an entire result. This matters when processing a large input, but remember that an iterator is usually consumed once.
from itertools import chain, combinations, islice
morning = ["ORD-1", "ORD-2"]
evening = ["ORD-3", "ORD-4"]
all_orders = chain(morning, evening)
first_three = list(islice(all_orders, 3))
print(first_three)
pairs = combinations(["Asha", "Ravi", "Meera"], 2)
print(list(pairs))itertools.groupby groups adjacent equal keys, not every matching value in an arbitrary unsorted sequence. Sort by the same key first if you need all records for a category together.
from itertools import groupby
rows = [
{"city": "Hyderabad", "order": "ORD-1"},
{"city": "Chennai", "order": "ORD-2"},
{"city": "Hyderabad", "order": "ORD-3"},
]
rows.sort(key=lambda row: row["city"])
for city, group in groupby(rows, key=lambda row: row["city"]):
print(city, [row["order"] for row in group])Run Programs with an Explicit Contract
subprocess.run() starts another process and waits for it. Pass an argument list, set a timeout, and decide how to handle a non-zero exit code. With check=True, Python raises CalledProcessError if the command fails. Capture output when the parent program needs to inspect it.
import subprocess
result = subprocess.run(
["python", "--version"],
check=True,
capture_output=True,
text=True,
timeout=10,
)
print(result.stdout or result.stderr)For a known script, prefer a list of arguments and the current interpreter path from sys.executable. Do not build a shell command by interpolating user-controlled strings. If you have a command-specific reason to use a shell, treat it as a security-sensitive design decision and validate inputs carefully.
import subprocess
import sys
result = subprocess.run(
[sys.executable, "scripts/reconcile.py", "--date", "2026-11-09"],
check=False,
capture_output=True,
text=True,
timeout=30,
)
if result.returncode != 0:
print("Reconciliation failed:", result.stderr)
else:
print(result.stdout)shell=True for ordinary calls; use an argument sequence, restrict what executable can run, and never assume external output is trustworthy.A Small Decision Table
| Task | Tool | Remember |
|---|---|---|
| Join, inspect, or search filesystem paths | pathlib.Path | Anchor relative paths deliberately |
| Count values / group by a key / keep recent items | Counter / defaultdict / deque | Know whether missing keys mutate the container |
| Combine or sample a stream lazily | itertools | Iterators are one-pass unless recreated |
| Launch a child program | subprocess.run | Pass args, check status, add a timeout |
These modules fit together in a batch workflow: discover input files with pathlib, stream rows, count or group with collections, combine records with itertools, then optionally call a separate, controlled executable. Keep each transformation small enough to test independently.
Small Convenience Features Have Real Semantics
- Assuming a relative path starts beside the script: it starts from the process working directory unless anchored. Build from
__file__or configured paths. - Using a mutable dictionary default:
defaultdict(list)creates a list on missing-key access. Use a normal dict when missing reads should not mutate. - Using groupby on unsorted records: only adjacent keys are grouped. Sort by the same key first or use a different aggregation strategy.
- Forgetting iterators are consumed: create a fresh iterator when traversing again; materialize intentionally if reuse is required.
- Ignoring a subprocess failure or hanging process: inspect the return code, use
check=Truewhen exceptions are appropriate, and set a timeout.
Practice: Build a Safe Batch Summary
- Use
pathlibto find CSV files in a configured input directory and write a report into a created output directory. - Count delivery states with
Counterand group order IDs by city withdefaultdict(list). - Use
isliceto inspect only the first few rows of a large iterator without reading the whole file. - Group sorted records by city with
groupby; then intentionally remove the sort and explain the changed output. - Run a harmless child command with
subprocess.run, capture its output, and demonstrate how a timeout or nonzero status is handled.
Review question: which parts need to remain lazy, which results should be persisted, and what should happen when an expected input file is missing?
🎯 Key Takeaways
- ✓pathlib.Path makes path composition portable, but relative paths still depend on the process working directory.
- ✓Counter, defaultdict, and deque each encode a useful collection pattern; choose based on required operations.
- ✓itertools composes lazy, often one-pass iterators; do not materialize large sequences without a reason.
- ✓groupby groups adjacent keys, so sort by the grouping key first when aggregating arbitrary records.
- ✓Use subprocess.run with an argument list, an explicit status policy, captured output when needed, and a timeout.
- ✓The standard library can remove dependencies, but it does not remove the need for careful input, path, and failure handling.
What comes next
Lesson 20 begins object-oriented Python with classes and instances.
Lesson 20 → Classes and Objects — The BasicsHELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
