Working with APIs in Python

Talking to a Web API — The requests Library

Almost every real application eventually needs to talk to something outside itself — a payment processor, a weather service, an internal microservice owned by another team. That conversation almost always happens over HTTP, using the same request/response model a browser uses when it loads a page, except the response is data (usually JSON) instead of HTML meant for a human to read.

Python's standard library has a built-in way to make HTTP requests (urllib), but it is verbose and easy to get wrong. requests — a third-party package, installed with pip install requests — became the de facto standard years ago precisely because it makes the common case simple, and it is what you will find in the overwhelming majority of real Python codebases that talk to APIs.

import requests

response = requests.get("https://api.github.com/users/octocat")

print(response.status_code)   # 200
print(response.headers["content-type"])   # application/json; charset=utf-8
print(response.text[:80])     # the raw response body, as a string

requests.get() returns a Response object, not the data itself — it carries the status code, the headers, and the body, all together, so you can inspect exactly what came back before deciding how to use it. This is a deliberate design: unlike a plain function that either returns your data or throws, requests hands you the full picture and lets you decide what "success" means for your specific call, which matters a great deal once you reach Part 03.

Query Parameters, Custom Headers, and Sending Data

Query parameters

Rather than manually building a URL with a trailing ?key=value&key2=value2 string — easy to get wrong, especially once a value needs URL-encoding — pass a plain dictionary as the params argument, and requests builds the query string correctly for you.

import requests

response = requests.get(
    "https://api.openweathermap.org/data/2.5/weather",
    params={"q": "Austin,TX,US", "units": "imperial", "appid": "YOUR_KEY"},
)

print(response.url)
# https://api.openweathermap.org/data/2.5/weather?q=Austin%2CTX%2CUS&units=imperial&appid=YOUR_KEY
# — note "Austin,TX,US" was automatically URL-encoded; you never had to think about it

Headers

Headers describe metadata about the request — what format you accept back, how you are authenticating (Part 04), or a custom header a specific API requires. Pass them as a dictionary too, via the headers argument.

response = requests.get(
    "https://api.example.com/orders",
    headers={"Accept": "application/json", "X-Client-Version": "3.2.1"},
)

POST requests — sending a JSON body

Creating or updating something on the server almost always means a POST (or PUT/PATCH) request with a body. Passing a Python dictionary as the json argument (not data) does two things at once — it serializes the dictionary to a JSON string, and it sets the Content-Type: application/json header automatically, which most modern APIs require in order to parse the body correctly at all.

import requests

payload = {"customer_id": 4471, "item": "wireless-mouse", "quantity": 2}

response = requests.post(
    "https://api.example.com/orders",
    json=payload,
    headers={"Authorization": "Bearer YOUR_TOKEN"},
)

print(response.status_code)   # 201, typically, for a successful creation
created_order = response.json()

Status Codes and raise_for_status() — Failing Loudly, Not Silently

requests does not raise an exception just because a server responded with an error status like 404 or 500 — as far as the HTTP transport is concerned, a 500 Internal Server Error is still a complete, successful response; it just happens to carry bad news in its status code. If your code doesn't explicitly check for this, it will happily treat an error page as if it were real data.

response = requests.get("https://api.example.com/orders/99999999")
print(response.status_code)   # 404 — the order doesn't exist

data = response.json()   # might raise its own error, or might return an
                          # {"error": "not found"} body that your code processes
                          # as if it were a real order, with no crash at all

response.raise_for_status() closes this gap: it inspects the status code and raises an requests.exceptions.HTTPError if it is 4xx or 5xx, doing nothing at all if it is 2xx. Calling it immediately after every request is one of the single highest-value habits in this entire module.

response = requests.get("https://api.example.com/orders/99999999")
response.raise_for_status()   # raises HTTPError here — code below never runs
data = response.json()

# requests.exceptions.HTTPError: 404 Client Error: Not Found for url:
# https://api.example.com/orders/99999999

2xx   Success — 200 OK, 201 Created, 204 No Content (success, empty body)
3xx   Redirection — requests follows these automatically by default
4xx   Client error — YOUR request was wrong (401 Unauthorized, 404 Not Found,
      429 Too Many Requests — you are being rate-limited)
5xx   Server error — the API itself failed; often worth retrying (Part 06)

API Keys and Bearer Tokens

Most real APIs require proving who you are on every request. The two patterns you will meet constantly are an API key (a fixed secret string identifying your application) and a bearer token (typically a short-lived token obtained after a login/OAuth step). Both are almost always sent as a header, not as a URL parameter — putting a secret in a URL means it ends up in server logs, browser history, and any proxy in between.

response = requests.get(
    "https://api.example.com/data",
    headers={"X-API-Key": "sk_live_51H8..."},
)

response = requests.get(
    "https://api.example.com/account",
    headers={"Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."},
)

The exact header name and format (X-API-Key, Authorization: Bearer ..., Authorization: Token ...) varies by provider — always check the specific API's documentation rather than assuming.

The Single Most Common Production Bug in requests Code

Here is a fact that catches an enormous number of engineers, often only after it causes a real incident: requests has no default timeout. If the server on the other end never responds — a network issue, an overloaded upstream service, a firewall silently dropping the connection — a call like requests.get(url) with no timeout argument will simply hang, waiting indefinitely, for as long as the process is alive.

import requests

# If this endpoint never responds, this line never returns.
# Not "eventually times out" — genuinely never, by default.
response = requests.get("https://api.example.com/slow-endpoint")

In a script run once from a terminal, a hang is merely annoying. In a production service, it is a genuinely serious failure mode: a worker thread or process blocked on a single hung request stops doing anything else, requests pile up behind it, and — depending on how the service is deployed — this can exhaust an entire worker pool and take the whole service down, triggered by one slow upstream dependency that never actually errored, just never answered.

response = requests.get(
    "https://api.example.com/slow-endpoint",
    timeout=5,   # seconds — raises requests.exceptions.Timeout if exceeded
)

# A tuple lets you set connect and read timeouts separately —
# genuinely useful, since a slow TCP handshake and a slow response body
# are different failure modes worth distinguishing:
response = requests.get(
    "https://api.example.com/slow-endpoint",
    timeout=(3.05, 10),   # (connect timeout, read timeout)
)

What Can Actually Go Wrong, and Catching It Correctly

response.json() can itself fail

response.json() parses the response body as JSON and raises a requests.exceptions.JSONDecodeError if the body isn't valid JSON at all — which happens more often than it sounds, since a misconfigured proxy, a maintenance page, or a plain-text error message from a load balancer can all return a 200 or an error status with an HTML or plain-text body instead of JSON.

try:
    response = requests.get(url, timeout=5)
    response.raise_for_status()
    data = response.json()
except requests.exceptions.JSONDecodeError:
    log_error(f"Non-JSON response from {url}: {response.text[:200]!r}")
    raise

The requests exception hierarchy

Every exception requests can raise inherits from requests.exceptions.RequestException, which makes it possible to catch broad network problems in one place while still handling specific cases (like a timeout) differently when it matters.

import requests

try:
    response = requests.get(url, timeout=5)
    response.raise_for_status()
except requests.exceptions.Timeout:
    log_error(f"Timed out calling {url}")
except requests.exceptions.ConnectionError:
    log_error(f"Could not connect to {url} — DNS failure, refused connection, or network is down")
except requests.exceptions.HTTPError as e:
    log_error(f"{url} returned an error status: {e}")
except requests.exceptions.RequestException as e:
    # catches anything else in the requests exception family —
    # a genuine safety net, without swallowing unrelated bugs the way a bare
    # "except:" would (see the Exception Handling module for why that matters)
    log_error(f"Unexpected requests error calling {url}: {e}")
    raise

Retries with backoff

A single transient failure — a brief network blip, a 503 while an upstream service restarts — is often worth retrying automatically rather than failing the whole operation immediately. urllib3 (which requests is built on) ships a Retry helper that can be attached to a session to retry automatically, with exponential backoff, on specific status codes.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry_strategy = Retry(
    total=3,                                 # retry up to 3 times
    backoff_factor=0.5,                      # 0.5s, 1s, 2s between attempts
    status_forcelist=[429, 500, 502, 503, 504],
)
session.mount("https://", HTTPAdapter(max_retries=retry_strategy))

response = session.get("https://api.example.com/data", timeout=5)

requests.Session() — Connection Pooling for Repeated Calls

Every plain requests.get(...) or requests.post(...) call, on its own, opens a fresh TCP connection (and, for HTTPS, redoes the full TLS handshake) — real, measurable overhead that adds up quickly if your code makes many calls to the same host, such as paginating through an API's results or calling several endpoints on the same service in a row.

import requests

for page in range(1, 11):
    response = requests.get(
        "https://api.example.com/orders",
        params={"page": page},
        timeout=5,
    )
    process(response.json())
# 10 separate TCP connections and TLS handshakes to the same host

import requests

with requests.Session() as session:
    session.headers.update({"Authorization": "Bearer YOUR_TOKEN"})

    for page in range(1, 11):
        response = session.get(
            "https://api.example.com/orders",
            params={"page": page},
            timeout=5,
        )
        process(response.json())
# The underlying TCP connection is kept alive and reused across all 10 calls

A Session also lets you set default headers (like Authorization) once instead of repeating them on every call, which is both less error-prone and exactly what the retry configuration from Part 06 is typically attached to. For any code making more than a handful of calls to the same host, reaching for a Session over plain requests.get() calls is close to a default best practice.

The Missing Timeout That Took Down an Austin Fintech's Checkout

def check_fraud_score(order):
    response = requests.post(
        "https://fraud-api.vendor.com/score",
        json={"order_id": order.id, "amount": order.total},
        headers={"Authorization": f"Bearer {FRAUD_API_KEY}"},
    )
    response.raise_for_status()
    return response.json()["score"]

def check_fraud_score(order):
    response = requests.post(
        "https://fraud-api.vendor.com/score",
        json={"order_id": order.id, "amount": order.total},
        headers={"Authorization": f"Bearer {FRAUD_API_KEY}"},
        timeout=(3, 5),
    )
    response.raise_for_status()
    return response.json()["score"]

Four Misconceptions About Working with APIs in Python

5 Interview Questions — With Complete Answers

API Integration Mistakes Engineers Make Constantly

Errors You Will Hit Working with APIs — And Exactly Why