Calling LLM APIs — SDKs, Messages and Streaming

Make your first model call from Python, structure system and user messages, stream tokens to the user, and handle errors and retries.

Est. 35 min Updated Suggest a fix
// Part 01 — The big picture

An LLM API Is Just an HTTPS Request

Every AI feature you have used — a support chatbot on a bank website, a resume reviewer, a code assistant — sends text to a model over HTTPS and gets text back. There is no magic in the transport. Your code builds a request containing instructions, the conversation so far, and some settings; the provider runs the model; you receive generated text plus usage information you will be billed for.

The three providers you will meet most often are OpenAI, Anthropic, and Google (Gemini). Their request shapes differ slightly, but the ideas are identical. Learn the ideas once and switching providers becomes a small refactor rather than a rewrite.

ConceptWhat it meansWhy you care
ModelThe specific model version you callDecides quality, speed, and price per token
System / instructionsRules the model should follow for the whole conversationSets tone, scope, and output format
MessagesThe ordered list of user and assistant turnsGives the model context; the API itself is stateless
Max output tokensUpper bound on the reply lengthCaps cost and latency
UsageInput and output token counts returned with each replyThis is what you pay for
💡 Note
The API is stateless. The model does not remember your previous request. A "conversation" is your code re-sending the earlier messages every time. That single fact explains most cost surprises in chat apps.
// Part 02 — Setup

Install an SDK and Keep Your Key Out of Your Code

Official SDKs wrap the HTTP calls, add types, and handle retries. Create a virtual environment and install the SDK for the provider you have a key for. The examples use OpenAI and Anthropic; the pattern is the same for Gemini.

bash
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install openai anthropic python-dotenv

API keys are passwords that spend money. Put them in an environment variable or a .env file that is listed in .gitignore — never in source code, screenshots, or a notebook you share.

.env
bash
OPENAI_API_KEY=sk-...your-key...
ANTHROPIC_API_KEY=sk-ant-...your-key...
LLM_MODEL=the-model-name-from-your-provider-docs
⚠️ Important
Leaked keys are found by automated scanners within minutes of being pushed to a public repository. If a key ever leaks, revoke it in the provider dashboard first, then rotate it everywhere it is used.
🎯 Pro Tip
Model names change often. Read the model name from configuration (LLM_MODEL above) instead of hard-coding it, so upgrading a model is a config change, not a code change.
// Part 03 — First call

Your First Request in Python

Here is the smallest useful call with each SDK. Both send an instruction and one user message, then print the reply and token usage.

first_call_openai.py
python
import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI()  # reads OPENAI_API_KEY from the environment

response = client.responses.create(
    model=os.environ["LLM_MODEL"],
    instructions="You are a concise assistant for Indian college students.",
    input="Explain what an API is in two sentences.",
    max_output_tokens=200,
)

print(response.output_text)
print(response.usage)  # input_tokens, output_tokens, total_tokens
first_call_anthropic.py
python
import os
from dotenv import load_dotenv
import anthropic

load_dotenv()
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

message = client.messages.create(
    model=os.environ["LLM_MODEL"],
    max_tokens=200,
    system="You are a concise assistant for Indian college students.",
    messages=[{"role": "user", "content": "Explain what an API is in two sentences."}],
)

print(message.content[0].text)
print(message.usage)  # input_tokens, output_tokens

Notice what is the same: a model name, an instruction, user content, a length limit, and usage in the response. Those five things are the whole contract.

// Part 04 — Conversations

Messages, Roles and Keeping History

To hold a conversation, keep a list of messages and append each turn. Roles tell the model who said what: user for the person, assistant for earlier model replies, and a system or instructions field for standing rules.

chat_loop.py
python
import os
import anthropic

client = anthropic.Anthropic()
SYSTEM = "You are a friendly tutor. Answer in at most 4 sentences."
history: list[dict] = []

while True:
    user_text = input("You: ").strip()
    if user_text.lower() in {"quit", "exit"}:
        break
    history.append({"role": "user", "content": user_text})

    reply = client.messages.create(
        model=os.environ["LLM_MODEL"],
        max_tokens=400,
        system=SYSTEM,
        messages=history,          # the WHOLE conversation, every time
    )
    answer = reply.content[0].text
    history.append({"role": "assistant", "content": answer})
    print("Tutor:", answer)

Because you resend history on every turn, input tokens grow with each message. Real apps trim or summarise old turns once the conversation gets long — you will learn budgeting in the next lesson.

📌 Real World Example
A food-delivery support bot does not need the last 50 messages to answer "where is my order?". It needs the system rules, the order details fetched from the database, and the last few turns. Sending less context is cheaper, faster, and often more accurate.
// Part 05 — Streaming

Stream Tokens So Users Are Not Staring at a Spinner

A long answer can take several seconds to finish. Streaming sends tokens as they are generated, so the first words appear almost immediately. Perceived speed matters more to users than total time.

stream_openai.py
python
import os
from openai import OpenAI

client = OpenAI()

stream = client.responses.create(
    model=os.environ["LLM_MODEL"],
    input="Give me 5 tips to prepare for a data engineering interview.",
    stream=True,
)

for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
print()
stream_anthropic.py
python
import os
import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model=os.environ["LLM_MODEL"],
    max_tokens=600,
    messages=[{"role": "user", "content": "Give me 5 tips to prepare for a data engineering interview."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
print()

In a web app, your server forwards these chunks to the browser using server-sent events or a streaming HTTP response. You will build that in the final lesson of this course.

// Part 06 — Reliability

Errors, Timeouts and Retries

Production calls fail for ordinary reasons. Handle each kind deliberately rather than wrapping everything in one bare except.

FailureTypical statusWhat to do
Rate limited429Retry with exponential backoff and jitter; reduce concurrency
Provider overloaded or server error500, 503, 529Retry a few times, then fall back or show a friendly error
Timeout—Set an explicit timeout; retry once; stream long answers
Bad request400Do not retry — fix the request (often too many tokens or a bad parameter)
Authentication401, 403Do not retry — check the key and permissions

Both SDKs already retry some errors. Configure the limits explicitly so behaviour is predictable:

reliable_client.py
python
import os
import time
import random
from openai import OpenAI, APIStatusError, APITimeoutError, RateLimitError

client = OpenAI(timeout=30.0, max_retries=2)   # SDK-level retries for transient errors

def ask(prompt: str, attempts: int = 3) -> str:
    for attempt in range(1, attempts + 1):
        try:
            response = client.responses.create(
                model=os.environ["LLM_MODEL"], input=prompt, max_output_tokens=300,
            )
            return response.output_text
        except (RateLimitError, APITimeoutError):
            if attempt == attempts:
                raise
            time.sleep(2 ** attempt + random.random())   # backoff with jitter
        except APIStatusError as error:
            if error.status_code >= 500 and attempt < attempts:
                time.sleep(2 ** attempt)
                continue
            raise   # 4xx errors are bugs in our request: surface them
    raise RuntimeError("unreachable")
🎯 Pro Tip
Log the request ID that providers return with every response. When you raise a support ticket or debug a bad answer, that ID is the fastest way to find the exact call.
// Part 07 — Design

Keep Provider Code Behind One Function

Wrap provider calls in a tiny module of your own. The rest of your app calls generate() and never imports an SDK directly. Swapping providers, adding logging, or recording cost then happens in one place.

llm.py
python
import os
from dataclasses import dataclass
import anthropic

_client = anthropic.Anthropic(timeout=30.0, max_retries=2)

@dataclass
class LLMResult:
    text: str
    input_tokens: int
    output_tokens: int

def generate(system: str, messages: list[dict], max_tokens: int = 500) -> LLMResult:
    reply = _client.messages.create(
        model=os.environ["LLM_MODEL"], system=system, messages=messages, max_tokens=max_tokens,
    )
    return LLMResult(
        text=reply.content[0].text,
        input_tokens=reply.usage.input_tokens,
        output_tokens=reply.usage.output_tokens,
    )
  1. Read the model name and limits from configuration.
  2. Return your own result type, not the SDK object.
  3. Record token usage on every call — you will need it for cost tracking.
  4. Add tracing and evaluation hooks here later (lessons 7 and 9).
// Part 08 — Pitfalls

Common Mistakes

Hard-coding the API key in a script

Why: It is quick while experimenting, then gets committed.

Fix: Load keys from environment variables and keep .env in .gitignore from day one.

Not setting a max output limit

Why: Defaults can allow very long replies.

Fix: Always set max tokens for the use case; a classifier needs 10 tokens, not 4,000.

Retrying every error

Why: A loop that retries 400 errors burns money and never succeeds.

Fix: Retry only rate limits, timeouts, and 5xx errors, with backoff.

Calling the provider from the browser

Why: Front-end code exposes the key to every visitor.

Fix: Call LLM APIs only from your server, and let the browser talk to your server.

// Part 09 — Practice

Build a Command-Line Study Buddy

Practice

A streaming CLI tutor with history and safe retries

  1. Create llm.py with a generate() function like Part 7, reading the model from LLM_MODEL.
  2. Write a chat loop that keeps history and streams each answer to the terminal.
  3. Print input and output tokens after every turn, and a running total at the end.
  4. Keep only the last 6 messages in history and observe how input tokens stop growing.
  5. Turn off your network mid-conversation and confirm the app shows a friendly error instead of a stack trace.

Hint: Streaming and usage both exist in the SDKs: Anthropic exposes the final message (with usage) via stream.get_final_message().

🎯 Key Takeaways

  • ✓An LLM API call is an HTTPS request with a model, instructions, messages, and limits; the response includes text and token usage.
  • ✓The API is stateless — conversations work because your code resends history, so input cost grows every turn.
  • ✓Keep API keys in environment variables and call providers only from server-side code.
  • ✓Stream long answers so users see output immediately.
  • ✓Retry rate limits, timeouts, and server errors with backoff; never retry 4xx request errors.
  • ✓Hide provider SDKs behind one function so switching models, logging, and cost tracking happen in one place.
  • Official references: the OpenAI, Anthropic, and Google Gemini API documentation. Check them for current model names and prices before you ship.

Next

What comes next

You can call a model. Next, learn what a token is, how to estimate the monthly bill for a feature before you build it, and how to stay inside rate limits.

Lesson 02 · Tokens, Pricing and Rate Limits →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...