Agent State and Memory

Model agent state explicitly, manage short-term context, add long-term memory with retrieval, and avoid memory that leaks or drifts.

Est. 30 min Updated Suggest a fix
// Part 01 — Why state matters

A Run Needs More Than a Prompt

In the previous lesson, a plain-Python loop kept an observations list while one request was running. That list was state: the information needed to decide what to do next. Real systems need a richer model of the current run, and often need to resume it after a timeout, a person's approval, or a server restart. If state is implicit in a long prompt or scattered through local variables, it is hard to test, save, redact, and recover.

Separate three related ideas. State is the structured snapshot for one run. Short-term memory is context from the current conversation or task, such as recent messages and tool results.Long-term memory is selected information retained across runs and retrieved when relevant. They solve different problems and should not be stored or governed as one undifferentiated transcript.

ConceptLifetimeExample
Run stateOne execution; may be checkpointedGoal, current step, observations, pending approval
Short-term contextCurrent conversation or taskRecent user and assistant messages, tool result summaries
Long-term memoryAcross conversations, under a retention policyA user-approved preference or a retrieved support note
💡 Note
An LLM API does not remember a previous call by itself. The application decides which state and history to send with each request, so memory design is also context, privacy, and cost design.
// Part 02 — State schema

Make the Run Explicit and Typed

Give the run a small schema. Store the user goal, message history, observations, a step count, and a status. Avoid stuffing every field into an opaque dictionary passed around without validation. A Python dataclass gives an approachable starting point; a typed state schema is also useful when you later map the workflow into LangGraph.

agent_state.py
python
from dataclasses import dataclass, field
from typing import Any, Literal

Role = Literal["user", "assistant", "tool"]

@dataclass
class Message:
    role: Role
    content: str

@dataclass
class AgentState:
    run_id: str
    user_id: str
    goal: str
    messages: list[Message] = field(default_factory=list)
    observations: list[dict[str, Any]] = field(default_factory=list)
    step_count: int = 0
    status: Literal["running", "waiting", "complete", "failed"] = "running"

Notice that identity is not inferred from the conversation text: user_id comes from authenticated application context. The model can help interpret goal; it cannot change the user identity or grant itself permission. Status values are explicit so callers can distinguish an answer from a waiting approval or a failed run.

🎯 Pro Tip
Store references and compact structured results when possible. A full tool response may contain sensitive fields or be far larger than the next decision needs.
// Part 03 — Short-term context

Keep Recent Messages Relevant

Short-term memory commonly contains the current user request, the recent assistant turns, and concise tool observations. It gives the model local continuity while it works. But resending an entire conversation on every call increases context size and can bury the details needed for the current action. Choose a policy: retain recent turns, preserve essential facts separately, and summarise older context with a way to correct mistakes.

context_window.py
python
def messages_for_model(history: list[dict[str, str]], summary: str, keep_recent: int = 6):
    recent = history[-keep_recent:]
    messages = []
    if summary:
        messages.append({"role": "system", "content": "Earlier context summary: " + summary})
    messages.extend(recent)
    return messages


history = [
    {"role": "user", "content": "My delivery is late."},
    {"role": "assistant", "content": "I can check the order."},
    {"role": "user", "content": "It is A104."},
]
print(messages_for_model(history, summary="Customer asks about delayed delivery."))

This helper illustrates selection, not a complete summariser: a summary should be generated or updated deliberately, and important identifiers should be validated against authoritative records. Do not rely on a model-generated summary as the only copy of a payment reference, safety instruction, or approval decision. Keep source records outside model context and retrieve them when needed.

Context itemKeep in prompt?Better source of truth
Recent conversationUsually, within a chosen limitConversation store with access controls
Order statusA concise current result may helpOrder service or database
User preferenceOnly when relevant and permittedUser-editable profile or consented memory store
Approval to refundDo not infer from a summaryDurable approval record with actor and timestamp
// Part 04 — Long-term memory

Retrieve, Do Not Paste Everything

Long-term memory lets an application bring forward selected facts across separate tasks. A vector index can retrieve semantically similar notes: convert a query and notes into embeddings, rank by similarity, and provide only a small, relevant set to the agent. A vector match is a retrieval hint, not proof that the note is correct, current, or authorised for this user. Filter by tenant and user before ranking, and retain an ordinary source record for facts that must be auditable.

similarity_search.py
python
from math import sqrt


def cosine(a: list[float], b: list[float]) -> float:
    if len(a) != len(b) or not a:
        raise ValueError("Vectors must have the same non-zero length")
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = sqrt(sum(x * x for x in a))
    norm_b = sqrt(sum(y * y for y in b))
    if norm_a == 0 or norm_b == 0:
        return 0.0
    return dot / (norm_a * norm_b)


def retrieve(query_vector: list[float], notes: list[dict], user_id: str, limit: int = 3):
    owned = [note for note in notes if note["user_id"] == user_id]
    ranked = sorted(owned, key=lambda note: cosine(query_vector, note["vector"]), reverse=True)
    return ranked[:limit]


notes = [
    {"user_id": "u1", "text": "Prefers concise answers", "vector": [0.8, 0.2]},
    {"user_id": "u2", "text": "Uses a screen reader", "vector": [0.7, 0.3]},
]
print([note["text"] for note in retrieve([0.9, 0.1], notes, "u1")])

The tiny example uses hand-written vectors so the ranking mechanics run without an embedding service or vector database. In practice, the embedding model and vector storage are separate components. Apply tenant filters in the retrieval query itself, encrypt and restrict stored data, and include a deletion path for each memory record.

⚠️ Important
Similarity is not authorization. Never query a shared index and hope that a prompt tells the model which customer results to ignore; apply ownership filters before any result is returned to the run.
// Part 05 — Memory lifecycle

Decide What May Be Remembered

Persisting everything is rarely a good default. Decide what can be written, why it is useful, who can see it, how long it lasts, and how a person can correct or delete it. A preference such as "show distances in kilometres" might be useful across sessions; a one-time transaction reference probably belongs in a protected transaction system rather than conversational memory. Avoid inferring sensitive traits from conversation.

  1. Select a candidate fact and its source; do not silently promote every chat message into memory.
  2. Check relevance, user consent, sensitivity, and whether an authoritative source already exists.
  3. Write a scoped record with provenance and a retention or expiry policy.
  4. Retrieve only when the current task needs it, then let the user inspect, correct, or delete it.

A state object can also contain a pointer to a pending approval or a checkpoint identifier. Keep those operational records distinct from semantic memory: workflow state must survive reliably, while a vector memory is an optional candidate context. A retrieval miss should not prevent the agent from asking the user or using the authoritative service.

// Part 06 — Summaries and drift

Summarise With a Provenance Trail

Summaries are lossy. They can omit a negation, blend two people's details, or preserve an assumption as if it were a verified fact. Keep the original source or a stable reference where appropriate, mark summaries as summaries, and refresh them when facts change. Use structured state for details that drive business rules; a prose summary should help the model orient itself, not replace the database.

memory_record.py
python
from dataclasses import dataclass
from datetime import datetime

@dataclass
class MemoryRecord:
    owner_id: str
    text: str
    source_id: str
    created_at: datetime
    expires_at: datetime | None
    approved_by_user: bool


def may_retrieve(record: MemoryRecord, owner_id: str, now: datetime) -> bool:
    if record.owner_id != owner_id or not record.approved_by_user:
        return False
    return record.expires_at is None or now < record.expires_at

This policy check is illustrative and should be paired with storage-level access control and deletion handling. Production records often need more metadata: purpose, sensitivity classification, source, and a revocation state. Keep the schema as small as the product can responsibly support, and make the controls executable rather than relying on a prompt that says "respect privacy."

📌 Real World Example
In an Indian learning app, a learner's preferred explanation language could be a useful saved preference if the learner chooses it. Do not retain a payment UPI ID in an agent memory just because it appeared in a conversation.
// Part 07 — Production

Test State Transitions and Recovery

State changes deserve tests just like functions. Check that a tool observation is appended once, a retry does not duplicate a side effect, an expired memory is not retrieved, and one user cannot access another user's notes. Also test what happens when a worker restarts halfway through the run. A saved state should have a versioned shape or a migration plan as the application evolves.

  1. Create fixtures for an empty state, a long conversation, a tool failure, and a pending human approval.
  2. Assert which messages and observations are included in each model request.
  3. Test user and tenant isolation at the storage query boundary.
  4. Measure prompt size and retrieval relevance; inspect false matches, not only successful ones.

Keep state transitions observable: record a run ID, transition name, timestamp, and result category. Redact personal data from logs, control who may inspect traces, and define a retention schedule. This lets a team diagnose drift without turning all customer conversations into permanent debugging data.

// Part 08 — Pitfalls

Common Mistakes

Using one growing transcript as every kind of memory

Why: Run state, conversation context, and durable facts have different lifetimes and access rules.

Fix: Separate typed run state, bounded short-term context, and intentionally retained memory.

Sending the full history on every model call

Why: Old turns consume context and can distract from the current task.

Fix: Select recent messages, preserve essential structured facts, and summarise older context carefully.

Treating nearest vector results as verified facts

Why: Semantic similarity can return stale, ambiguous, or unrelated notes.

Fix: Filter by ownership, inspect provenance, validate authoritative facts, and allow a retrieval miss.

Saving everything by default

Why: Unnecessary retention increases privacy risk and can make later answers worse.

Fix: Define purpose, consent, retention, correction, and deletion before enabling persistent memory.

// Part 09 — Practice

Add Safe Memory to the Support Agent

Practice

A stateful but scoped assistant

  1. Create an AgentState instance for a delivery-support run and update its status through running and complete.
  2. Trim a conversation to a chosen number of recent turns while retaining a separate short summary.
  3. Use the cosine helper to retrieve one relevant note, then verify a different user cannot retrieve it.
  4. Add expiry and approval fields to memory records and write tests for expired and unapproved records.
  5. Document one fact that should live in an order system rather than agent memory, and explain why.

Hint: Use fake vectors and sample notes for the exercise. The goal is to test state boundaries and filtering, not to choose an embedding provider.

🎯 Key Takeaways

  • ✓Run state is a structured snapshot; short-term context is task history; long-term memory is selected information across runs.
  • ✓Keep identity and authorization from trusted application context, not model-generated conversation text.
  • ✓Bound short-term history and summarise carefully; summaries can lose facts and do not replace source records.
  • ✓Retrieve long-term memory only when relevant, with ownership filters applied before results enter model context.
  • ✓Set purpose, consent, provenance, retention, correction, and deletion rules for persistent memory.
  • ✓Test state transitions, recovery, isolation, expiry, and retrieval quality as part of the agent, not as an afterthought.

Next

What comes next

With state and context made explicit, explore how to plan multi-step work, reflect on evidence, and recover without unbounded retries.

Lesson 03 · Planning, Reflection and Error Recovery →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...