Memory Types & Architecture

The first thing I noticed: "memory" is not one thing. The support system needed to remember what happened in this conversation (short-term), what happened in past conversations with this customer (episodic), what the billing system actually does (semantic), and how to run a refund process (procedural). These are four distinct problems with four distinct storage strategies.

graph TD
    A[Agent Memory] --> B[Short-Term\nContext Window]
    A --> C[Episodic\nSession History]
    A --> D[Semantic\nFacts & Knowledge]
    A --> E[Procedural\nSkills & How-To]
    B --> F[In-memory, expires]
    C --> G[Persistent store, time-stamped]
    D --> H[Persistent store, indexed]
    E --> I[Config / tool definitions]

Here's the working taxonomy I landed on:

• Short-term — the context window itself. Fast, always available, gone when the session ends.
• Episodic — a log of past sessions: what was tried, what worked, what failed, when.
• Semantic — facts, product knowledge, domain concepts. What is true about the world.
• Procedural — how to do things. Tool schemas, runbook steps, escalation paths.

Each type maps to a different storage mechanism and retrieval pattern. Getting the taxonomy right before writing any code saved me from building one monolithic "memory table" that would have been impossible to query effectively.

Short-term memory is the context window — the messages the model sees right now. It is fast, zero-latency, and inherently stateless: when the API call ends, the context is gone.

The practical problem with short-term memory is capacity. A long support conversation can easily hit 50,000+ tokens across multiple tool calls. Most models have context limits, and even models with large windows get slower and more expensive as the context grows.

The two common patterns for managing context window size:

Sliding window — keep only the last N turns. Simple to implement, but the agent "forgets" earlier turns abruptly.

Summarization — use the model to compress older turns into a summary, then replace them. Preserves intent but loses detail. Costs a model call.

Episodic memory is a time-stamped log of past sessions and their outcomes. It answers the question: "What happened before, with this user, in this context?"

In the support-automation system, the absence of episodic memory caused a specific and infuriating failure: a customer would open a new session about a billing issue, and the agent would walk through the same diagnostic steps it had already run with them twice before. The customer knew this; the agent didn't. The session started from zero every time.

The fix was simple in concept: persist a record of each session — what was tried, what the outcome was, when it happened — and retrieve it at the start of any new session with the same customer.

Retrieving these at session start lets the agent open with: "I see we've looked at this billing issue twice before — both times the card processor returned a timeout. Let me check the processor status first this time."

Semantic memory holds facts that are true about the world — product documentation, billing system rules, escalation policies, known bugs. It is not specific to any session or customer; it's shared knowledge the agent needs to reason correctly.

The key distinction from episodic memory: semantic memory answers "what is true?" while episodic memory answers "what happened?"

In the support system, semantic memory included things like:

• How the billing processor handles retry logic
• What error codes map to which failure modes
• Which account types are eligible for automatic refunds

Storing semantic memory separately from session history means it can be updated once (when a policy changes) and immediately affects all future queries, without re-training anything.

The simplest viable semantic memory is a table of facts with a text description, queried with keyword search to start and upgraded to vector search when keyword search breaks down:

Each memory type has a natural fit. The mistake I made early on was reaching for episodic memory when I actually needed semantic memory, and vice versa. Here's the decision guide I use:

Start with short-term. The context window is free and instant. Only move information to persistent memory when you hit a real limit: the session ends and you need it again (→ episodic), the agent lacks domain knowledge it can't infer (→ semantic), or a workflow is too complex to keep in context (→ procedural).

Episodic vs. semantic confusion is the most common mistake. The rule: if the information is about a specific past event ("this customer's card was declined at 2pm on Tuesday"), it's episodic. If it's a general fact that would be true for any customer ("our billing processor retries failed charges after 24 hours"), it's semantic.

Don't over-engineer procedural memory. In most systems I've built, procedural memory is just tool schemas and runbook YAML, not a separate database. The agent reads the tool definition and knows how to use it. Only build a dynamic procedural memory store if the set of tools changes frequently at runtime.