Building Memory Management

Keyword search operates on exact token overlap. If the query contains "billing" and a document contains "billing", they match. If the semantically relevant document uses "payment failure" and "account access revoked" instead, it doesn't match — even though it's exactly what the user is looking for.

Embeddings solve this by converting text into a vector of numbers — a point in a high-dimensional space where meaning determines proximity. Documents about billing problems and documents about payment failures land near each other in that space, even if they share no words.

Think of it like a city map. Two addresses can be completely different strings — "123 Main St" and "the coffee shop on the corner of Main and Oak" — yet they might be 50 feet apart. Embeddings create a map of meaning where closeness reflects conceptual similarity, not lexical overlap.

The practical implication for agent memory is significant. When a user asks "what happened with my payment last week?", a keyword retrieval system searches for "payment" and "last week". A semantic retrieval system encodes the intent — account billing, recent history, financial issue — and finds the nearest memories in meaning-space regardless of exact wording.

Here is what the transition looked like on the query that exposed the problem:

The difference was not marginal. Switching to embedding-based retrieval was the change that made the support agent actually useful in production.

An embedding model takes a piece of text and returns a fixed-length list of floating-point numbers. For the Voyage AI voyage-3 model — the embedding model I use in the support-automation system — that list is 1024 numbers long. Similar texts produce vectors that point in similar directions in that space; dissimilar texts point in different directions.

The Python package is voyageai, installed with pip install voyageai. It reads VOYAGE_API_KEY from the environment, following the same pattern as the Anthropic SDK.

The vector itself is not human-interpretable. You cannot read the 47th dimension and learn what concept it encodes. What matters is that two vectors with high cosine similarity came from texts with similar meaning. Cosine similarity ranges from -1 (opposite meaning) to 1 (identical meaning). In practice, scores above 0.85 indicate strong semantic alignment; scores below 0.6 are usually noise.

You can embed multiple texts in a single batch call — the API accepts a list. This matters for throughput: loading 100 historical tickets is far more efficient as one batch than 100 individual calls.

Checkpoint: Before continuing, consider this — if you embed "account was locked" and "user cannot authenticate" with the same voyage-3 model, would you expect their cosine similarity to be closer to 0.9 or 0.2? The answer shapes how you think about retrieval quality for the rest of this module.

The answer is closer to 0.9. Both phrases describe the same underlying concept — an authentication failure — even though they use different vocabulary. This is exactly the property that makes embeddings useful for agent memory: the model has already learned that "locked" and "cannot authenticate" belong to the same region of meaning-space. You inherit that knowledge by using the embedding model; you do not need to teach it yourself.

Storing 1024-dimensional vectors in a plain database column and computing cosine similarity with a full table scan works at small scale. At tens of thousands of memories, it becomes too slow for real-time retrieval.

A vector store is a database — or database extension — designed for this exact problem. It indexes vectors so that "find the 5 nearest neighbors to this query vector" runs in milliseconds instead of seconds. I use pgvector, a PostgreSQL extension, because the support-automation system already uses PostgreSQL for its relational data. Adding vector search to an existing Postgres instance requires one CREATE EXTENSION and one CREATE INDEX.

Here is the schema for the memories table:

The VECTOR(1024) column type is provided by the pgvector extension. The metadata JSONB column holds ticket IDs, priority levels, source identifiers, or anything else that should travel with the memory but is not part of the searchable content.

The index is what makes retrieval fast:

ivfflat is an approximate nearest-neighbor index. vector_cosine_ops tells it to optimize for cosine distance. lists = 100 is a reasonable default for up to a few hundred thousand rows; tune it closer to sqrt(row_count) as the table grows.

With the schema in place, semantic search reduces to a SQL query. The key is pgvector's <=> operator, which computes cosine distance between two vectors — and it uses the ivfflat index, so it does not scan every row.

The expression 1 - distance converts cosine distance (lower = closer) into cosine similarity (higher = more similar). The ORDER BY clause uses the ivfflat index. You pass the embedded query vector as a parameter from Python — the full implementation is in the MemoryManager.retrieve() method in the next section.

To make the before-and-after concrete, here is the same support query through both paths:

The keyword path searched for documents containing "billing" and "login" together. It found an FAQ page and a department contact page, then fell back to a generic response. The relevant ticket — filed under authentication errors but describing a payment failure — scored zero matches.

The embedding path converted the query to a voyage-3 vector and ranked by cosine similarity. The top three results came back as: "payment failure caused silent account lockout" (0.91), "subscription lapse revokes API access" (0.87), and "charge declined, user sees auth error" (0.84). The agent found the root-cause pattern without any word overlap between query and results.

Reflective prompt: In your own agent, what text would you embed as the memory's content — the raw user message, a structured summary, or an LLM-generated distillation? Each choice changes what the retrieval surface looks like. What are the trade-offs?

The retrieve function above works as a utility, but a production agent loop needs a single object that manages the database connection, owns the embedding model reference, and exposes a clean store / retrieve interface. The MemoryManager class does that in three segments.

The three segments below build up the class progressively. Read each segment for what it introduces before moving to the next — the interface shapes the store method, and the store method shapes the retrieve method.

Segment 1 — interface and initialization:

The constructor falls back to DATABASE_URL from the environment if no explicit URL is provided. Recording self.model on the instance means every store and retrieve call uses the same model — a mismatch here would corrupt similarity scores silently.

Segment 2 — storing a memory:

The method embeds the text, inserts the vector alongside the raw content and caller-supplied metadata, and returns the new row ID. RETURNING id avoids a second round-trip to the database.

Segment 3 — retrieving by semantic similarity:

With both methods in place, the agent loop becomes straightforward. Store a ticket when it arrives; retrieve the nearest memories when a user submits a query: