PROJECT · 3 hours
Ship a memory layer to a self-hosted Mac Mini environment — async embed-on-write, in-memory LRU caching, observability, and graceful degradation when the vector store is unavailable.
Before writing any new code, it helps to name exactly what can go wrong. A memory system that works in a notebook or single-turn test can still fail a production agent in three specific ways.
Generating an embedding is an API call — typically 100–400 ms depending on the model and provider. If you embed text synchronously before returning a response, every turn the agent takes pays that cost as visible delay. In a multi-turn conversation this adds up fast: the agent feels sluggish even when Claude responds quickly.
The key insight is that the embedding does not need to be ready before the current response. It needs to be ready before the next retrieval. Write the message to context now; schedule embed-and-store in the background.
Think of the vector store like a library catalog: fast for a small collection, noticeably slower when the shelves are full and you are searching across thousands of entries. Retrieval on a small store takes a few milliseconds. At 50,000 memories with a larger embedding model it can reach 800 ms. Across a ten-turn conversation that adds several seconds of wait time the user never asked for.
The answer is a cache in front of the store. Repeated questions — and in any real deployment, certain questions recur constantly — hit the cache at microsecond speed instead of querying the vector store at all.
A self-hosted vector store goes down: the disk fills, the database crashes, a routine OS update restarts the process. An agent that treats the vector store as a hard dependency fails completely when this happens. The production rule is: memory retrieval must degrade gracefully, never block the agent. Without memories the response is weaker; without availability the agent is broken for everyone.
These three concerns — write latency, retrieval latency, store downtime — map directly to three engineering decisions: async write, LRU cache, fallback path. The rest of this module implements each one.
Reflective prompt: Before reading on, think through your own agent's failure modes. What happens today if the vector store is unreachable for thirty seconds? What does the user experience? Knowing that answer shapes how aggressively you invest in the fallback path.
The async embed-on-write pattern is a coroutine you fire with asyncio.create_task. The agent does not await it — it continues generating the response while the embedding runs in the background.
This requires two things: your application already runs an async event loop (standard in any FastAPI or async Python service), and you use an embedding client that supports async. The Anthropic SDK does not expose a dedicated embedding endpoint, so this module uses Voyage AI's AsyncClient — Voyage AI is Anthropic's recommended embedding partner, and voyage-3 is their current general-purpose model.
Here is the core write pattern. Notice that store_memory_async is synchronous from the caller's perspective — it fires the coroutine and returns immediately:
The caller — your agent turn handler — calls store_memory_async with the user message and then returns immediately. If the embedding call fails, the error is logged and the agent is never blocked waiting for it.
Pass at minimum a conversation ID and a timestamp in metadata so memories are filterable by session and recency later. Beyond that, include whatever your retrieval queries will need to filter on.
asyncio.create_task only works inside a running event loop. For synchronous CLI tools or scripts, use a thread pool (asyncio.run_in_executor) or a background task queue such as ARQ instead.
An LRU (least-recently-used) cache evicts the oldest entries when it reaches capacity. For memory retrieval results, two additional parameters matter: capacity (how many query results to hold) and TTL (how long a result stays valid before expiring).
Python's collections.OrderedDict is the standard building block. move_to_end promotes the recently accessed entry; popitem(last=False) removes the least recently accessed one when the cache is full:
The cache key is the raw query string. If the agent asks the same question twice within the TTL window, the second call returns the cached result without touching the vector store or spending embedding tokens.
A capacity of 100 and a 5-minute TTL are starting values. Whether they are right depends on how many distinct queries your agent actually receives and how frequently new memories arrive that would make cached results stale. The next section gives you the data to answer that.
You cannot tune what you cannot measure. The three numbers worth tracking are retrieval latency, cache hit rate, and embedding token usage. None of these require a dashboard or external service — structured log lines to stdout are enough.
Wire the cache and metrics together into a single retrieval function. This is the final version that goes to production:
After a few hundred turns you will have real data: cache hit rate, median retrieval latency, and daily embedding token spend. A hit rate below 20% usually means queries are too varied to benefit from exact-match caching — consider lowercasing and stripping punctuation from the query before using it as a key.
Use a JSON log formatter so your aggregator can index each field independently. Log lines become a queryable time series rather than a wall of text to grep through.
There are two ways to ship the memory layer: as a library imported directly into the agent process, or as a standalone HTTP service the agent calls over localhost.
Library pattern — the cache, vector store client, and embedding client all live in the same process as the agent. Simpler to start with. Works well for a single agent on Mac Mini. Main downside: every restart of the agent flushes the in-memory cache.
Standalone service pattern — the memory layer runs as a separate FastAPI process. Advantages: the cache survives agent restarts; multiple agent instances can share one memory store; you can redeploy the agent without touching the memory service. Worth the extra complexity once you have more than one agent process or when cache persistence between restarts becomes important.
Regardless of pattern, expose the configuration that matters as environment variables so you can tune without code changes. The four that matter most are the embedding model name, cache capacity, cache TTL, and vector store URL.
Graceful degradation belongs in every deployment. Wrap your retrieval function in a fallback that returns an empty list when the vector store is unavailable:
An agent that works without memories is weaker. An agent that crashes on a database hiccup is broken for everyone until it is manually restarted.
1. Why should embedding-on-write happen asynchronously rather than synchronously?
Correct Answer: B
Embeddings need to be stored before the next retrieval, not before the current response. Running embed-and-store as a background task removes the latency cost from the agent's critical path.
2. What does `OrderedDict.move_to_end(key)` accomplish in the LRU cache on a cache hit?
Correct Answer: B
move_to_end promotes the recently accessed entry. When the cache is full, popitem(last=False) removes the entry at the front — the one that has gone longest without being accessed. Together they implement LRU eviction.
3. A cache hit rate of 15% tells you what about your caching strategy?
Correct Answer: B
A low hit rate usually means cache keys are too diverse to collide. Normalising queries — lowercase, strip punctuation, deduplicate whitespace — increases the probability that semantically identical questions reuse the same cached result.
4. What should happen when the vector store raises an exception during retrieval?
Correct Answer: C
Graceful degradation means the agent returns a weaker response rather than no response. Log the failure for visibility, return an empty list, and let the agent continue generating output.
5. Which production concern does an LRU cache directly address?
Correct Answer: C
The cache addresses retrieval latency: repeated queries are served in microseconds from memory rather than milliseconds from the vector store. It does not help with write-path latency (that is async) or store downtime (that requires a fallback path).
6. What is the main advantage of the standalone service deployment pattern over the library pattern?
Correct Answer: B
When the cache lives in a separate process, restarting the agent does not flush the cache. Multiple agent instances can also share the same memory service, which is not possible with a per-process library.
7. What is the first thing you would add to this memory system before going live?
Correct Answer: B
Before tuning capacity, TTL, or changing embedding models, you need visibility into what the system is actually doing. Structured logs for latency and cache hit rate are the minimum baseline — they cost nothing and answer the questions that come up in the first week of real traffic.
Nice work. Here's what's next.
Back to this pathOr book the diagnostic call