Scaling Strategies for Solo AI Services

When I deployed the clinical-AI platform on a Mac Mini, I expected traffic to be my scaling problem. It wasn't.

The first time a few concurrent users hit the system at the same time, latency spiked from roughly 2 seconds to over 6. The server itself barely moved — CPU was idle. The problem was upstream: every user request fired a separate call to the Claude API, and the API's rate limit was already saturated.

A queue is a polite waiting room — better than everyone rushing the door at once. That one insight completely changed how I thought about scaling a solo AI service.

The key shift: Solo AI engineers don't need more servers. They need smarter usage of the single API they're calling.

The three levers that matter:

1. Cache — skip the API call entirely when you already have the answer.
2. Queue — control how many calls go out at once so you never hit the rate-limit wall.
3. Back-off — when you do get rate-limited, wait intelligently before retrying.

This module walks through each lever with working TypeScript patterns you can drop into a Next.js API route or a background service.

Anthropic enforces rate limits on two dimensions simultaneously:

• Requests per minute (RPM) — the number of API calls you can make per minute.
• Tokens per minute (TPM) — the total input + output tokens across all calls in that window.

When either limit is exceeded, the API returns HTTP 429 Too Many Requests along with a Retry-After header that tells you how long to wait.

What Happens in Practice

Imagine a support-automation service where every incoming ticket triggers a Claude API call. At low volume (one ticket per minute) everything looks fine. But when five tickets arrive in thirty seconds:

• Call 1 goes through immediately.
• Calls 2–4 stack up in milliseconds.
• Call 5 lands while the first four are still in-flight.
• The API returns 429 for calls 3 and 5.
• Without queuing, both fail and the user sees an error.

The service isn't overloaded — the Mac Mini is idle. But from the API's perspective, the concurrency contract was violated.

Reflective prompt: Think about your own AI service (current or planned). What is the worst-case burst of concurrent users you'd expect in a five-minute window? How does that compare to your tier's RPM limit?

The cheapest API call is the one you never make. For AI services, caching has unusually high value because:

• LLM calls are slow (hundreds of milliseconds to seconds).
• LLM calls are billed by tokens — every call has a real cost.
• Many real-world requests are semantically identical: the same user asking the same question with slightly different wording, or different users asking common questions.

A simple in-memory LRU (Least Recently Used) cache handles the most common case — exact or near-exact repeated prompts.

Design notes worth internalizing:

• LRU eviction — when the cache is full, the oldest entry is evicted. Map preserves insertion order, so keys().next().value is always the oldest key.
• TTL — one hour is a reasonable default for stable reference lookups. Shorten it for conversational context that may change frequently.
• Key normalization — collapsing extra whitespace means "what is RAG?" and "what is RAG?" share the same cache entry. Add further normalization (lowercase, punctuation stripping) if your prompts are user-typed.

Caching helps with repeated requests. But what about a burst of new requests — ones that all need a live API call?

Without concurrency control, all those calls fire simultaneously and the API rate-limits you. The fix is a semaphore: a simple counter that limits how many calls can be in-flight at the same time.

Think of a semaphore like a restaurant with a fixed number of tables. New customers wait at the door if every table is full — they don't all try to squeeze in at once and knock the tables over.

The maxConcurrent value depends on your tier's RPM limit and your target response time. A conservative default for a solo production service is 3–5 concurrent calls. If your average call takes 2 seconds and you allow 3 concurrent, you're capped at roughly 90 RPM — comfortably under most tier limits.

Even with a semaphore, you may occasionally hit a 429 — perhaps your service and another app share the same API key, or there's a burst at startup. The correct response is exponential back-off with jitter.

• Exponential — each retry waits twice as long as the previous one (1 s → 2 s → 4 s → 8 s).
• Jitter — randomize the wait slightly so multiple parallel callers don't all retry at the exact same moment (the "thundering herd" problem).

Wrap the actual API call inside withBackoff to get automatic retry behaviour:

Sometimes the right answer is to fail quickly rather than retry. If a user is waiting for an interactive response and the API is saturated, queuing them for 30 seconds is worse than returning a friendly message immediately.

When to fail fast:

• The user is waiting synchronously (chat, real-time suggestion, form submission feedback).
• The request deadline has already passed.
• You've exhausted the maximum number of retries.

When to queue and wait:

• The request is background or asynchronous (scheduled digest, webhook handler, batch job).
• The user has opted into a "notify me when ready" flow.
• The task is idempotent — it can safely run a few minutes later with the same result.

Request Flow: All Three Layers Together

Here is how cache, semaphore, and back-off fit together in a production API route:

flowchart TD
    A[Incoming Request] --> B{Cache hit?}
    B -- Yes --> C[Return cached response]
    B -- No --> D{Semaphore slot open?}
    D -- No --> E[Wait in queue]
    E --> F[Slot acquired]
    D -- Yes --> F
    F --> G[Call Claude API]
    G -- Success --> H[Store in cache]
    H --> I[Return response]
    G -- 429 --> J{Retries left?}
    J -- Yes --> K[Exponential back-off]
    K --> G
    J -- No --> L[Return fallback or error]

Summary: Three Levers, One Service

Scaling a solo AI service isn't about adding hardware. It's about respecting the constraints of the external API you depend on:

• Cache first. If the answer already exists in memory, return it — no API call needed.
• Queue second. A semaphore with a small maxConcurrent value prevents burst traffic from firing too many parallel calls.
• Back off gracefully. When a 429 does land, exponential back-off with jitter gives the API time to recover before you retry.
• Fail fast when appropriate. For interactive flows, a friendly error returned immediately is better than a 30-second queue wait.

These patterns cost nothing to implement and are often the difference between a stable service and one that is unexpectedly unavailable on a busy evening.