architecture
Token Bucket Rate Limiting
A rate-limiting algorithm that allows bursts up to a capacity cap while enforcing an average throughput ceiling — the pattern used in production APIs and my workflow-engine-rs implementation.
How Token Bucket Works
A token bucket holds up to capacity tokens. Tokens refill at rate per second. Each request consumes one token. If the bucket is empty the request is rejected (HTTP 429) or queued.
This allows short bursts — a client can use up to capacity tokens at once — while enforcing the average rate over time.
Why It Beats a Fixed Window
A fixed-window counter resets at clock boundaries, which allows a burst of 2 × limit requests straddling a reset. Token bucket smooths that by refilling continuously.
Pseudocode
Grounded in workflow-engine-rs
In my workflow-engine-rs project, each typed node in the execution graph carries a token-bucket limiter that caps how fast it can be invoked by upstream nodes. This prevents a fast producer node from overwhelming a slow consumer node during concurrent graph execution — the same backpressure problem that appears in API gateway rate limiting, just applied intra-process.
When Not to Use It, and What Breaks
Token bucket allows bursts — that is its strength and its risk. If a downstream dependency cannot absorb a sudden burst up to capacity, a smoother algorithm like leaky bucket (which drains at a fixed rate, no bursts) is the safer choice.
The in-memory bucket above is also per-process. Across multiple replicas each holds its own bucket, so the effective limit becomes capacity × replica_count — usually not what you intended. A distributed deployment needs a shared store (Redis, for example) to keep the bucket consistent, trading a network round-trip for correctness. Reach for the simple in-memory version only on a single instance, or where approximate limits are acceptable.