Haskell Rate Limiting for APIs With a Token Bucket
One client with a retry loop can push your API from 200 requests per second to 4,000 in under a minute, and every other caller pays for it with timeouts.

One client with a retry loop can push your API from 200 requests per second to 4,000 in under a minute, and every other caller pays for it with timeouts. Haskell rate limiting is the fix most teams reach for after that first incident, but the version they write in a hurry usually leaks memory, locks too much, or limits the wrong thing.
This guide walks through a token bucket built on STM, how to key it per client, how to wire it into a WAI middleware, and where the common advice gets it wrong.
What Haskell rate limiting actually needs to do
A rate limiter answers one question per request: does this caller have budget left right now? The answer must come back in microseconds, stay correct under heavy concurrency, and not grow without bound as new client keys appear.
Three algorithms show up in most discussions:
- Fixed window. Count requests per key per minute and reset at the boundary. Simple, but a client can send 100 requests at 12:00:59 and another 100 at 12:01:00, doubling the intended rate.
- Sliding log. Store a timestamp for every request and count the ones inside the window. Accurate, but memory grows with traffic, which is exactly when you can least afford it.
- Token bucket. Each key holds a number of tokens that refills at a steady rate up to a cap. A request spends one token. Bursts are allowed up to the cap, and the long term rate is fixed by the refill speed.
The token bucket wins for most APIs because it needs only two numbers per key: the current token count and the time of the last refill.
Building a token bucket with STM
Represent each bucket as a TVar holding a small record with two strict fields: tokens as a Double and lastRefill as a monotonic timestamp in nanoseconds. Read the clock with getMonotonicTimeNSec from GHC.Clock outside the transaction, then pass it in. Never call the clock inside STM, where a retried transaction would see a stale time.
The take operation does four things inside one atomically block:
- Read the bucket.
- Compute elapsed seconds since lastRefill and add elapsed times refillRate tokens, capped at capacity.
- If the result is at least 1, subtract 1, write the new state, and return Allowed.
- Otherwise write the refilled state and return Denied along with the seconds until the next token, which becomes your Retry-After header.
Make both record fields strict with bang patterns, and write fully evaluated values. A lazy Double in a TVar written 500 times a second builds a thunk chain that shows up as a slow climb in resident memory.
For a typical public API, a capacity of 20 with a refill rate of 5 per second is a reasonable start: bursts of 20, then a steady 5 per second. Tune from real traffic.
Why STM instead of an MVar
An MVar per bucket works, but a slow handler holding the lock blocks every other request for that key. STM transactions on separate TVars never contend, and transactions on the same TVar retry cheaply because the work inside is tiny. The deeper trade-offs are covered in Haskell STM for shared state in concurrent services.
Keying buckets per client without leaking memory
The map from client key to bucket is where most homemade limiters fail. A single TVar holding a Data.Map of every bucket turns every request into a write on one shared variable, so all transactions conflict with each other.
Two better options:
- StmContainers.Map from the stm-containers package. It is a concurrent hash trie where operations on different keys touch different TVars.
- Sharding. Create 64 independent maps, each in its own TVar, and pick a shard by hashing the key. This uses only base and containers and cuts contention by roughly the shard count.
Whatever structure you choose, you need eviction. A bot that cycles through 100,000 IP addresses will otherwise leave 100,000 buckets in memory forever. Run a background thread every 60 seconds that removes buckets whose lastRefill is older than the time needed to refill completely. A full bucket is indistinguishable from a missing one, so deleting it changes no behavior.
Wiring the limiter into WAI middleware
A WAI Middleware is a function from Application to Application, which makes the limiter easy to slot in. Extract a key from the request, call take, and either pass the request through or respond immediately with status 429 and a Retry-After header rounded up to whole seconds.
Choosing the key matters more than the algorithm:
| Key | Good for | Watch out for |
|---|---|---|
| API token or user id | Authenticated APIs | Unauthenticated endpoints have no token |
| Client IP | Login and signup forms | Office NAT puts 300 people behind one address |
| Token plus route | Expensive endpoints like exports | More buckets per client |
If the service sits behind a load balancer, remoteHost will show the balancer's address for every request. Read the forwarded header your proxy sets, and trust only the last hop it appends, or a client can spoof its way into a fresh bucket on every call.
Place the limiter before body parsing and before you check out a database connection. Rejecting a request should cost almost nothing. If your limiter sits after the pool checkout, a flood still drains the pool, which is the failure described in our guide to Haskell connection pooling for Postgres services.
The common advice that does not hold up
You will often read that an in-process limiter is pointless once you run more than one replica, and that you should move straight to Redis. That advice skips the cost. A Redis round trip adds 0.3 to 1 millisecond to every request and introduces a new dependency that can fail.
For many services, a local limiter per replica is good enough. With 4 replicas and a per-replica limit of 5 per second, a client's real ceiling is somewhere between 5 and 20 per second depending on how the balancer spreads traffic. If the goal is protecting your own service from overload, that range is fine. Set the local limit to the global target divided by the replica count and accept the imprecision.
Reach for a shared store only when the limit is a contract, such as a paid plan that promises exactly 1,000 calls per hour, or when the thing you are protecting is a third party that bills you per call.
Testing that the limiter holds under load
Unit tests catch arithmetic mistakes in the refill math. Write a pure version of the refill function that takes the old state, the current time, and the config, and test it with QuickCheck: tokens never exceed capacity, never go negative, and two refills over intervals a and b equal one refill over a plus b.
Load tests catch everything else. Run a tool such as wrk or oha with 200 connections against one key and confirm that the accepted rate settles at your refill rate within a few percent. Then run 10,000 distinct keys and confirm resident memory plateaus once eviction kicks in. Concurrency bugs in limiters tend to appear only at scale, a pattern explored in why distributed systems fail in ways unit tests never catch.
Frequently asked questions
Is there a Haskell library for rate limiting?
Yes, several small packages on Hackage implement token buckets and WAI middleware. They are fine for a quick start, but check whether they evict idle keys and how they handle the forwarded IP header.
What status code should a rate limited request return?
Return 429 Too Many Requests with a Retry-After header in whole seconds. Avoid 503, because monitoring will treat it as an outage on your side.
How do I rate limit WebSocket messages in Haskell?
Use the same bucket, but call take once per incoming message instead of once per connection. When a client runs out of tokens, either drop the message and send a warning frame or close the connection with a policy violation code after repeated denials.
More in Systems
Systems
Haskell Bounded Concurrency for Background Workers
A worker that drains a job queue is the first Haskell service most teams ship, and it is also the first one that gets killed by...
Systems
Haskell Health Check Endpoints for Kubernetes Probes
A student of mine shipped her first Haskell service to a small Kubernetes cluster last spring, and it restarted itself 41 times...
Systems
Haskell Structured Logging for Production Services
Your Haskell service crashed at 03:12, and the only clue is a wall of putStrLn output with no request id, no timestamp...