Haskell Retry With Exponential Backoff for Flaky Services
Your Haskell service calls a payment API, a Postgres replica, or an S3 bucket, and roughly one request in two hundred fails with a timeout that would have succeeded half a second later.

Your Haskell service calls a payment API, a Postgres replica, or an S3 bucket, and roughly one request in two hundred fails with a timeout that would have succeeded half a second later. You can either surface that failure to the user or retry it, and retrying badly is how a small blip becomes an outage. This guide covers Haskell retry with exponential backoff using the retry package, how to pick the numbers, which errors deserve a second attempt, and where retries quietly make things worse.
I have run this pattern in services handling a few thousand requests per second, and the code is the easy part. The judgment calls are where teams get burned.
Why a plain loop is the wrong Haskell retry
The first version most people write is a recursive function that catches an exception, sleeps for a fixed 500 milliseconds with threadDelay, and tries again up to three times. It works on a laptop. In production it has three problems.
- Fixed delays synchronise clients. If 400 workers all fail at the same moment because a downstream node restarted, they all retry at exactly the same moment 500 milliseconds later, and hit the recovering node with the same spike that knocked it over.
- Catching everything catches too much. A bare catch on SomeException also swallows async exceptions such as ThreadKilled or a timeout from System.Timeout, so your retry loop can keep a thread alive after its parent tried to cancel it.
- There is no total budget. Three retries at a growing delay can easily add 7 seconds to a request whose caller gave up after 2.
Exponential backoff with jitter fixes the first problem, a typed handler fixes the second, and a delay cap plus a deadline fixes the third.
Setting up exponential backoff with the retry package
The retry package on Hackage, under the Control.Retry module, gives you composable policies. A policy is a value of type RetryPolicyM m, and policies combine with the Semigroup operator, which takes the stricter of each decision. A sensible starting point for an HTTP call looks like this in words:
- Start from fullJitterBackoff 100000, which means a base of 100 milliseconds (the package counts in microseconds) doubled on every attempt, with the actual delay drawn randomly between zero and that ceiling.
- Combine it with limitRetries 5 so the action runs at most six times in total.
- Wrap the whole thing in capDelay 2000000 so no single wait exceeds 2 seconds.
- Run your action with recovering, passing that policy and a list of handlers that each return True only for errors worth retrying.
With those numbers the worst case ceilings are 100, 200, 400, 800, 1600 and then 2000 milliseconds after the cap, which sums to about 5.1 seconds of waiting. With full jitter the expected total is roughly half that. Write the sum down for every policy you ship. If it exceeds the timeout of whoever calls you, the policy is lying.
The recovering function also passes a RetryStatus to your action, which carries the iteration number and cumulative delay. Log the iteration on each failure. When you later read an incident timeline, "attempt 4 of 6 failed after 1.4 s" tells you far more than a stack of identical timeout lines.
Choosing between full jitter and plain exponential
The package also offers exponentialBackoff without randomness. Use it only when a single client talks to a dependency, such as a CLI tool hitting one API. Any time you have many replicas or many green threads retrying the same target, full jitter spreads the load. In a load test I ran against a deliberately flaky mock, 300 concurrent clients using plain exponential backoff produced visible retry waves every doubling interval, while full jitter flattened them into steady background noise.
Deciding which errors to retry
This is the part that actually matters. A retry is only correct when the failure is transient and the operation is safe to repeat. Split your errors into three buckets before you write a single handler.
| Error | Retry? | Reason |
|---|---|---|
| Connection refused, connection reset | Yes | Usually a restart or a load balancer shuffle |
| HTTP 502, 503, 504 | Yes | Gateway or overload, often gone in seconds |
| HTTP 429 | Yes, honour Retry-After | The server is telling you exactly how long to wait |
| HTTP 400, 401, 404, 422 | No | Sending the same bad request again changes nothing |
| Postgres serialization failure (40001) | Yes, whole transaction | Expected under SERIALIZABLE isolation |
| Postgres unique violation (23505) | No | A logic or data problem, not a blip |
In Haskell this maps cleanly to handlers. For http-client, match on HttpExceptionRequest and inspect the content: ConnectionFailure, ResponseTimeout and ConnectionTimeout are retryable, a StatusCodeException needs a status check. If you model failures as values instead of exceptions, which I prefer for business errors as covered in Haskell Either for error handling without exceptions, use the retrying function instead, which decides based on the returned value rather than on a thrown exception.
Never retry async exceptions. The recovering function skips them by default in current versions, but if you hand roll a loop, rethrow anything that is an instance of SomeAsyncException. Otherwise a cancelled request can sleep and retry for seconds after its thread was told to die.
Idempotency is not optional
Retrying a GET is safe. Retrying a POST that charges a card is not, unless the server supports an idempotency key. Generate a UUID once, before the retry block, and send it on every attempt. If you generate it inside the action, each retry looks like a new payment. I have seen exactly this bug double charge 11 customers during a 40 second network partition.
Where common retry advice goes wrong
The standard advice is "retry transient errors at every layer so the system is resilient". I disagree, and the reason is arithmetic. If your handler retries 3 times, the service it calls retries 3 times, and that service's database client retries 3 times, one user request can become 4 times 4 times 4, or 64 calls against the database during an incident. Retries multiply. They do not add.
The better rule is to retry at exactly one layer, usually the one closest to the failing dependency, and make every layer above it fail fast. Pair that with a cap on how many in-flight retries a process allows. A simple approach is an STM counter, the same kind of shared state described in Haskell STM for shared state in concurrent services: increment before a retry, decrement after, and skip the retry if more than, say, 10 percent of recent requests are already retries. That single guard turns a retry storm into a clean, visible error spike that your alerts can catch.
Retries also hide problems. If your dashboards only show final success, a dependency failing 30 percent of first attempts looks healthy. Emit a counter for every retry, labelled by target and error kind.
Testing retry behaviour before production
Retry code is rarely exercised until the bad day, so test it on purpose.
- Make the policy a parameter of your client, so tests can pass limitRetries 3 with a zero base delay and run in milliseconds.
- Use an IORef counter in a fake action that fails twice and then succeeds, and assert it ran exactly three times.
- Add a test where the fake throws a 400 equivalent and assert it ran exactly once.
- Check the policy arithmetic with the package's simulatePolicy, which returns the delays a policy would produce without sleeping.
For the wider system, inject faults in staging: drop 5 percent of connections to one dependency and watch total request latency at the 99th percentile. If it climbs past your timeout budget, shrink the cap or the retry count. The failure modes that only appear under real concurrency are covered in why distributed systems fail in ways unit tests never catch, and retry storms are one of the classic examples.
Frequently asked questions
How many times should a Haskell service retry a failed request?
For interactive requests, two to four retries is usually enough, with a total wait under your caller's timeout. Background jobs can afford more, sometimes 8 to 10 with a cap of 30 seconds or more. The right number comes from your latency budget, not from a round figure.
What is jitter in exponential backoff?
Jitter adds randomness to each delay so that many clients failing at once do not retry at the same instant. Full jitter picks a random delay between zero and the exponential ceiling. It slightly increases average wait for one client but sharply reduces load spikes on the recovering service.
Should I retry database transactions in Haskell?
Retry serialization failures and deadlocks, and always retry the entire transaction from the start rather than the single statement that failed. Do not retry constraint violations, since they reflect data problems. Keep the retry outside the transaction block so each attempt opens a fresh one.
More in Systems
Systems
Haskell Bounded Concurrency for Background Workers
A worker that drains a job queue is the first Haskell service most teams ship, and it is also the first one that gets killed by...
Systems
Haskell Health Check Endpoints for Kubernetes Probes
A student of mine shipped her first Haskell service to a small Kubernetes cluster last spring, and it restarted itself 41 times...
Systems
Haskell Structured Logging for Production Services
Your Haskell service crashed at 03:12, and the only clue is a wall of putStrLn output with no request id, no timestamp...