Haskell World logo Haskell WorldWrite code, build worlds
Systems

Haskell Retry With Exponential Backoff for Flaky Services

Your Haskell service calls a payment API, a Postgres replica, or an S3 bucket, and roughly one request in two hundred fails with a timeout that would have succeeded half a second later.

Server rack in a dim data centre with blinking status lights and a laptop showing a terminal log on a cart

Your Haskell service calls a payment API, a Postgres replica, or an S3 bucket, and roughly one request in two hundred fails with a timeout that would have succeeded half a second later. You can either surface that failure to the user or retry it, and retrying badly is how a small blip becomes an outage. This guide covers Haskell retry with exponential backoff using the retry package, how to pick the numbers, which errors deserve a second attempt, and where retries quietly make things worse.

I have run this pattern in services handling a few thousand requests per second, and the code is the easy part. The judgment calls are where teams get burned.

Why a plain loop is the wrong Haskell retry

The first version most people write is a recursive function that catches an exception, sleeps for a fixed 500 milliseconds with threadDelay, and tries again up to three times. It works on a laptop. In production it has three problems.

  • Fixed delays synchronise clients. If 400 workers all fail at the same moment because a downstream node restarted, they all retry at exactly the same moment 500 milliseconds later, and hit the recovering node with the same spike that knocked it over.
  • Catching everything catches too much. A bare catch on SomeException also swallows async exceptions such as ThreadKilled or a timeout from System.Timeout, so your retry loop can keep a thread alive after its parent tried to cancel it.
  • There is no total budget. Three retries at a growing delay can easily add 7 seconds to a request whose caller gave up after 2.

Exponential backoff with jitter fixes the first problem, a typed handler fixes the second, and a delay cap plus a deadline fixes the third.

Setting up exponential backoff with the retry package

The retry package on Hackage, under the Control.Retry module, gives you composable policies. A policy is a value of type RetryPolicyM m, and policies combine with the Semigroup operator, which takes the stricter of each decision. A sensible starting point for an HTTP call looks like this in words:

  1. Start from fullJitterBackoff 100000, which means a base of 100 milliseconds (the package counts in microseconds) doubled on every attempt, with the actual delay drawn randomly between zero and that ceiling.
  2. Combine it with limitRetries 5 so the action runs at most six times in total.
  3. Wrap the whole thing in capDelay 2000000 so no single wait exceeds 2 seconds.
  4. Run your action with recovering, passing that policy and a list of handlers that each return True only for errors worth retrying.

With those numbers the worst case ceilings are 100, 200, 400, 800, 1600 and then 2000 milliseconds after the cap, which sums to about 5.1 seconds of waiting. With full jitter the expected total is roughly half that. Write the sum down for every policy you ship. If it exceeds the timeout of whoever calls you, the policy is lying.

The recovering function also passes a RetryStatus to your action, which carries the iteration number and cumulative delay. Log the iteration on each failure. When you later read an incident timeline, "attempt 4 of 6 failed after 1.4 s" tells you far more than a stack of identical timeout lines.

Choosing between full jitter and plain exponential

The package also offers exponentialBackoff without randomness. Use it only when a single client talks to a dependency, such as a CLI tool hitting one API. Any time you have many replicas or many green threads retrying the same target, full jitter spreads the load. In a load test I ran against a deliberately flaky mock, 300 concurrent clients using plain exponential backoff produced visible retry waves every doubling interval, while full jitter flattened them into steady background noise.

Deciding which errors to retry

This is the part that actually matters. A retry is only correct when the failure is transient and the operation is safe to repeat. Split your errors into three buckets before you write a single handler.

ErrorRetry?Reason
Connection refused, connection resetYesUsually a restart or a load balancer shuffle
HTTP 502, 503, 504YesGateway or overload, often gone in seconds
HTTP 429Yes, honour Retry-AfterThe server is telling you exactly how long to wait
HTTP 400, 401, 404, 422NoSending the same bad request again changes nothing
Postgres serialization failure (40001)Yes, whole transactionExpected under SERIALIZABLE isolation
Postgres unique violation (23505)NoA logic or data problem, not a blip

In Haskell this maps cleanly to handlers. For http-client, match on HttpExceptionRequest and inspect the content: ConnectionFailure, ResponseTimeout and ConnectionTimeout are retryable, a StatusCodeException needs a status check. If you model failures as values instead of exceptions, which I prefer for business errors as covered in Haskell Either for error handling without exceptions, use the retrying function instead, which decides based on the returned value rather than on a thrown exception.

Never retry async exceptions. The recovering function skips them by default in current versions, but if you hand roll a loop, rethrow anything that is an instance of SomeAsyncException. Otherwise a cancelled request can sleep and retry for seconds after its thread was told to die.

Idempotency is not optional

Retrying a GET is safe. Retrying a POST that charges a card is not, unless the server supports an idempotency key. Generate a UUID once, before the retry block, and send it on every attempt. If you generate it inside the action, each retry looks like a new payment. I have seen exactly this bug double charge 11 customers during a 40 second network partition.

Where common retry advice goes wrong

The standard advice is "retry transient errors at every layer so the system is resilient". I disagree, and the reason is arithmetic. If your handler retries 3 times, the service it calls retries 3 times, and that service's database client retries 3 times, one user request can become 4 times 4 times 4, or 64 calls against the database during an incident. Retries multiply. They do not add.

The better rule is to retry at exactly one layer, usually the one closest to the failing dependency, and make every layer above it fail fast. Pair that with a cap on how many in-flight retries a process allows. A simple approach is an STM counter, the same kind of shared state described in Haskell STM for shared state in concurrent services: increment before a retry, decrement after, and skip the retry if more than, say, 10 percent of recent requests are already retries. That single guard turns a retry storm into a clean, visible error spike that your alerts can catch.

Retries also hide problems. If your dashboards only show final success, a dependency failing 30 percent of first attempts looks healthy. Emit a counter for every retry, labelled by target and error kind.

Testing retry behaviour before production

Retry code is rarely exercised until the bad day, so test it on purpose.

  • Make the policy a parameter of your client, so tests can pass limitRetries 3 with a zero base delay and run in milliseconds.
  • Use an IORef counter in a fake action that fails twice and then succeeds, and assert it ran exactly three times.
  • Add a test where the fake throws a 400 equivalent and assert it ran exactly once.
  • Check the policy arithmetic with the package's simulatePolicy, which returns the delays a policy would produce without sleeping.

For the wider system, inject faults in staging: drop 5 percent of connections to one dependency and watch total request latency at the 99th percentile. If it climbs past your timeout budget, shrink the cap or the retry count. The failure modes that only appear under real concurrency are covered in why distributed systems fail in ways unit tests never catch, and retry storms are one of the classic examples.

Frequently asked questions

How many times should a Haskell service retry a failed request?

For interactive requests, two to four retries is usually enough, with a total wait under your caller's timeout. Background jobs can afford more, sometimes 8 to 10 with a cap of 30 seconds or more. The right number comes from your latency budget, not from a round figure.

What is jitter in exponential backoff?

Jitter adds randomness to each delay so that many clients failing at once do not retry at the same instant. Full jitter picks a random delay between zero and the exponential ceiling. It slightly increases average wait for one client but sharply reduces load spikes on the recovering service.

Should I retry database transactions in Haskell?

Retry serialization failures and deadlocks, and always retry the entire transaction from the start rather than the single statement that failed. Do not retry constraint violations, since they reflect data problems. Keep the retry outside the transaction block so each attempt opens a fresh one.

FB
Finnian Burke

Finnian Burke spent eight years building distributed systems before leaving to write about the programming patterns that actually hold up under pressure. He has worked with Haskell, Rust, and Go in production, and writes mostly about the parts of functional programming that turn out to be useful regardless of the language you use day to day.

More posts by Finnian

More in Systems

01doorganico 084goad 3hot8 4465456 after5internprogram ahandfulofblessings ahmadpc ahomcybersecurity allodialresources annodex antiquesandperiodcollectibles aotc asianworldkashmir beedlectrics bloggertemplatefree bluefeatheragency bodyweightsolution campaign08blog carfixadvisor catstreetcollective cbstvsports cnfanspurchasing committeeof500yearsofdignityandresistance covertopsdjs cs-kaiyuangame cuestacomunicaciontotal cxoutlet dancebetgame deskgearhub destinyvillastay detectmagazine di33 diamondsteelbuildings donmo dyanadropship emarsysfinance emimattl escapechan eskitarih evautoplay everythingweddings-orlando farmzone feedviewerapp foxstudiosscratch gamevui123 gleasongrade greetingsday hacklabasu iw-kaiyun jagoanproperti jilidskripsi joltbusinessfinance juegodepokergratis juegospucca kaeru-festival latestmehndidesigns lestanzediottaviano litemagazines localcommunityhero lustrsazan mariashebets menswheels newwebstechs1 nikkit ootheday oxigenvpn plurfitness prefetial probudgetcarpetcleaning psychicmrsmartin pyaten qvantor rejoignezlogis rexzbot riselayouts ruchin rwxqsgd5 sakaltime shibiaowang shop-bot skrtips spotlesshaven strcoding tetapatrium topselller uneplanteartificielle vipertechcr webtech1223411 wo-kygaming yamaha-mekar yeknava yoybuysheets zenunwa yamaha-mekar jilidskripsi ankertoto ankertoto ankertoto