Haskell Graceful Shutdown for Long Running Services

Your Haskell service gets a SIGTERM during a deploy, the process dies mid-request, and somewhere a client sees a 502 while a half-written row sits in Postgres. Getting Haskell graceful shutdown right means deciding, ahead of time, what the process does in the 30 seconds between "please stop" and the kernel pulling the plug.

This guide walks through the pieces I wire into every long running Haskell service: catching the signal, stopping new work, draining in-flight work, flushing state, and exiting with a sensible code. The examples use plain base, async, and Warp, because that is what most production services already depend on.

What Haskell graceful shutdown actually has to do

A shutdown has four jobs, and they happen in a fixed order:

1. Notice the signal. Kubernetes, systemd, and Docker all send SIGTERM first, then SIGKILL after a grace period. Kubernetes defaults to 30 seconds, systemd to 90, Docker to 10.
2. Stop accepting new work. Close the listening socket, stop polling the queue, and fail the readiness probe so the load balancer routes elsewhere.
3. Finish or abandon in-flight work. Let current requests and jobs complete, up to a deadline you choose.
4. Release resources and exit. Flush logs and metrics, return pool connections, commit offsets, then exit.

Most broken shutdowns skip step 2 or have no deadline on step 3. The first causes dropped requests; the second causes a SIGKILL that undoes all your care.

Catching SIGTERM in Haskell

By default the GHC runtime treats SIGINT as an asynchronous UserInterrupt thrown to the main thread, but SIGTERM simply kills the process. No finalizers, no bracket cleanup. That surprises people who assume bracket will save them. It will not, because nothing is thrown.

The fix is to install a handler with System.Posix.Signals from the unix package. The handler should do almost nothing itself. Its only job is to flip a flag that the rest of the program watches:

• Create a TMVar () or MVar () called something like shutdownSignal in main.
• Call installHandler sigTERM (Catch (void (tryPutMVar shutdownSignal ()))) Nothing, and do the same for sigINT so local Ctrl+C behaves identically.
• Have main block on readMVar shutdownSignal after starting workers, or race it against the server with race_ from async.

Using tryPutMVar matters. A second SIGTERM, which operators often send when they get impatient, should not block or crash the handler thread. If you want a second signal to force an immediate exit, check whether the MVar is already full and call exitImmediately in that branch.

Draining Warp without dropping requests

Warp has supported graceful shutdown since version 3.3 through two settings: setInstallShutdownHandler and setGracefulShutdownTimeout. The first gives you a callback that receives an action to close the listening socket. The second tells Warp how many seconds to wait for open connections to finish once that socket is closed.

The pattern looks like this in words:

1. In the install handler, register your SIGTERM handler so that it calls the provided close action.
2. Set the graceful timeout to something below your orchestrator's grace period. With Kubernetes at 30 seconds, I use 20, leaving 10 seconds for the rest of cleanup.
3. When runSettings returns, the socket is closed and connections have drained or timed out. Now run the rest of your teardown.

Before closing the socket, flip your readiness endpoint to return 503 and wait a few seconds. Kubernetes removes a pod from its Service endpoints asynchronously, and for roughly 2 to 5 seconds after SIGTERM it may still send traffic to the old pod. A threadDelay 5000000 between "readiness off" and "close socket" removes a whole class of mysterious deploy-time errors. I covered why these timing bugs never show up in tests in why distributed systems fail in ways unit tests never catch.

Long-lived connections

WebSockets and server-sent event streams will never finish on their own, so the graceful timeout will always expire for them. Track them in a TVar (Set ThreadId) and send each a close frame when shutdown begins. Clients that reconnect will land on a healthy pod.

Background workers and the async package

HTTP is usually the easy part. Queue consumers, schedulers, and outbox pollers are where data gets lost. Each worker loop should check the shutdown flag between units of work, never in the middle of one.

A clean structure:

• Represent the flag as a TVar Bool. Workers read it with readTVarIO at the top of each loop iteration and exit when it is true.
• Start workers with withAsync or concurrently_, so an exception in any worker cancels the others and the program cannot leave orphaned threads.
• Give the drain phase a deadline with timeout from System.Timeout. If workers are still busy after 15 seconds, cancel them and accept that the current unit will be retried.

That last point is the reason idempotency matters. If your queue consumer acknowledges a message only after the side effect commits, a cancelled job just gets redelivered. If you acknowledge first, a cancellation loses it. If you share state between these workers, Haskell STM for shared state in concurrent services covers how to keep that coordination safe.

The advice I disagree with

A common recommendation is to wrap everything in bracket and finally and trust exceptions to unwind cleanly on shutdown. I think this is the wrong default for services, for two reasons.

First, as noted above, SIGTERM does not throw anything unless you make it. Second, even when you do convert the signal into an async exception thrown to main, it arrives at an arbitrary point. A thread killed in the middle of a database transaction will roll back, which is fine, but a thread killed between writing a file and renaming it leaves debris. Asynchronous exceptions are a fine last resort and a poor plan.

Cooperative shutdown, where each component checks a flag at a safe point, is slower by a few hundred milliseconds and far easier to reason about. Keep bracket for resource release, which is what it is good at, and use a flag for control flow.

Flushing, exit codes, and testing

Once workers have stopped, release resources in reverse order of acquisition. Typical order for a service I maintain:

Exit with 0 after a requested shutdown. A nonzero code tells systemd or Kubernetes the service crashed, which pollutes alerts and can trigger restart backoff. Reserve nonzero codes for real failures, including a drain that exceeded its deadline.

Test the whole thing locally with a load generator such as hey or oha running 200 requests per second while you send kill -TERM to the process. The success criterion is simple: zero failed requests and a clean exit within your budget. If memory climbs during the drain, the techniques in finding and fixing a Haskell space leak apply here too.

Frequently asked questions

Does GHC run finalizers when a Haskell program receives SIGTERM?

Not by default. Without a handler, SIGTERM terminates the process immediately and no Haskell code runs. Install a handler with System.Posix.Signals to turn the signal into something your program can react to.

How long should a graceful shutdown timeout be?

Set it a few seconds below whatever your orchestrator allows, so your own cleanup finishes before SIGKILL. With the Kubernetes default of 30 seconds, a 20 second HTTP drain plus 5 seconds of readiness delay is a safe split. Raise terminationGracePeriodSeconds if requests regularly run longer.

Does graceful shutdown work the same on Windows?

No. Windows has no SIGTERM, and the unix package does not build there. Use the console control handler from the Win32 package, or the cross-platform signal helpers some libraries provide, and keep the same flag-based design behind it.