Haskell Health Check Endpoints for Kubernetes Probes
A student of mine shipped her first Haskell service to a small Kubernetes cluster last spring, and it restarted itself 41 times in one afternoon. Nothing was wrong with the code that served users.

A student of mine shipped her first Haskell service to a small Kubernetes cluster last spring, and it restarted itself 41 times in one afternoon. Nothing was wrong with the code that served users. The liveness probe called an endpoint that opened a Postgres connection, Postgres was slow for a few minutes during a backup, the probe timed out after one second, and the kubelet killed a perfectly healthy process over and over until the backup finished.
That is the usual way health check endpoints go wrong. They get written in ten minutes at the end of a project, they check too much or too little, and nobody looks at them again until an outage. This guide covers what each probe should actually test, how to build them in Warp with plain IORefs and STM, and the settings I now copy into every deployment manifest.
Three probes, three different questions
Kubernetes gives you three probe types, and they ask different things. Treating them as one endpoint is the root of most trouble.
| Probe | Question it answers | What happens on failure | What it should check |
|---|---|---|---|
| Liveness | Is this process stuck beyond repair? | Container is killed and restarted | Only things a restart can fix: a deadlocked main loop, a wedged worker thread |
| Readiness | Should traffic go here right now? | Pod is removed from the Service endpoints, process keeps running | Dependencies needed to serve requests, warm caches, shutdown flag |
| Startup | Has the process finished booting? | Liveness and readiness are held off until it passes | Migrations done, config loaded, initial cache fill complete |
The rule that would have saved my student: a liveness check must never depend on anything outside the process. If the database is down, restarting your web server does not bring the database back. It just adds a cold start, a fresh connection storm against a struggling Postgres, and a CrashLoopBackOff timer that grows to five minutes.
A liveness endpoint that checks the process, not the world
The cheapest useful liveness check is a heartbeat. Your important background loop writes the current time into an IORef every iteration, and the endpoint reports failure if that timestamp is older than some limit. A plain "return 200" handler tells you only that the Warp accept loop is alive, which it almost always is, even when the worker that does the real job has been blocked on an MVar for an hour.
import Data.IORef
import Data.Time.Clock
data Health = Health
{ lastBeat :: IORef UTCTime, shutting :: TVar Bool, dbOk :: TVar Bool
}
beat :: Health -> IO ()
beat h = getCurrentTime >>= atomicWriteIORef (lastBeat h)
livez :: Health -> IO (Status, ByteString)
livez h = do
now <- getCurrentTime
last <- readIORef (lastBeat h)
pure $ if diffUTCTime now last < 60
then (status200, "ok")
else (status503, "worker heartbeat stale")
Call beat at the top of each iteration of the loop you care about. Sixty seconds is generous on purpose. If one iteration can legitimately take 45 seconds, say during a big batch, then a 20 second limit will kill you during your busiest moments, which is the worst possible time for a restart.
Students often ask whether to use atomicWriteIORef or a plain write here. For a single value written by one thread and read by another, the atomic version adds a memory barrier and costs nothing you will measure. Use it and stop thinking about it.
Readiness is where the dependency checks belong
Readiness is allowed to look outside. If the pod cannot reach Postgres, it should stop receiving traffic, and Kubernetes will route requests to pods that can. Nothing gets restarted. When the dependency recovers, the pod rejoins.
The mistake I see next is doing the dependency check inside the request handler. The kubelet probes every 10 seconds by default, you might run 12 replicas, and each probe opens a connection, runs SELECT 1, and returns it. That is fine until the probe timeout is shorter than the time it takes to acquire a connection from an exhausted pool, at which point readiness flaps every time you get busy.
Instead, run the check in its own thread on its own schedule and let the endpoint read a cached result:
dbChecker :: Pool Connection -> Health -> IO ()
dbChecker pool h = forever $ do
ok <- try (withResource pool (\c -> query_ c "SELECT 1"))
atomically $ writeTVar (dbOk h) (either (const False) isOne ok)
threadDelay 5000000
readyz :: Health -> IO (Status, ByteString)
readyz h = do
(down, db) <- atomically $ () <$> readTVar (shutting h) <*> readTVar (dbOk h)
pure $ case (down, db) of
(True, _) -> (status503, "shutting down")
(_, False) -> (status503, "database unreachable")
_ -> (status200, "ready")
Now the probe handler does no IO beyond two TVar reads in one transaction, and it answers in microseconds no matter how loaded the service is. Reading both values in one atomically block also means you never report a mix of old and new state. If STM is new to you, the article on Haskell STM for shared state covers why TVars suit this better than MVars.
Give the checker thread a connection budget. With a pool of 10 connections and a checker that holds one for a few milliseconds every 5 seconds, the cost rounds to zero. If you size your pool tightly, read the notes on connection pooling for Postgres services before adding a permanent consumer to it.
The advice to check every dependency is usually wrong
A lot of health check libraries and blog posts encourage a big aggregated endpoint: database, Redis, the payments API, the S3 bucket, the email provider, each with a green or red light. It looks thorough. In practice it often makes outages bigger.
Suppose your service mostly reads from Postgres and occasionally sends email. The email provider has a bad 20 minutes. If readiness includes the email check, every replica goes unready at once, the Service has zero endpoints, and users get connection refused for pages that never touch email. You turned a partial degradation into a full outage.
My rule now is narrow: readiness includes only dependencies without which the pod cannot serve the majority of its traffic. For a typical CRUD API that means the primary database and maybe nothing else. Optional dependencies get metrics and alerts, not probe membership. Your handlers should already treat them as fallible, with timeouts and something like the patterns in retry with exponential backoff.
I will admit this is a judgement call, and I have seen one team where a cache really was mandatory because the database alone could not hold the read load. For them, putting Redis in readiness was right. Count which requests break without each dependency before you decide.
Wiring readiness into graceful shutdown
The shutting flag in the record above exists for deploys. When a pod gets SIGTERM, Kubernetes starts removing it from endpoints, but that removal propagates through kube-proxy and any ingress controller with a delay of a few seconds. Requests keep arriving during that window.
The sequence that works for me:
- On SIGTERM, set
shuttingto True so readiness fails immediately. - Keep serving for about 5 seconds while the endpoint removal spreads.
- Then stop the listener and drain in-flight requests within whatever is left of
terminationGracePeriodSeconds.
The full drain logic is in the guide to graceful shutdown for long running services. The only addition here is that the flag lives in the same Health record, so the signal handler and the probe share one source of truth.
Probe settings I start from
Defaults in Kubernetes are a one second timeout, a 10 second period and a failure threshold of 3. For a Haskell service compiled with the threaded runtime, a one second timeout is usually fine for the endpoints above, because they do no real work. The trouble appears during a major GC on a large heap, where a pause of 400 to 900 milliseconds is not unusual with the default copying collector.
The values I put in a fresh manifest:
- Liveness: period 10s, timeout 3s, failure threshold 6. That is a full minute of consecutive failures before a restart.
- Readiness: period 5s, timeout 2s, failure threshold 2, success threshold 1.
- Startup: period 5s, failure threshold 36, which allows three minutes for migrations on first boot.
Serve the probe endpoints on a separate port, something like 8081, with its own small Warp instance. That keeps them out of your public routing, away from auth middleware, and away from request logging that would otherwise add 17,000 useless log lines a day per replica.
FAQ
Should a Haskell liveness probe check the database?
No. A restart cannot fix the database, so a database check in liveness only adds restarts on top of an outage. Put the database in readiness.
What path names should health endpoints use?
The convention /livez and /readyz comes from the Kubernetes API server itself. Any names work, but these make the purpose obvious to whoever reads the manifest next.
Do I need a startup probe if I already have readiness?
Only if boot can take longer than liveness tolerates. Without one, a slow migration can trip liveness and restart the pod in the middle of the migration.
Can GC pauses fail a health check?
Yes, on large heaps with the copying collector. Raise the probe timeout to 2 or 3 seconds, or try the nonmoving collector with the +RTS -xn flag if pauses keep showing up.
More in Systems
Systems
Haskell Bounded Concurrency for Background Workers
A worker that drains a job queue is the first Haskell service most teams ship, and it is also the first one that gets killed by...
Systems
Haskell Structured Logging for Production Services
Your Haskell service crashed at 03:12, and the only clue is a wall of putStrLn output with no request id, no timestamp...
Systems
Haskell Retry With Exponential Backoff for Flaky Services
Your Haskell service calls a payment API, a Postgres replica, or an S3 bucket, and roughly one request in two hundred fails...