Haskell World logo Haskell WorldWrite code, build worlds
Systems

Haskell Health Check Endpoints for Kubernetes Probes

A student of mine shipped her first Haskell service to a small Kubernetes cluster last spring, and it restarted itself 41 times in one afternoon. Nothing was wrong with the code that served users.

A terminal showing kubectl describe pod output with probe results next to a Haskell source file in an editor

A student of mine shipped her first Haskell service to a small Kubernetes cluster last spring, and it restarted itself 41 times in one afternoon. Nothing was wrong with the code that served users. The liveness probe called an endpoint that opened a Postgres connection, Postgres was slow for a few minutes during a backup, the probe timed out after one second, and the kubelet killed a perfectly healthy process over and over until the backup finished.

That is the usual way health check endpoints go wrong. They get written in ten minutes at the end of a project, they check too much or too little, and nobody looks at them again until an outage. This guide covers what each probe should actually test, how to build them in Warp with plain IORefs and STM, and the settings I now copy into every deployment manifest.

Three probes, three different questions

Kubernetes gives you three probe types, and they ask different things. Treating them as one endpoint is the root of most trouble.

ProbeQuestion it answersWhat happens on failureWhat it should check
LivenessIs this process stuck beyond repair?Container is killed and restartedOnly things a restart can fix: a deadlocked main loop, a wedged worker thread
ReadinessShould traffic go here right now?Pod is removed from the Service endpoints, process keeps runningDependencies needed to serve requests, warm caches, shutdown flag
StartupHas the process finished booting?Liveness and readiness are held off until it passesMigrations done, config loaded, initial cache fill complete

The rule that would have saved my student: a liveness check must never depend on anything outside the process. If the database is down, restarting your web server does not bring the database back. It just adds a cold start, a fresh connection storm against a struggling Postgres, and a CrashLoopBackOff timer that grows to five minutes.

A liveness endpoint that checks the process, not the world

The cheapest useful liveness check is a heartbeat. Your important background loop writes the current time into an IORef every iteration, and the endpoint reports failure if that timestamp is older than some limit. A plain "return 200" handler tells you only that the Warp accept loop is alive, which it almost always is, even when the worker that does the real job has been blocked on an MVar for an hour.

import Data.IORef
import Data.Time.Clock

data Health = Health
 { lastBeat :: IORef UTCTime, shutting :: TVar Bool, dbOk :: TVar Bool
 }

beat :: Health -> IO ()
beat h = getCurrentTime >>= atomicWriteIORef (lastBeat h)

livez :: Health -> IO (Status, ByteString)
livez h = do
 now <- getCurrentTime
 last <- readIORef (lastBeat h)
 pure $ if diffUTCTime now last < 60
 then (status200, "ok")
 else (status503, "worker heartbeat stale")

Call beat at the top of each iteration of the loop you care about. Sixty seconds is generous on purpose. If one iteration can legitimately take 45 seconds, say during a big batch, then a 20 second limit will kill you during your busiest moments, which is the worst possible time for a restart.

Students often ask whether to use atomicWriteIORef or a plain write here. For a single value written by one thread and read by another, the atomic version adds a memory barrier and costs nothing you will measure. Use it and stop thinking about it.

Readiness is where the dependency checks belong

Readiness is allowed to look outside. If the pod cannot reach Postgres, it should stop receiving traffic, and Kubernetes will route requests to pods that can. Nothing gets restarted. When the dependency recovers, the pod rejoins.

The mistake I see next is doing the dependency check inside the request handler. The kubelet probes every 10 seconds by default, you might run 12 replicas, and each probe opens a connection, runs SELECT 1, and returns it. That is fine until the probe timeout is shorter than the time it takes to acquire a connection from an exhausted pool, at which point readiness flaps every time you get busy.

Instead, run the check in its own thread on its own schedule and let the endpoint read a cached result:

dbChecker :: Pool Connection -> Health -> IO ()
dbChecker pool h = forever $ do
 ok <- try (withResource pool (\c -> query_ c "SELECT 1"))
 atomically $ writeTVar (dbOk h) (either (const False) isOne ok)
 threadDelay 5000000

readyz :: Health -> IO (Status, ByteString)
readyz h = do
 (down, db) <- atomically $ () <$> readTVar (shutting h) <*> readTVar (dbOk h)
 pure $ case (down, db) of
 (True, _) -> (status503, "shutting down")
 (_, False) -> (status503, "database unreachable")
 _ -> (status200, "ready")

Now the probe handler does no IO beyond two TVar reads in one transaction, and it answers in microseconds no matter how loaded the service is. Reading both values in one atomically block also means you never report a mix of old and new state. If STM is new to you, the article on Haskell STM for shared state covers why TVars suit this better than MVars.

Give the checker thread a connection budget. With a pool of 10 connections and a checker that holds one for a few milliseconds every 5 seconds, the cost rounds to zero. If you size your pool tightly, read the notes on connection pooling for Postgres services before adding a permanent consumer to it.

The advice to check every dependency is usually wrong

A lot of health check libraries and blog posts encourage a big aggregated endpoint: database, Redis, the payments API, the S3 bucket, the email provider, each with a green or red light. It looks thorough. In practice it often makes outages bigger.

Suppose your service mostly reads from Postgres and occasionally sends email. The email provider has a bad 20 minutes. If readiness includes the email check, every replica goes unready at once, the Service has zero endpoints, and users get connection refused for pages that never touch email. You turned a partial degradation into a full outage.

My rule now is narrow: readiness includes only dependencies without which the pod cannot serve the majority of its traffic. For a typical CRUD API that means the primary database and maybe nothing else. Optional dependencies get metrics and alerts, not probe membership. Your handlers should already treat them as fallible, with timeouts and something like the patterns in retry with exponential backoff.

I will admit this is a judgement call, and I have seen one team where a cache really was mandatory because the database alone could not hold the read load. For them, putting Redis in readiness was right. Count which requests break without each dependency before you decide.

Wiring readiness into graceful shutdown

The shutting flag in the record above exists for deploys. When a pod gets SIGTERM, Kubernetes starts removing it from endpoints, but that removal propagates through kube-proxy and any ingress controller with a delay of a few seconds. Requests keep arriving during that window.

The sequence that works for me:

  • On SIGTERM, set shutting to True so readiness fails immediately.
  • Keep serving for about 5 seconds while the endpoint removal spreads.
  • Then stop the listener and drain in-flight requests within whatever is left of terminationGracePeriodSeconds.

The full drain logic is in the guide to graceful shutdown for long running services. The only addition here is that the flag lives in the same Health record, so the signal handler and the probe share one source of truth.

Probe settings I start from

Defaults in Kubernetes are a one second timeout, a 10 second period and a failure threshold of 3. For a Haskell service compiled with the threaded runtime, a one second timeout is usually fine for the endpoints above, because they do no real work. The trouble appears during a major GC on a large heap, where a pause of 400 to 900 milliseconds is not unusual with the default copying collector.

The values I put in a fresh manifest:

  • Liveness: period 10s, timeout 3s, failure threshold 6. That is a full minute of consecutive failures before a restart.
  • Readiness: period 5s, timeout 2s, failure threshold 2, success threshold 1.
  • Startup: period 5s, failure threshold 36, which allows three minutes for migrations on first boot.

Serve the probe endpoints on a separate port, something like 8081, with its own small Warp instance. That keeps them out of your public routing, away from auth middleware, and away from request logging that would otherwise add 17,000 useless log lines a day per replica.

FAQ

Should a Haskell liveness probe check the database?

No. A restart cannot fix the database, so a database check in liveness only adds restarts on top of an outage. Put the database in readiness.

What path names should health endpoints use?

The convention /livez and /readyz comes from the Kubernetes API server itself. Any names work, but these make the purpose obvious to whoever reads the manifest next.

Do I need a startup probe if I already have readiness?

Only if boot can take longer than liveness tolerates. Without one, a slow migration can trip liveness and restart the pod in the middle of the migration.

Can GC pauses fail a health check?

Yes, on large heaps with the copying collector. Raise the probe timeout to 2 or 3 seconds, or try the nonmoving collector with the +RTS -xn flag if pauses keep showing up.

AS
Adil Sato

Adil Sato teaches programming concepts at a community college and writes online guides for the questions students ask most in the second week of class, not the first. His focus is on making type systems, monads, and functional patterns understandable without reducing them to metaphors that break down the moment you try to use them for real work.

More posts by Adil

More in Systems

01doorganico 084goad 3hot8 4465456 after5internprogram ahandfulofblessings ahmadpc ahomcybersecurity allodialresources annodex antiquesandperiodcollectibles aotc asianworldkashmir beedlectrics bloggertemplatefree bluefeatheragency bodyweightsolution campaign08blog carfixadvisor catstreetcollective cbstvsports cnfanspurchasing committeeof500yearsofdignityandresistance covertopsdjs cs-kaiyuangame cuestacomunicaciontotal cxoutlet dancebetgame deskgearhub destinyvillastay detectmagazine di33 diamondsteelbuildings donmo dyanadropship emarsysfinance emimattl escapechan eskitarih evautoplay everythingweddings-orlando farmzone feedviewerapp foxstudiosscratch gamevui123 gleasongrade greetingsday hacklabasu iw-kaiyun jagoanproperti jilidskripsi joltbusinessfinance juegodepokergratis juegospucca kaeru-festival latestmehndidesigns lestanzediottaviano litemagazines localcommunityhero lustrsazan mariashebets menswheels newwebstechs1 nikkit ootheday oxigenvpn plurfitness prefetial probudgetcarpetcleaning psychicmrsmartin pyaten qvantor rejoignezlogis rexzbot riselayouts ruchin rwxqsgd5 sakaltime shibiaowang shop-bot skrtips spotlesshaven strcoding tetapatrium topselller uneplanteartificielle vipertechcr webtech1223411 wo-kygaming yamaha-mekar yeknava yoybuysheets zenunwa yamaha-mekar jilidskripsi ankertoto ankertoto ankertoto