No Docker healthchecks: the daemon keeps /metrics' counts itself (#218)

Docker runs a healthcheck with exec, and containerd puts each exec's state in XDG_RUNTIME_DIR,
/run/user/0 on Tower, which elogind removes when root's last session ends. Every check then
failed and every container showed unhealthy while all of them answered. The checks are gone from
the sample compose file and production's; ipx's library counts for /metrics, which only the
healthcheck's status call refreshed, are counted every 30 s by the daemon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.claude/skills/ipx-prod-check/SKILL.md

@@ -43,7 +43,7 @@ Start every check by seeing that all three run, since Loki cannot see a containe
 docker ps -a --filter name=iPX --filter name=ipx-web --filter name=ipx-directory --format '{{.Names}} {{.Status}}'
 ```
 
-All three should be `(healthy)`. Then work through ipx (the checks below), the page, the
+All three should be `Up`, with no recent restart; they have no Docker healthcheck (#218). Then work through ipx (the checks below), the page, the
 directory, Tower and the monitoring itself, in that order.
 
 ## ipx
@@ -226,8 +226,8 @@ a count says something happened, the lines and traces say why.
 
 Caddy logs one JSON line a request (#210), logger `http.log.access.log0`: Loki's `| json` gives
 `request_uri`, `request_method`, `request_remote_ip`, `status`, `duration` (seconds, not ms), `size`.
-The request's headers and a `?token=` are filtered out before it is written. Its healthcheck
-requests come from `127.0.0.1` every 30 s; leave them out. `/api` and `/media` pass on to ipx and
+The request's headers and a `?token=` are filtered out before it is written. Requests from
+`127.0.0.1` are local checks; leave them out. `/api` and `/media` pass on to ipx and
 are in ipx's own access log too, with ipx's time; Caddy's `duration` for them adds the hop.
 
 ```sh

CLAUDE.md

@@ -103,11 +103,13 @@ docker tag mirror.gcr.io/library/rust:1-slim-trixie rust:1-slim-trixie
 
 Run those again now and then, or the local copies go stale.
 
-The healthcheck runs `ipx status` against the control socket, so `(healthy)` in `docker ps` means
-the daemon answers there and can read its database, not just that the web port is up. The socket
-answers `status` itself instead of queuing it behind the worker's current job, so a long scan or
-download does not fail the check; it also means a worker stuck on one job would still pass. The
-container restarts on its own after a reboot.
+The containers have no Docker healthcheck (#218): Docker runs one with `exec`, which on Tower
+fails whenever root is logged out, because elogind removes `/run/user/0`, where containerd puts
+each exec's state. Whether ipx is well is Prometheus's question (`up`, the worker's job and time)
+and `ipx status` against the control socket, which says the daemon answers and can read its
+database. The socket answers `status` itself instead of queuing it behind the worker's current
+job, so a worker stuck on one job still answers it. The containers restart on their own after a
+reboot.
 
 Before the container, ipx ran by hand in code-server, with its files in `/config/.config/ipx/` and
 `/config/.local/share/ipx/`. Those are still there and the container does not read them. If you run
@@ -199,8 +201,7 @@ seeking through files, a flood of wrong passwords. Things about them:
   127.0.0.1, as production trusts Cloudflare Access's; only the sign-in test uses a password.
 * Each threshold is a budget well above what the request takes now: it is there to catch a query
   that has started asking once per feed, or writers queueing on a lock, not a slow minute.
-* `ipx status`, the Docker healthcheck, runs every second throughout, and a run fails if one takes
-  the healthcheck's 5s.
+* `ipx status` runs every second throughout, and a run fails if one takes 5s.
 * One at a time: `node tests/load/run.js signin`. All four take about six minutes.
 * The daemon is on **SQLite** unless `--postgres`, which puts it in `IPX_TEST_DATABASE_URL`, in
   a schema of its own (`ipx_load`) dropped and made again each run. That URL, in `/src/.envrc`, is

crates/directory/CHANGELOG.md

@@ -10,6 +10,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- The example in docs/directory.md has no healthcheck; `ipx-directory health` is still there to run by hand (#218).
+
 ## [1.0.0] - 2026-10-12
 
 ### Added

crates/directory/src/main.rs

@@ -25,8 +25,7 @@ enum Cmd {
         #[command(subcommand)]
         cmd: TokenCmd,
     },
-    /// Check that the directory on this machine answers and can read its database: the
-    /// container's healthcheck.
+    /// Check that the directory on this machine answers and can read its database.
     Health,
 }
 

crates/ipx/CHANGELOG.md

@@ -12,6 +12,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- `docker-compose.yml` has no healthchecks: Docker runs them with exec, which can fail on the host while ipx is well. The library's counts in `/metrics` are kept every 30 s by the daemon itself (#218).
+
 ## [1.0.0] - 2026-10-12
 
 ### Added

crates/ipx/src/ipc.rs

@@ -198,7 +198,7 @@ fn log_event(e: &Event, wire: bool) {
             )
         };
     }
-    // The healthcheck's answer, every 30s: a reply on the socket rather than work done, so it
+    // `ipx status`'s answer: a reply on the socket rather than work done, so it
     // stays with the rest of the conversation, and the Scans tab stays about scans.
     if matches!(e, Event::Status { .. }) {
         return line!(info, "ipx::io", format!("<- {json}"));

crates/ipx/src/main.rs

@@ -254,8 +254,8 @@ impl Ctx {
 #[tokio::main]
 async fn main() -> Result<()> {
     let cli = Cli::parse();
-    // The daemon only: `ipx status` runs every half minute as the healthcheck, and a trace
-    // for each would bury the ones worth looking at.
+    // The daemon only: a trace for every `ipx status` a script runs would bury the ones worth
+    // looking at.
     let otel = match cli.command {
         Command::Daemon { .. } => ipx_directory::otel::provider("ipx")?,
         _ => None,
@@ -568,7 +568,7 @@ pub(crate) async fn status(ctx: &Ctx) -> Event {
     match ctx.db.counts().await {
         Ok((pending, downloaded)) => {
             let feeds = subscriptions(ctx).await.map(|s| s.len()).unwrap_or(0);
-            // Kept for /metrics, which asks the database nothing (#195): the healthcheck asks this every 30 s.
+            // Kept for /metrics, which asks the database nothing (#195); the daemon counts every 30 s.
             use std::sync::atomic::Ordering::Relaxed;
             let st = &metrics::STATS;
             st.feeds.store(feeds as i64, Relaxed);
@@ -663,6 +663,21 @@ async fn daemon(
     // must not wait behind it.
     tokio::spawn(deadair::run(ctx.clone(), rx_stop.clone()));
     tokio::spawn(directory::run(ctx.clone(), rx_stop.clone()));
+    // The library's counts for /metrics. The Docker healthcheck asked for them every 30 s until
+    // it was removed: Docker runs it with exec, which failed on Tower whenever root was logged
+    // out (#218).
+    {
+        let (ctx, mut stop) = (ctx.clone(), rx_stop.clone());
+        tokio::spawn(async move {
+            let mut every = tokio::time::interval(std::time::Duration::from_secs(30));
+            loop {
+                tokio::select! {
+                    _ = every.tick() => { status(&ctx).await; }
+                    _ = stop.changed() => break,
+                }
+            }
+        });
+    }
 
     // Runs one job, abandoning it if a signal arrives. Returns false to end the loop.
     async fn until_stopped(

crates/ipx/src/metrics.rs

@@ -36,7 +36,7 @@ pub struct Stats {
     /// Artwork served from disk, and fetched from the publisher because it was not there.
     pub art_hits: AtomicU64,
     pub art_misses: AtomicU64,
-    /// The library as the healthcheck last counted it, every 30 s; -1 until it has.
+    /// The library as the daemon last counted it, every 30 s; -1 until it has.
     pub feeds: AtomicI64,
     pub pending: AtomicI64,
     pub downloaded: AtomicI64,
@@ -218,9 +218,9 @@ fn work(out: &mut String, st: &Stats, event_queue: usize, retries: usize, art_li
     counter(out, "ipx_art_requests_total", "Artwork asked for: served from disk, or fetched from the publisher.",
         &[("result=\"hit\"", st.art_hits.load(Relaxed) as f64), ("result=\"miss\"", st.art_misses.load(Relaxed) as f64)]);
     for (name, help, v) in [
-        ("ipx_feeds", "Feeds in the catalogue, as the healthcheck last counted.", &st.feeds),
-        ("ipx_downloads_pending", "Files a scan will download on its own, as the healthcheck last counted.", &st.pending),
-        ("ipx_files_downloaded", "Files on disk, as the healthcheck last counted.", &st.downloaded),
+        ("ipx_feeds", "Feeds in the catalogue, as last counted.", &st.feeds),
+        ("ipx_downloads_pending", "Files a scan will download on its own, as last counted.", &st.pending),
+        ("ipx_files_downloaded", "Files on disk, as last counted.", &st.downloaded),
     ] {
         let v = v.load(Relaxed);
         if v >= 0 {
@@ -293,7 +293,7 @@ mod tests {
         assert!(out.contains("ipx_worker_busy 0\n") && !out.contains("ipx_worker_job_seconds"), "{out}");
         assert!(out.contains("ipx_jobs_total{kind=\"scan\"} 1\n") && out.contains("# TYPE ipx_jobs_total counter"), "{out}");
         assert!(out.contains("ipx_event_queue 3\n") && out.contains("ipx_feed_retries 1\n"));
-        // The library's counts are left out until the healthcheck has counted it.
+        // The library's counts are left out until the daemon has counted it.
         assert!(!out.contains("ipx_feeds "));
     }
 

docker-compose.yml

@@ -22,12 +22,6 @@ services:
       - ./config:/config         # config.toml, written on first start
       - ./data:/data             # state.db
       - ./downloads:/downloads
-    healthcheck:
-      test: ["CMD", "ipx", "status"]
-      interval: 30s
-      timeout: 5s
-      retries: 3
-      start_period: 10s
 
   ipx-web:
     image: ipx-web
@@ -47,8 +41,3 @@ services:
       - "8099:8080"          # the web UI
     depends_on:
       - ipx
-    healthcheck:
-      test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:8080/favicon.png"]
-      interval: 30s
-      timeout: 5s
-      retries: 3

docs/directory.md

@@ -110,8 +110,6 @@ services:
       - "8100:8100"
     volumes:
       - ./directory:/config:ro    # directory.toml
-    healthcheck:
-      test: ["CMD", "ipx-directory", "health"]
 ```
 
 ## What it will not hold

tests/load/run.js

@@ -8,8 +8,7 @@
 // It builds a release binary (debug Argon2 alone takes a second a sign-in, which would measure
 // the build, not ipx), serves generated feeds from this process, starts a daemon on its own
 // config and data under /tmp/ipx-load, wiped first, seeds listeners, then runs each k6 script.
-// While each runs, `ipx status` -- the Docker healthcheck -- is run every second, and the run
-// fails if any answer takes as long as the healthcheck's 5s timeout.
+// While each runs, `ipx status` is run every second, and the run fails if any answer takes 5s.
 //
 // The daemon is on SQLite unless --postgres, which puts it in IPX_TEST_DATABASE_URL, the
 // database the Rust tests use on Postgres too (ipodderx_test on the server production's is on, in