diff --git a/.claude/skills/ipx-prod-check/SKILL.md b/.claude/skills/ipx-prod-check/SKILL.md index a48cd44..ed9fd8f 100644 --- a/.claude/skills/ipx-prod-check/SKILL.md +++ b/.claude/skills/ipx-prod-check/SKILL.md @@ -43,7 +43,7 @@ Start every check by seeing that all three run, since Loki cannot see a containe docker ps -a --filter name=iPX --filter name=ipx-web --filter name=ipx-directory --format '{{.Names}} {{.Status}}' ``` -All three should be `(healthy)`. Then work through ipx (the checks below), the page, the +All three should be `Up`, with no recent restart; they have no Docker healthcheck (#218). Then work through ipx (the checks below), the page, the directory, Tower and the monitoring itself, in that order. ## ipx @@ -226,8 +226,8 @@ a count says something happened, the lines and traces say why. Caddy logs one JSON line a request (#210), logger `http.log.access.log0`: Loki's `| json` gives `request_uri`, `request_method`, `request_remote_ip`, `status`, `duration` (seconds, not ms), `size`. -The request's headers and a `?token=` are filtered out before it is written. Its healthcheck -requests come from `127.0.0.1` every 30 s; leave them out. `/api` and `/media` pass on to ipx and +The request's headers and a `?token=` are filtered out before it is written. Requests from +`127.0.0.1` are local checks; leave them out. `/api` and `/media` pass on to ipx and are in ipx's own access log too, with ipx's time; Caddy's `duration` for them adds the hop. ```sh diff --git a/CLAUDE.md b/CLAUDE.md index f0802c5..3cab993 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -103,11 +103,13 @@ docker tag mirror.gcr.io/library/rust:1-slim-trixie rust:1-slim-trixie Run those again now and then, or the local copies go stale. -The healthcheck runs `ipx status` against the control socket, so `(healthy)` in `docker ps` means -the daemon answers there and can read its database, not just that the web port is up. The socket -answers `status` itself instead of queuing it behind the worker's current job, so a long scan or -download does not fail the check; it also means a worker stuck on one job would still pass. The -container restarts on its own after a reboot. +The containers have no Docker healthcheck (#218): Docker runs one with `exec`, which on Tower +fails whenever root is logged out, because elogind removes `/run/user/0`, where containerd puts +each exec's state. Whether ipx is well is Prometheus's question (`up`, the worker's job and time) +and `ipx status` against the control socket, which says the daemon answers and can read its +database. The socket answers `status` itself instead of queuing it behind the worker's current +job, so a worker stuck on one job still answers it. The containers restart on their own after a +reboot. Before the container, ipx ran by hand in code-server, with its files in `/config/.config/ipx/` and `/config/.local/share/ipx/`. Those are still there and the container does not read them. If you run @@ -199,8 +201,7 @@ seeking through files, a flood of wrong passwords. Things about them: 127.0.0.1, as production trusts Cloudflare Access's; only the sign-in test uses a password. * Each threshold is a budget well above what the request takes now: it is there to catch a query that has started asking once per feed, or writers queueing on a lock, not a slow minute. -* `ipx status`, the Docker healthcheck, runs every second throughout, and a run fails if one takes - the healthcheck's 5s. +* `ipx status` runs every second throughout, and a run fails if one takes 5s. * One at a time: `node tests/load/run.js signin`. All four take about six minutes. * The daemon is on **SQLite** unless `--postgres`, which puts it in `IPX_TEST_DATABASE_URL`, in a schema of its own (`ipx_load`) dropped and made again each run. That URL, in `/src/.envrc`, is diff --git a/crates/directory/CHANGELOG.md b/crates/directory/CHANGELOG.md index 0d8121a..3362945 100644 --- a/crates/directory/CHANGELOG.md +++ b/crates/directory/CHANGELOG.md @@ -10,6 +10,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- The example in docs/directory.md has no healthcheck; `ipx-directory health` is still there to run by hand (#218). + ## [1.0.0] - 2026-10-12 ### Added diff --git a/crates/directory/src/main.rs b/crates/directory/src/main.rs index 0f38daa..460f543 100644 --- a/crates/directory/src/main.rs +++ b/crates/directory/src/main.rs @@ -25,8 +25,7 @@ enum Cmd { #[command(subcommand)] cmd: TokenCmd, }, - /// Check that the directory on this machine answers and can read its database: the - /// container's healthcheck. + /// Check that the directory on this machine answers and can read its database. Health, } diff --git a/crates/ipx/CHANGELOG.md b/crates/ipx/CHANGELOG.md index 1dfe3f7..af3e159 100644 --- a/crates/ipx/CHANGELOG.md +++ b/crates/ipx/CHANGELOG.md @@ -12,6 +12,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- `docker-compose.yml` has no healthchecks: Docker runs them with exec, which can fail on the host while ipx is well. The library's counts in `/metrics` are kept every 30 s by the daemon itself (#218). + ## [1.0.0] - 2026-10-12 ### Added diff --git a/crates/ipx/src/ipc.rs b/crates/ipx/src/ipc.rs index b1b5587..26c216d 100644 --- a/crates/ipx/src/ipc.rs +++ b/crates/ipx/src/ipc.rs @@ -198,7 +198,7 @@ fn log_event(e: &Event, wire: bool) { ) }; } - // The healthcheck's answer, every 30s: a reply on the socket rather than work done, so it + // `ipx status`'s answer: a reply on the socket rather than work done, so it // stays with the rest of the conversation, and the Scans tab stays about scans. if matches!(e, Event::Status { .. }) { return line!(info, "ipx::io", format!("<- {json}")); diff --git a/crates/ipx/src/main.rs b/crates/ipx/src/main.rs index 7dbc73e..5e1917c 100644 --- a/crates/ipx/src/main.rs +++ b/crates/ipx/src/main.rs @@ -254,8 +254,8 @@ impl Ctx { #[tokio::main] async fn main() -> Result<()> { let cli = Cli::parse(); - // The daemon only: `ipx status` runs every half minute as the healthcheck, and a trace - // for each would bury the ones worth looking at. + // The daemon only: a trace for every `ipx status` a script runs would bury the ones worth + // looking at. let otel = match cli.command { Command::Daemon { .. } => ipx_directory::otel::provider("ipx")?, _ => None, @@ -568,7 +568,7 @@ pub(crate) async fn status(ctx: &Ctx) -> Event { match ctx.db.counts().await { Ok((pending, downloaded)) => { let feeds = subscriptions(ctx).await.map(|s| s.len()).unwrap_or(0); - // Kept for /metrics, which asks the database nothing (#195): the healthcheck asks this every 30 s. + // Kept for /metrics, which asks the database nothing (#195); the daemon counts every 30 s. use std::sync::atomic::Ordering::Relaxed; let st = &metrics::STATS; st.feeds.store(feeds as i64, Relaxed); @@ -663,6 +663,21 @@ async fn daemon( // must not wait behind it. tokio::spawn(deadair::run(ctx.clone(), rx_stop.clone())); tokio::spawn(directory::run(ctx.clone(), rx_stop.clone())); + // The library's counts for /metrics. The Docker healthcheck asked for them every 30 s until + // it was removed: Docker runs it with exec, which failed on Tower whenever root was logged + // out (#218). + { + let (ctx, mut stop) = (ctx.clone(), rx_stop.clone()); + tokio::spawn(async move { + let mut every = tokio::time::interval(std::time::Duration::from_secs(30)); + loop { + tokio::select! { + _ = every.tick() => { status(&ctx).await; } + _ = stop.changed() => break, + } + } + }); + } // Runs one job, abandoning it if a signal arrives. Returns false to end the loop. async fn until_stopped( diff --git a/crates/ipx/src/metrics.rs b/crates/ipx/src/metrics.rs index c5de83b..013ab4c 100644 --- a/crates/ipx/src/metrics.rs +++ b/crates/ipx/src/metrics.rs @@ -36,7 +36,7 @@ pub struct Stats { /// Artwork served from disk, and fetched from the publisher because it was not there. pub art_hits: AtomicU64, pub art_misses: AtomicU64, - /// The library as the healthcheck last counted it, every 30 s; -1 until it has. + /// The library as the daemon last counted it, every 30 s; -1 until it has. pub feeds: AtomicI64, pub pending: AtomicI64, pub downloaded: AtomicI64, @@ -218,9 +218,9 @@ fn work(out: &mut String, st: &Stats, event_queue: usize, retries: usize, art_li counter(out, "ipx_art_requests_total", "Artwork asked for: served from disk, or fetched from the publisher.", &[("result=\"hit\"", st.art_hits.load(Relaxed) as f64), ("result=\"miss\"", st.art_misses.load(Relaxed) as f64)]); for (name, help, v) in [ - ("ipx_feeds", "Feeds in the catalogue, as the healthcheck last counted.", &st.feeds), - ("ipx_downloads_pending", "Files a scan will download on its own, as the healthcheck last counted.", &st.pending), - ("ipx_files_downloaded", "Files on disk, as the healthcheck last counted.", &st.downloaded), + ("ipx_feeds", "Feeds in the catalogue, as last counted.", &st.feeds), + ("ipx_downloads_pending", "Files a scan will download on its own, as last counted.", &st.pending), + ("ipx_files_downloaded", "Files on disk, as last counted.", &st.downloaded), ] { let v = v.load(Relaxed); if v >= 0 { @@ -293,7 +293,7 @@ mod tests { assert!(out.contains("ipx_worker_busy 0\n") && !out.contains("ipx_worker_job_seconds"), "{out}"); assert!(out.contains("ipx_jobs_total{kind=\"scan\"} 1\n") && out.contains("# TYPE ipx_jobs_total counter"), "{out}"); assert!(out.contains("ipx_event_queue 3\n") && out.contains("ipx_feed_retries 1\n")); - // The library's counts are left out until the healthcheck has counted it. + // The library's counts are left out until the daemon has counted it. assert!(!out.contains("ipx_feeds ")); } diff --git a/docker-compose.yml b/docker-compose.yml index edf2647..b4229ad 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -22,12 +22,6 @@ services: - ./config:/config # config.toml, written on first start - ./data:/data # state.db - ./downloads:/downloads - healthcheck: - test: ["CMD", "ipx", "status"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 10s ipx-web: image: ipx-web @@ -47,8 +41,3 @@ services: - "8099:8080" # the web UI depends_on: - ipx - healthcheck: - test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:8080/favicon.png"] - interval: 30s - timeout: 5s - retries: 3 diff --git a/docs/directory.md b/docs/directory.md index c42d5bd..eca7c57 100644 --- a/docs/directory.md +++ b/docs/directory.md @@ -110,8 +110,6 @@ services: - "8100:8100" volumes: - ./directory:/config:ro # directory.toml - healthcheck: - test: ["CMD", "ipx-directory", "health"] ``` ## What it will not hold diff --git a/tests/load/run.js b/tests/load/run.js index edc6c35..a6e940b 100644 --- a/tests/load/run.js +++ b/tests/load/run.js @@ -8,8 +8,7 @@ // It builds a release binary (debug Argon2 alone takes a second a sign-in, which would measure // the build, not ipx), serves generated feeds from this process, starts a daemon on its own // config and data under /tmp/ipx-load, wiped first, seeds listeners, then runs each k6 script. -// While each runs, `ipx status` -- the Docker healthcheck -- is run every second, and the run -// fails if any answer takes as long as the healthcheck's 5s timeout. +// While each runs, `ipx status` is run every second, and the run fails if any answer takes 5s. // // The daemon is on SQLite unless --postgres, which puts it in IPX_TEST_DATABASE_URL, the // database the Rust tests use on Postgres too (ipodderx_test on the server production's is on, in