--- name: ipx-prod-check description: Look for problems in production ipx (the ipx-web, iPX and ipx-directory containers on Tower, Tower itself and the monitoring that watches them) from their logs in Loki, their traces in Tempo and the metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site. --- # Checking production ipx Production is three containers in the Arcane project `ipx` on Tower (`/mnt/fast/arcane/projects/ipx/compose.yaml`), one request passing through them in this order: | container | what | logs, in Loki | traces, in Tempo | metrics, in Prometheus | |---|---|---|---|---| | `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | service `ipx-web`: a span a request, passing `traceparent` on to ipx (#213) | `job="ipx-web"`: Caddy's own `caddy_http_*`, from `METRICS_BIND`, published as `192.168.1.130:9190` (#371) | | `iPX` | the daemon: API, scans, downloads, DeadAir | `{container="iPX"}`, JSON (`IPX_LOG_FORMAT=json`) | service `ipx`; passes `traceparent` to the directory | `job="ipx"`, its `/metrics` | | `ipx-directory` | the directory of public podcasts, `directory.ipxreader.app` (#204, #214) | `{container="ipx-directory"}`, JSON (`DIRECTORY_LOG_FORMAT=json`) | service `ipx-directory`: a span a request, joining ipx's trace when ipx called; a trace a refresher round (#213) | `job="ipx-directory"`, its `/metrics` | Grafana's alert rules (`grafana/alerts.py`, #365) watch all of it between checks: start by asking which are firing (the monitoring section). Under them, Tower itself: `job="node"` (the node exporter: load, CPU, memory, disks). The Grafana dashboard `iPX` draws all of it in that order (`grafana/dashboard.py`). A trace without the production tag (`resource.deployment.environment.name="production"`, on all three) is a daemon run by hand, not production. Since #213 a request is one trace across the services it touched: a Directory page is ipx-web, then ipx, then the directory, so start from the trace, not the logs. Alloy ships the logs; Loki, Tempo and Prometheus publish no query port to this machine, so use the script beside this file: ```sh Q=.claude/skills/ipx-prod-check/query.py $Q logs '' [since] # lines, newest first (500 at most) $Q metric '' [since] # $range becomes `since` $Q traces '' [since] # slowest first $Q spans '' [since] # the matching spans, each with its own time, slowest first $Q trace # one trace as a tree, with its log lines $Q prom '' [since] # Prometheus, now; $range becomes `since` $Q flame '' [since] [out.svg] # where the matching spans' time went, summed over up to # 100 traces: a flame graph, and out.svg.folded for speedscope ``` `since` is `30m`, `1h`, `24h`, `7d`. Default to `24h`; widen it to see whether something is new. "Since the last deploy" is since that container started (each deploys on its own): ```sh for c in iPX ipx-web ipx-directory; do S=$(docker inspect $c -f '{{.State.StartedAt}}'); echo "$c $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m"; done ``` Start every check by seeing that all three run, since Loki cannot see a container that is not: ```sh docker ps -a --filter name=iPX --filter name=ipx-web --filter name=ipx-directory --format '{{.Names}} {{.Status}}' ``` All three should be `Up`, with no recent restart; they have no Docker healthcheck (#218). Then work through ipx (the checks below), the page, the directory, Tower and the monitoring itself, in that order. ## ipx ### What the log carries Every line has `timestamp`, `level`, `message` and `target`; lines inside a span have `span` (the innermost: `{"name":"feed","feed":"x"}`). Loki's `| json` flattens it to `span_name`, `span_feed`. Lines inside a traced span, requests and scans, also carry `trace_id` and `span_id`: give the `trace_id` to `$Q trace` to see the whole request or scan. (From 2026-09-29 16:30 UTC; before that, lines had no trace id, the access log's time was `ms`, and each event was logged twice, words under `ipx::scan` and fields under `ipx::io`.) What is routine is logged at debug, which goes to the admin page's Log view and not to Loki: a request that went well (#364), a scan's end (`scan_done`, #363), a feed not due (`feed_skip`), a feed starting (`feed_start`), and a feed that brought nothing (`feed_done` with every count 0). Count those from the metrics, not the log. | target | fields | what | |---|---|---| | `ipx::http` | `method`, `path`, `route`, `status`, `duration_ms` | a warning for each request answered 4xx or 5xx; `route` is the pattern, empty for an unrouted path. Every request, as info, until #364; now those that went well are only in `ipx_http_request_duration_seconds` | | `ipx::scan` | `ev` and the event's own: `feed`, `new`, `downloaded`, `failed`, `bytes`, `msg`, `url`, `feeds`, `reason`, `from`, `to`; on a failure `error.type` and, from an HTTP error, `http.response.status_code` | the daemon's events, one line each, in words; warnings are feed and download failures | | `ipx::io` | `ev`, `feeds`, `pending`, `downloaded` on the `status` reply | commands arriving on the socket (`-> {...}`) and the answer to `ipx status` | | `ipx::daemon`, `ipx::cli`, `ipx::catalogue`, `ipx::scan`, ... | message, sometimes fields | start-up, shutdown, account and config messages, under the module that logs them; plain `ipx` before 2026-10-24, when main.rs was split (#356) | | `ipx::client` | `ev`, `user`, `feed`, `enclosure`, `deadair`; `msg`, `src`, `line` on an error; `ms` on a timing | what the page reports from the browser (#190): `page_error`, `rejection` and `player_error` as warnings; `play_start` (ms from play to sound), `stall` (ms waiting mid-play) and `list_shown` (ms from a click to its list drawn, with `feed` the view and `filter`, #246) as info. Lines carry the trace of the action in the page: its API requests send `traceparent`, one trace id per tap or key | Events (`ev`): `feed_start` (debug), `feed_done` (new, downloaded, failed, torrents), `feed_skip` (not due, debug), `feed_error` (msg), `feed_moved` (from, to: a permanent redirect followed, the address updated), `download_done` (bytes), `download_error` (msg, url), `torrent_deferred`, `reaped`, `reap_done`, `scan_done` (feeds; debug), `status` (feeds, pending, downloaded: the answer to `ipx status`, only when someone runs it since #218), `deadair_done` (feed, enclosure, `saved_s` at the default pause), `error` (msg). DeadAir also logs `DeadAir found the silences` (`cuts`, `saved_s`, `took_ms`) under `ipx::deadair`, and warns `DeadAir could not read a file` when ffmpeg fails on one, which it then passes over until the next restart. `error.type` is the HTTP status (`404`, `503`) or one of `dns`, `redirect_loop`, `timeout`, `tls`, `not_a_feed`, `site_message`, `connect`, `parse`, `other`; Loki's `| json` names it `error_type`. Filter on the text before `| json` where you can (`|= "\"ev\":\"feed_error\""`): it is much cheaper than parsing every line. Lines before 2026-09-29 14:00 UTC are text, not JSON, and `| json | __error__=""` drops them. ### What Prometheus carries The daemon's own state between requests, which no log line or trace holds (#194): ipx serves `/metrics` on `[web] metrics_bind` (production: `192.168.1.130:9187`, LAN only), on a thread and port of its own, from memory alone, and Prometheus scrapes it every 15 s as `job="ipx"`. Kept from 2026-10-11 15:15 UTC. Gauges, but for the counters (`_total`) and the one histogram: | metric | what | |---|---| | `ipx_db_connections{state="in_use"\|"idle"}`, `ipx_db_connections_max` | the database pool: in use reaching the max means requests and scans wait for a connection | | `ipx_worker_queue` | commands waiting for the sequential worker (scans, downloads, adds) | | `ipx_event_listeners` | open pages and socket clients on the live-update stream | | `ipx_torrent_slots_free` | of 2 | | `ipx_deadair_playing` | episodes being played through DeadAir's playlist | | `ipx_runtime_workers`, `ipx_runtime_tasks`, `ipx_runtime_queue` | tokio: threads, live tasks, tasks waiting for a thread | | `ipx_process_resident_bytes`, `ipx_process_open_fds` | memory and open files and sockets; mimalloc gives back what is freed, so memory follows use and a rise that never comes down is a leak (#224) | | `ipx_feed_fetches_total`, `ipx_feed_bytes_total` | counters: feeds fetched (304s included) and the bytes of their bodies, read whole, so what scans allocate | | `ipx_uptime_seconds`, `ipx_build_info{version}` | since the daemon started; what runs | | `ipx_worker_busy`, `ipx_worker_job_seconds{kind}` | the worker's job now (`scan`, `reap`, `download`, `status`, `reload`) and for how long; `ipx status` still answers while it is stuck | | `ipx_jobs_total{kind}`, `ipx_job_seconds_total{kind}` | counters: jobs finished, and their seconds all told | | `ipx_last_scan_age_seconds` | since a scan last finished; absent until one has since starting | | `ipx_http_requests_in_flight`, `ipx_scan_feeds_fetching`, `ipx_downloads_active` | work in progress | | `ipx_db_query_seconds{kind}`, `ipx_db_wait_seconds` | histograms (#367): each statement's time once it has a connection, by its first word (`select`, `insert`, ...); and the wait for a connection, measured for hand-written SQL only, about half the queries. A wait growing while queries stay flat is the pool full | | `ipx_page_seconds{ev}` | histogram: what the page timed, `list_shown`, `play_start`, `stall`, as the `ipx::client` lines say one by one (#369) | | `ipx_feed_failures_total{error_type}`, `ipx_new_items_total`, `ipx_downloads_total{result="saved"\|"failed"}`, `ipx_deadair_saved_seconds_total` | counters of what the events say (#368): feeds failed by kind, as the `feed_error` lines' `error.type`; new items; downloads; seconds of silence DeadAir found | | `ipx_http_request_duration_seconds{method,route,status}` | histogram: each request answered and its seconds, buckets 5 ms to 10 s; `_count` is requests, `_sum` their time. `route` is the pattern, `""` for an unrouted path; the event stream is left out (#364). From 2026-10-25 | | `ipx_download_bytes_total` | counter; `rate()` is the download speed | | `ipx_event_queue`, `ipx_events_missed_total` | live updates unread by some listener; counter of those dropped for one that fell behind | | `ipx_deadair_backlog`, `ipx_deadair_busy_seconds` | files of shows with DeadAir on not yet looked at (episodes everyone heard through stay counted); how long it has been on its file | | `ipx_feed_retries` | feeds waiting to retry a failed name lookup | | `ipx_art_cache_bytes`, `ipx_art_cache_limit_bytes` | artwork kept, as of the last sweep, and the most kept: kept sits at the limit by design | | `ipx_art_evicted_total`, `ipx_art_requests_total{result="hit"\|"miss"}` | counters: images the sweep threw out to stay under the limit, and artwork served from disk or fetched from the publisher; many evicted at every sweep with a falling hit rate means the limit is too small | | `ipx_signals_judged_total{user,key}`, `ipx_signals_hits_total{user,key}` | counters: items Jev judged for Signals, and Signals they fit, by person and whose key paid (`server` or `own`, #278) (#254) | | `ipx_jev_requests_total{result="ok"\|"key_refused"\|"malformed"\|"rate_limited"\|"error"}`, `ipx_jev_seconds_total`, `ipx_jev_paced_seconds_total` | counters: calls to TypeSafe's Jev, Signals' and icons', by how they ended; their seconds, the wait for a turn included; and that wait alone. `malformed` is ipx's request at fault (#257); `rate_limited` is a 429 from TypeSafe, whose limits are 80 calls and 100,000 input tokens a second, which ipx paces itself under at 60 and 80,000 (#277) | | `ipx_jev_tokens_total{kind="input"\|"output",user,key}` | counter: tokens TypeSafe counted, as its answers say, by person and whose key paid: what Signals, icon picks and key checks cost (#275, #278). A call shared on the server's key is split by each person's share of its questions | | `ipx_signals_last_pass_age_seconds`, `ipx_signals_keys_refused` | since the Signals judge last finished a pass (at most 5 minutes when well; absent until one has); people whose key TypeSafe refused | | `directory_process_resident_bytes`, `directory_process_open_fds`, `directory_feed_fetches_total`, `directory_feed_bytes_total`, `directory_uptime_seconds`, `directory_build_info`, `directory_http_request_duration_seconds{method,route,status}`, `directory_feed_checks_total`, `directory_feed_checks_failed_total`, `directory_snapshots_total` | the directory's (the last four #370), `job="ipx-directory"`, from its `metrics_bind` (production `192.168.1.130:9188`, LAN only) | | `ipx_feeds`, `ipx_downloads_pending`, `ipx_files_downloaded` | the library, as the daemon counts it every 30 s | ### The checks Run these, then read the lines behind whatever stands out. Most of the time is in the reading: a count says something happened, the lines and traces say why. 1. **Warnings and errors, grouped.** What went wrong, how often, and since when. ``` $Q metric 'sum by (target, message) (count_over_time({container="iPX"} | json | __error__="" | level=~"WARN|ERROR" [$range]))' ``` Feed and download failures name the feed in the message; group them in the next check instead. 2. **Failing feeds and downloads.** ``` $Q metric 'sum by (feed, error_type) (count_over_time({container="iPX"} |= "\"ev\":\"feed_error\"" | json | __error__="" [$range]))' 7d $Q metric 'sum by (feed, error_type) (count_over_time({container="iPX"} |= "\"ev\":\"download_error\"" | json | __error__="" [$range]))' 7d ``` Tell the publisher's problems from ipx's. A 404, 410, DNS failure or 503 from the feed's own server is the publisher (worth saying, since the feed may have moved; one issue for a feed that has been dead for days, not for a 503 once). A parse error on a feed that loads in a browser, a redirect loop ipx should follow, or the same failure on many feeds at once is ipx. 3. **Server errors and slow requests.** ``` $Q prom 'sum by (method, route, status) (increase(ipx_http_request_duration_seconds_count{job="ipx",status=~"4..|5.."}[$range])) > 0' 24h $Q prom 'topk(10, histogram_quantile(0.95, sum by (method, route, le) (rate(ipx_http_request_duration_seconds_bucket{job="ipx",route!=""}[$range]))))' 24h $Q logs '{container="iPX"} |= "\"target\":\"ipx::http\""' 24h ``` Any 5xx is worth a look; the log has each failed request's own line, with its trace id. 401s are people signing in, not a problem unless one address is hammering. The 95th percentile is read off the buckets, so it says which bucket, not the exact time: for a slow route, find its spans (check 5) and see which step holds the time. The histogram restarts at 0 with the daemon, which `increase` and `rate` allow for. 4. **Is the worker keeping up?** Scans should finish regularly, the queue should drain, and the daemon should not be restarting on its own. ``` $Q prom 'sum(increase(ipx_jobs_total{job="ipx",kind="scan"}[$range]))' 6h $Q prom 'max_over_time(ipx_downloads_pending{job="ipx"}[$range])' 6h $Q logs '{container="iPX"} |= "daemon started"' 7d ``` A `daemon started` not matched by a deploy (see `git log` and the image's build time) is a crash or an OOM kill: check `docker inspect iPX -f '{{.State.OOMKilled}} {{.RestartCount}}'` and the lines just before it. A pending count that only grows means downloads are not keeping up or not running. Since 2026-10-02 the daemon sleeps until the next feed is due (at most 10 minutes) instead of scanning every minute (#114), so expect a scan every few minutes, most of one or two feeds; none at all for over 10 minutes means the worker is stuck. A scan's end (`scan_done`) is logged at debug since #363, so it is in the page's log view and not in Loki. And `pending` is the real queue (#113): files a scan will download on its own. Back-catalogue files are `held`, listed but not counted, so it is usually 0 or a handful. 5. **Slow and failed traces.** Look at ipx's requests as spans, not traces: a trace started by a tap in the page also holds that page's live updates and any audio it streamed, minutes long, so a list of traces over a few seconds is nothing but those. `spans` lists each matching span with its own time and route, slowest first: ``` R='resource.deployment.environment.name="production" && resource.service.name="ipx"' $Q spans "{$R && span.http.route != \"\" && span.http.route != \"/api/events\" && span.http.route !~ \"/media/.*\" && duration > 1s} | select(span.http.route)" $Q spans "{$R && name=\"step\" && duration > 500ms} | select(span.of)" $Q traces '{resource.deployment.environment.name="production" && status = error}' $Q trace ``` Tempo keeps about two weeks. A trace id from a log line that Tempo answers 404 for, or a search that finds only the last quarter of an hour, is Tempo not reading its stored blocks: see the monitoring section. Group the first by hour and by route (`awk '{print substr($3,1,2), $NF}' | sort | uniq -c`), then lay the hours beside Tower's (the Tower section): load over 100 with processes blocked and memory under 4 GB is the host stalling, and every request of those minutes is slow with it, ipx's queries included. On 2026-10-18 that was 120 of 130 requests over a second, up to 43 s. What is left at quiet hours is ipx's own: open its trace and see which step holds the time. A request far longer than its steps added up waited outside them, for a connection or a thread. The second line finds steps over half a second wherever they ran; a step slow only once, first thing after hours idle, is Postgres reading from disk what it no longer had in memory (All Subscriptions' `entries_in`, a second where it is 60 ms warm). Scans (`scan`) are long by nature, since they fetch many feeds one after another: look for one `feed` or `fetch` span holding most of it (`$Q spans '{... && name="feed" && duration > 10s}'`), or a `download` far slower than its size explains. Steps are `step` spans named by `of` (#187): an item list's `entries_in` (with the list asked for: `filter`, `sort`, `dir`, `offset`, `limit`, `search`), `sanitize`, `deadair`, `count_in`; the feed list's `catalogue`, `feed_list` and its queries; a `reap`'s `missing_files`, `prune_abandoned_failures`, `reap_candidates`, `clean_directory` (once a day since #193, so a reap is about a second), `trim_artwork`, with what it removed on the `reap` span. DeadAir's work on a file is a `deadair` trace (feed, enclosure, cuts) with `probe`, `silencedetect`, `frame_bytes`, `store_plans`: seconds each, more is a file ffmpeg struggles with. A trace begun in the page (#190, #192) has the tap or key as its root span, `tap` or `key` (#376), from the moment it began to its last API answer, with the requests it made under it (API calls, the playlist and audio of a play, position saves, the page's own report): a tap long with its requests short is the page itself, drawing. The page reports the tap when the next one begins or it is hidden, so the root arrives last; a trace with no name in `$Q traces` is requests no tap began (a live update's, a timer's), or a tap not reported yet. Every command the worker runs from the socket, the CLI or the page is a `command` span (`cmd`, #373) with its job inside; a scan on the clock is a `scan` of its own. A list's sync is `list_sync` (`list`, `listed`) with steps `upsert_listed`, `subscribe_readers`, `drop_unlisted`, and an OPML upload `opml_import` (#372). The daemon's start is a `start` trace (#374), from `open_database` and `assemble_config` to `daemon started`: how long a deploy took to answer. 6. **The daemon's state, from Prometheus.** Pressure and leaks that requests and traces only hint at. ``` $Q prom 'up{job="ipx"}' $Q prom 'max(max_over_time(ipx_db_connections{job="ipx",state="in_use"}[$range]))' 24h $Q prom 'max(max_over_time(ipx_db_connections_max{job="ipx"}[$range]))' 24h $Q prom 'max(max_over_time(ipx_worker_queue{job="ipx"}[$range]))' 24h $Q prom 'max(max_over_time(ipx_runtime_queue{job="ipx"}[$range]))' 24h $Q prom 'max(resets(ipx_uptime_seconds{job="ipx"}[$range]))' 24h $Q prom 'ipx_process_resident_bytes{job="ipx"} - ipx_process_resident_bytes{job="ipx"} offset $range' 24h $Q prom 'ipx_process_open_fds{job="ipx"} - ipx_process_open_fds{job="ipx"} offset $range' 24h $Q prom 'ipx_runtime_tasks{job="ipx"} - ipx_runtime_tasks{job="ipx"} offset $range' 24h $Q prom 'max(max_over_time(ipx_worker_job_seconds{job="ipx"}[$range])) by (kind)' 24h $Q prom 'max(max_over_time(ipx_last_scan_age_seconds{job="ipx"}[$range]))' 24h $Q prom 'sum by (kind) (increase(ipx_job_seconds_total{job="ipx"}[$range]))' 24h $Q prom 'max(max_over_time(ipx_http_requests_in_flight{job="ipx"}[$range]))' 24h $Q prom 'histogram_quantile(0.99, sum by (le) (rate(ipx_db_wait_seconds_bucket{job="ipx"}[$range])))' 24h $Q prom 'histogram_quantile(0.95, sum by (le, kind) (rate(ipx_db_query_seconds_bucket{job="ipx"}[$range])))' 24h $Q prom 'sum(increase(ipx_events_missed_total{job="ipx"}[$range]))' 24h $Q prom 'max(max_over_time(ipx_deadair_busy_seconds{job="ipx"}[$range]))' 24h $Q prom 'sum by (result) (increase(ipx_jev_requests_total{job="ipx"}[$range]))' 24h $Q prom 'max(max_over_time(ipx_signals_last_pass_age_seconds{job="ipx"}[$range]))' 24h ``` `up` 0 means Prometheus cannot reach the port: the container is down or `metrics_bind` is unset. The daemon keeps all 20 connections open on Postgres (#241), so `idle` is the pool's 20 less those in use; in use at 20, idle at 0, means something is waiting for one: find the slow requests or scan of that time in check 3 and 5. A worker queue that only grows means downloads are not keeping up, or the worker is stuck on one job. A runtime queue above 0 for more than a scrape or two means every thread is busy: something is blocking a thread or burning one. Memory, open files or tasks only climbing across a day with no restart is a leak (a file or socket not closed, a task never ending); a step after a deploy is not. A reset of the uptime is a restart: match it with a deploy, or it is a crash (check 4). The worker: one job over 30 minutes (`ipx_worker_job_seconds`), or a scan last finished over 15 minutes ago, is the worker stuck, which `ipx status` does not show; its trace and the log lines of that time say on what. A kind whose share of `ipx_job_seconds_total` jumps is where the worker's time went (reap was most of it until #193). Web requests in flight near the pool's max mean the pool is what they wait on, and `ipx_db_wait_seconds` for how long. Live updates missed mean a page showed stale counts until reloaded. DeadAir over a minute on one file is a file ffmpeg chokes on. Any `rate_limited` Jev call means the pacing is not holding ipx under TypeSafe's limits (#277); each `jev` span has `paced_ms`, its wait for a turn. Any `malformed` Jev call is ipx asking wrongly (#257); `key_refused` is someone's key; the Signals judge over 10 minutes since a pass is stuck on Jev: its `signals` trace shows which `jev` call. 7. **What the page saw.** The browser's own errors and timings, which no server line shows. ``` $Q metric 'sum by (ev, msg) (count_over_time({container="iPX"} |= "\"target\":\"ipx::client\"" | json | __error__="" | level="WARN" [$range]))' 7d $Q logs '{container="iPX"} |= "\"target\":\"ipx::client\"" | json | ev=~"play_start|stall" | line_format "{{.timestamp}} {{.user}} {{.ev}} {{.ms}}ms deadair={{.deadair}} {{.feed}}"' 24h $Q metric 'quantile_over_time(0.95, {container="iPX"} |= "\"ev\":\"list_shown\"" | json | __error__="" | unwrap ms [$range]) by (feed, filter)' 24h ``` A `player_error` is playback that failed in front of someone: `hls.js: ...` is DeadAir's playlist (its trace has the playlist and audio requests), `media error 4` a file the browser cannot play. A `page_error` or `rejection` is a bug in the page; `src` and `line` point into the minified `app.js`, so match the message to `web/src`. A `list_shown` over 1000 ms is a list someone waited for: its trace says whether the API or the network held it. `play_start` over 2000 ms, or a `stall`, is worth a trace: was it the server, Cloudflare, or the network. `docker logs --since 10m iPX 2>&1 | tail -5` shows the newest lines when Loki is behind. ## ipx-web, the page Caddy logs one JSON line a request (#210), logger `http.log.access.log0`: Loki's `| json` gives `request_uri`, `request_method`, `request_remote_ip`, `status`, `duration` (seconds, not ms), `size`. The request's headers and a `?token=` are filtered out before it is written. Requests from `127.0.0.1` are local checks; leave them out. `/api` and `/media` pass on to ipx, which counts them in `ipx_http_request_duration_seconds` with its own time; Caddy's `duration` for them adds the hop. This is the only log with a line for every request. Caddy's own metrics, `job="ipx-web"` (`caddy_http_request_duration_seconds{code,handler,method}` and the like), count every request too, 502s included, which ipx never sees. ```sh W='{container="ipx-web"} |= "http.log.access" | json | __error__="" | request_remote_ip != "127.0.0.1"' $Q metric "sum by (status) (count_over_time($W [\$range]))" $Q metric "sum by (request_uri) (count_over_time($W | status >= 500 [\$range]))" $Q metric "topk(10, quantile_over_time(0.95, $W | request_uri !~ \"/(api|media)/.*\" | unwrap duration [\$range]) by (request_uri))" $Q logs '{container="ipx-web"} | json | __error__="" | level=~"warn|error" | msg !~ "admin endpoint disabled|automatic HTTPS is completely disabled|HTTP/[23] skipped.*|exiting; byeee.*" | request_uri != "/api/events"' ``` A 502 is ipx not answering Caddy: a restart (match the deploy) or ipx down; Caddy's error line names it (`dial tcp ...:8099: connect: connection refused`). The page itself (`/`, `/admin`) waits on `/api/page-check` before it is sent, so a slow page is usually a slow page-check: its route in the histogram (check 3), or ipx's span inside Caddy's in the trace. 401 and 303 on `/` are people signed out being sent to `/login`, not a problem. Caddy's own notices at each start (no admin endpoint, no HTTPS or HTTP/3, its goodbye on SIGTERM) are by design and filtered out above, and so is `aborting with incomplete response` on `/api/events`: a page closing or reconnecting its live-update stream, about 600 a week. Each access line carries the request's `traceID` (`| json` gives `traceID`): give it to `$Q trace` to see where the time went, Caddy's span with ipx's inside it. A slow page with ipx's span short is Caddy or the network; a long one is ipx. ```sh $Q spans '{resource.service.name="ipx-web" && span.url.path != "/api/events" && span.url.path !~ "/media/.*" && duration > 2s} | select(span.url.path)' $Q traces '{resource.service.name="ipx-web" && status = error}' ``` Who reaches it: `request_remote_ip` is `192.168.64.1` for the tunnel and anything else on Tower (Docker's gateway), `192.168.1.130` for Tower itself, a LAN address for the LAN. Caddy passes Cloudflare Access's headers on only from `192.168.64.1` (`TUNNEL_PEER`), and ipx trusts them only from Caddy, with a valid signed token. Testing sign-in from Tower proves nothing about the tunnel (#209): send it from a container on another network. ## ipx-directory It logs a line a request (`target` `ipx_directory::http`: `method`, `path`, `status`, `duration_ms`, `token`, the token's name and never the token), a line a snapshot (`message` `snapshot`: `instance`, `accepted`, `refused_private`, `refused_blocked`, `refused_invalid`), a line a refresher round (`feeds checked`: `checked`), a line a failed feed check (`feed check failed`: `feed`, the directory's id, and `error`), the admin's actions (`feed removed and blocked`, `feed unblocked`, `snapshot dropped`, each with `by`), and `config read again` when its token file changes. Its lines carry their own fields and no span (#363). Its metrics, as `job="ipx-directory"` (production `192.168.1.130:9188`): `directory_process_resident_bytes`, `directory_process_open_fds`, `directory_feed_fetches_total`, `directory_feed_bytes_total`, `directory_uptime_seconds`, `directory_build_info`. Its traces (#213): a span a request (`http`), a child of ipx's when ipx called with a `traceparent`, and a trace a refresher round (`refresh`, `feeds`) with a `feed` span for each check. Its lines inside a span carry `trace_id` and `span_id`, as ipx's do (#375): a `feed check failed` leads to its check's span. ```sh $Q traces '{resource.service.name="ipx-directory" && duration > 2s}' $Q traces '{resource.service.name="ipx-directory" && name="feed" && duration > 10s}' $Q traces '{resource.service.name="ipx" && name="directory_share"}' ``` A slow Directory page in ipx is usually its `http` span in the directory: the trace says whether. A refresher round far over a minute is one publisher's feed holding it, the slow `feed` span. ```sh D='{container="ipx-directory"} | json | __error__=""' $Q metric "sum by (status, token) (count_over_time({container=\"ipx-directory\"} |= \"ipx_directory::http\" | json | __error__=\"\" [\$range]))" $Q metric "topk(5, max_over_time({container=\"ipx-directory\"} |= \"ipx_directory::http\" | json | __error__=\"\" | unwrap duration_ms [\$range]) by (path))" $Q logs "$D | level=~\"WARN|ERROR\"" $Q logs "$D | message=\"snapshot\"" 7d $Q metric "sum(count_over_time($D | message=\"snapshot\" [\$range])) by (instance)" $Q metric "sum by (feed, error) (count_over_time($D | message=\"feed check failed\" [\$range]))" 7d $Q metric "sum(sum_over_time($D | message=\"feeds checked\" | unwrap checked [\$range]))" 24h ``` Expect a snapshot from each server when what it shares changes and every half hour, so about 48 a day each; one every half minute is ipx resending what has not changed (#211, fixed). A burst of 401s with an empty `token` is someone without a token trying the public address, expected now and then; a steady stream is worth an issue, since Cloudflare can rate-limit it. A snapshot whose `refused_private` is not 0 means an ipx server tried to send a feed that looks private: find which in that server's log, since the directory refuses it but the sender should not have sent it. Many `feed check failed` lines for one feed are the publisher's, not the directory's. Each feed is checked every twelve hours (#243), 25 at a time, so `feeds checked` over a day should be about twice the feeds listed; none for over twelve hours means the refresher is stuck. `could not store a feed check` or `could not read which feeds are due` is its database (`ipx_directory` on the `databases` project's Postgres). A config that does not load (`config changed but does not load`) keeps the old tokens: worth fixing before someone's new token is needed. Weekly, `Apple Podcasts synced` (`shows`, `accepted`) brings Apple's charts in as the `apple-podcasts` snapshot (#252); `Apple Podcasts' sync failed` retries in an hour, worth an issue when it keeps failing, and `Apple's podcast categories are not ipx's` (`added`, `gone`) is an issue to update `CATEGORIES` in `crates/feed`. ## Tower The three share Tower with everything else, so a slow or failing ipx can be the host. The node exporter (`job="node"`) has the history; twelve threads. ```sh $Q prom 'max_over_time(node_load1{job="node"}[$range])' 24h $Q prom 'min_over_time(node_memory_MemAvailable_bytes{job="node"}[$range]) / 1e9' 24h $Q prom 'sum by (mode) (rate(node_cpu_seconds_total{job="node", mode=~"user|system|iowait"}[$range]))' 1h $Q prom 'max by (device) (rate(node_disk_io_time_seconds_total{job="node", device=~"nvme.*|md.*"}[$range]))' 1h ``` For when, step back with `offset`: `max_over_time(node_load1[10m] offset 60m)`. Load far above twelve with little CPU busy is processes waiting, on a disk or on the user shares: on 2026-10-12 debug builds linking through `/mnt/user` (shfs, FUSE) took load to 248 with mostly system time and stalled everything on the shares, ipx's downloads included. Available memory near nothing makes it worse. Disk busy near 100% on `nvme1n1` is `/mnt/fast`, appdata and Postgres's data; on `md*` the array behind `/mnt/user`, the downloads. Say what was running at the time if you can tell (`docker stats --no-stream`); a host problem is the user's, not an ipx issue, unless ipx caused it. ## The monitoring itself A quiet dashboard can be a dead exporter. Check what Prometheus scrapes and that each container's lines are reaching Loki: ```sh curl -s http://192.168.1.130:9090/api/v1/targets | python3 -c 'import json,sys for t in json.load(sys.stdin)["data"]["activeTargets"]: print(t["health"], t["labels"]["job"], t["scrapeUrl"], t.get("lastError","")[:80])' for c in iPX ipx-web ipx-directory; do echo "$c $($Q metric "sum(count_over_time({container=\"$c\"} [24h]))" 24h | tail -1)"; done docker ps -a --filter label=com.docker.compose.project=monitoring --format '{{.Names}} {{.Status}}' docker logs --since 1h tempo 2>&1 | grep -E 'failed to poll|blocklist poll complete' | tail -2 ``` `ipx`, `ipx-directory`, `ipx-web`, `node` and the monitoring's own, `prometheus`, `tempo`, `loki` and `alloy` (#366), should be `up`. Alloy is scraped at `alloy:12345` because its command sets `--server.http.listen-addr`; it listens on 127.0.0.1 without it. The alert rules (#365) are `grafana/alerts.py`, written to `/mnt/fast/arcane/projects/monitoring/grafana-provisioning/alerting/ipx.json` and loaded by `POST /api/admin/provisioning/alerting/reload`. There is no contact point yet, so one firing is only in Grafana: ask it which are, at the start of every check. The password is in the monitoring project's compose file; never print it. ```sh PW=$(grep -oP 'GF_SECURITY_ADMIN_PASSWORD=\K\S+' /mnt/fast/arcane/projects/monitoring/compose.yaml) curl -s -u "admin:$PW" http://192.168.1.130:3444/api/prometheus/grafana/api/v1/rules | python3 -c 'import json,sys for g in json.load(sys.stdin)["data"]["groups"]: for r in g["rules"]: print(r["state"], r["name"])' | sort ``` Tempo looks over its stored blocks every 5 minutes. `failed to poll or create index for tenant` means it could not, and then searches and trace lookups see only what it still holds in memory, about the last 15 minutes, while its 2,000 blocks sit unread on disk (its search log says `total_blocks=0`). On 2026-10-25 that was one block left empty, every file 0 bytes, by a write cut short the night before: `unexpected end of JSON input`. Find it and move it out of `/mnt/user/logs/tempo/blocks/single-tenant/`; the next look over, within 5 minutes, logs `writing tenant index` and finds the rest. Tell the user, since it is the monitoring's, not ipx's: ```sh for d in /mnt/user/logs/tempo/blocks/single-tenant/*/; do python3 -c 'import json,sys; json.load(open(sys.argv[1]))' "$d/meta.json" 2>/dev/null || echo "$d"; done ``` Every monitoring container should be running; after Docker restarts, one can be left `Exited (255)` though its policy is `unless-stopped` (Prometheus, on 2026-10-12, when Docker lost its exit during an overload): `docker start` it and say so. blocky and nut on 192.168.2.21 are another machine, not ipx's; leave them. A container with no lines in an hour while it runs means Alloy is not shipping it; Alloy can take a few minutes to pick up a container after a deploy. ## Rules `docker logs` and `docker exec iPX ipx ...` are fine. On production's Postgres, `EXPLAIN (ANALYZE, BUFFERS)` of any query is allowed, read only, as CLAUDE.md's "Production's database" shows; any other query, ask the user first. ## What to do with what you find Follow the repository's rules in CLAUDE.md: **every problem found gets a Gitea issue**, with a closed stdin and a timeout on `tea`. Before filing, list the open issues and do not file one twice; comment on the existing issue with the new evidence instead. ```sh R="--login git.sdf1.net --repo rays/ipx" t() { timeout 30 /src/tea "$@" < /dev/null; } t issues list $R --state open t issues create $R -t "" -L bug -d "" ``` Put in each issue what would let someone pick it up cold: the LogQL or TraceQL that shows it, a trace id, the first time it was seen and how often. A publisher's dead feed is worth one issue saying so (the user may want to unsubscribe or find its new address); a single 503 is not. Finish with a short report to the user, a line for each of ipx-web, ipx, the directory, Tower and the monitoring: what is healthy, what is wrong (with the issue numbers), and anything you could not tell from logs, traces and metrics alone. Do not fix things unless asked; the check is for finding them. ## When the checks come back empty Check that there is data before concluding all is well (the monitoring section's per-container count over a day): iPX a few hundred (new items, Signals, shares with the directory, the page's timings; more on a busy day), ipx-web two thousand or so while the page is in use (a line a request), ipx-directory several hundred. Since #363 and #364 iPX can go an hour at night with no line at all, so judge it by the day. Nothing in a day means Alloy is not shipping it, or the container is down.