ipx/.claude/skills/ipx-prod-check/SKILL.md

name, description

name

description

ipx-prod-check

Look for problems in production ipx (the ipx-web, iPX and ipx-directory containers on Tower, Tower itself and the monitoring that watches them) from their logs in Loki, ipx's traces in Tempo and the metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site.

Checking production ipx

Production is three containers in the Arcane project ipx on Tower (/mnt/fast/arcane/projects/ipx/compose.yaml), one request passing through them in this order:

container	what	logs, in Loki	traces, in Tempo	metrics, in Prometheus
ipx-web	Caddy: serves the page, passes /api and /media to ipx (#203). Port 8099, the LAN and the tunnel	{container="ipx-web"}, JSON: a line a request (http.log.access), its own errors	service ipx-web: a span a request, passing traceparent on to ipx (#213)	none
iPX	the daemon: API, scans, downloads, DeadAir	{container="iPX"}, JSON (IPX_LOG_FORMAT=json)	service ipx; passes traceparent to the directory	job="ipx", its /metrics
ipx-directory	the directory of public podcasts, ipx-dir.sdf1.net (#204)	{container="ipx-directory"}, JSON (DIRECTORY_LOG_FORMAT=json)	service ipx-directory: a span a request, joining ipx's trace when ipx called; a trace a refresher round (#213)	none

Under them, Tower itself: job="node" (the node exporter: load, CPU, memory, disks). The Grafana dashboard iPX draws all of it in that order (grafana/dashboard.py). A trace without the production tag (resource.deployment.environment.name="production", on all three) is a daemon run by hand, not production. Since #213 a request is one trace across the services it touched: a Directory page is ipx-web, then ipx, then the directory, so start from the trace, not the logs. Alloy ships the logs; Loki, Tempo and Prometheus publish no query port to this machine, so use the script beside this file:

Q=.claude/skills/ipx-prod-check/query.py
$Q logs    '<LogQL>'           [since]   # lines, newest first (500 at most)
$Q metric  '<LogQL metric>'    [since]   # $range becomes `since`
$Q traces  '<TraceQL>'         [since]   # slowest first
$Q trace   <trace id>                    # one trace as a tree, with its log lines
$Q prom    '<PromQL>'          [since]   # Prometheus, now; $range becomes `since`

since is 30m, 1h, 24h, 7d. Default to 24h; widen it to see whether something is new. "Since the last deploy" is since that container started (each deploys on its own):

for c in iPX ipx-web ipx-directory; do S=$(docker inspect $c -f '{{.State.StartedAt}}'); echo "$c $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m"; done

Start every check by seeing that all three run, since Loki cannot see a container that is not:

docker ps -a --filter name=iPX --filter name=ipx-web --filter name=ipx-directory --format '{{.Names}} {{.Status}}'

All three should be (healthy). Then work through ipx (the checks below), the page, the directory, Tower and the monitoring itself, in that order.

ipx

What the log carries

Every line has timestamp, level, message and target; lines inside a span have span (the innermost: {"name":"feed","feed":"x"}). Loki's | json flattens it to span_name, span_feed. Lines inside a traced span, requests and scans, also carry trace_id and span_id: give the trace_id to $Q trace to see the whole request or scan. (From 2026-09-29 16:30 UTC; before that, lines had no trace id, the access log's time was ms, and each event was logged twice, words under ipx::scan and fields under ipx::io.)

target	fields	what
ipx::http	method, path, route, status, duration_ms	one per web request; route is the pattern, empty for an unrouted path
ipx::scan	ev and the event's own: feed, new, downloaded, failed, bytes, msg, url, feeds, reason, from, to; on a failure error.type and, from an HTTP error, http.response.status_code	the daemon's events, one line each, in words; warnings are feed and download failures
ipx::io	ev, feeds, pending, downloaded on the status reply	commands arriving (-> {...}) and the healthcheck's answer
ipx	message, sometimes fields	start-up, shutdown, account and config messages
ipx::client	ev, user, feed, enclosure, deadair; msg, src, line on an error; ms on a timing	what the page reports from the browser (#190): page_error, rejection and player_error as warnings; play_start (ms from play to sound) and stall (ms waiting mid-play) as info. Lines carry the trace of the action in the page: its API requests send traceparent, one trace id per tap or key

Events (ev): feed_start, feed_done (new, downloaded, failed, torrents), feed_skip (not due, routine), feed_error (msg), feed_moved (from, to: a permanent redirect followed, the address updated), download_done (bytes), download_error (msg, url), torrent_deferred, reaped, scan_done (feeds checked), reap_done, status (feeds, pending, downloaded: the healthcheck's, every 30s), deadair_done (feed, enclosure, saved_s at the default pause), error (msg). DeadAir also logs DeadAir found the silences (cuts, saved_s, took_ms) under ipx::deadair, and warns DeadAir could not read a file when ffmpeg fails on one, which it then passes over until the next restart.

error.type is the HTTP status (404, 503) or one of dns, redirect_loop, timeout, tls, not_a_feed, site_message, connect, parse, other; Loki's | json names it error_type.

Filter on the text before | json where you can (|= "\"ev\":\"feed_error\""): it is much cheaper than parsing every line. Lines before 2026-09-29 14:00 UTC are text, not JSON, and | json | __error__="" drops them.

What Prometheus carries

The daemon's own state between requests, which no log line or trace holds (#194): ipx serves /metrics on [web] metrics_bind (production: 192.168.1.130:9187, LAN only), on a thread and port of its own, from memory alone, and Prometheus scrapes it every 15 s as job="ipx". Kept from 2026-10-11 15:15 UTC. All gauges:

metric	what
ipx_db_connections{state="in_use"|"idle"}, ipx_db_connections_max	the database pool: in use reaching the max means requests and scans wait for a connection
ipx_worker_queue	commands waiting for the sequential worker (scans, downloads, adds)
ipx_event_listeners	open pages and socket clients on the live-update stream
ipx_torrent_slots_free	of 2
ipx_deadair_playing	episodes being played through DeadAir's playlist
ipx_runtime_workers, ipx_runtime_tasks, ipx_runtime_queue	tokio: threads, live tasks, tasks waiting for a thread
ipx_process_resident_bytes, ipx_process_open_fds	memory and open files and sockets
ipx_uptime_seconds, ipx_build_info{version}	since the daemon started; what runs
ipx_worker_busy, ipx_worker_job_seconds{kind}	the worker's job now (scan, reap, download, status, reload) and for how long; the healthcheck passes while it is stuck
ipx_jobs_total{kind}, ipx_job_seconds_total{kind}	counters: jobs finished, and their seconds all told
ipx_last_scan_age_seconds	since a scan last finished; absent until one has since starting
ipx_http_requests_in_flight, ipx_scan_feeds_fetching, ipx_downloads_active	work in progress
ipx_download_bytes_total	counter; rate() is the download speed
ipx_event_queue, ipx_events_missed_total	live updates unread by some listener; counter of those dropped for one that fell behind
ipx_deadair_backlog, ipx_deadair_busy_seconds	files of shows with DeadAir on not yet looked at (episodes everyone heard through stay counted); how long it has been on its file
ipx_feed_retries	feeds waiting to retry a failed name lookup
ipx_art_cache_bytes, ipx_art_cache_limit_bytes	artwork kept, as of the last sweep, and the most kept: kept sits at the limit by design
ipx_art_evicted_total, ipx_art_requests_total{result="hit"|"miss"}	counters: images the sweep threw out to stay under the limit, and artwork served from disk or fetched from the publisher; many evicted at every sweep with a falling hit rate means the limit is too small
ipx_feeds, ipx_downloads_pending, ipx_files_downloaded	the library, as the healthcheck last counted it (every 30 s)

The checks

Run these, then read the lines behind whatever stands out. Most of the time is in the reading: a count says something happened, the lines and traces say why.

1. Warnings and errors, grouped. What went wrong, how often, and since when.

   $Q metric 'sum by (target, message) (count_over_time({container="iPX"} | json | __error__="" | level=~"WARN|ERROR" [$range]))'
   

   Feed and download failures name the feed in the message; group them in the next check instead.

2. Failing feeds and downloads.

   $Q metric 'sum by (feed, error_type) (count_over_time({container="iPX"} |= "\"ev\":\"feed_error\"" | json | __error__="" [$range]))' 7d
   $Q metric 'sum by (feed, error_type) (count_over_time({container="iPX"} |= "\"ev\":\"download_error\"" | json | __error__="" [$range]))' 7d
   

   Tell the publisher's problems from ipx's. A 404, 410, DNS failure or 503 from the feed's own server is the publisher (worth saying, since the feed may have moved; one issue for a feed that has been dead for days, not for a 503 once). A parse error on a feed that loads in a browser, a redirect loop ipx should follow, or the same failure on many feeds at once is ipx.

3. Server errors and slow requests.

   $Q metric 'sum by (method, route, status) (count_over_time({container="iPX"} |= "\"target\":\"ipx::http\"" | json | __error__="" | status >= 500 [$range]))'
   $Q metric 'topk(10, quantile_over_time(0.95, {container="iPX"} |= "\"target\":\"ipx::http\"" | json | __error__="" | route != "" | route != "/api/events" | unwrap duration_ms [$range]) by (method, route))'
   

   Any 5xx is worth a look. 401s are people signing in, not a problem unless one address is hammering. For a slow route, find its traces (check 5) and see which span holds the time.

4. Is the worker keeping up? Scans should finish regularly, the queue should drain, and the daemon should not be restarting on its own.

   $Q metric 'sum(count_over_time({container="iPX"} |= "\"ev\":\"scan_done\"" [$range]))' 6h
   $Q logs '{container="iPX"} |= "\"ev\":\"status\"" | json | line_format "{{.timestamp}} pending={{.pending}} downloaded={{.downloaded}}"' 6h
   $Q logs '{container="iPX"} |= "daemon started"' 7d
   

   A daemon started not matched by a deploy (see git log and the image's build time) is a crash or an OOM kill: check docker inspect iPX -f '{{.State.OOMKilled}} {{.RestartCount}}' and the lines just before it. A pending count that only grows means downloads are not keeping up or not running.

   Since 2026-10-02 the daemon sleeps until the next feed is due (at most 10 minutes) instead of scanning every minute (#114), so expect tens of scans in 6 hours, not 360, nearly all with feeds above 0; none at all for over 10 minutes means the worker is stuck. And pending is the real queue (#113): files a scan will download on its own. Back-catalogue files are held, listed but not counted, so it is usually 0 or a handful.

5. Slow and failed traces. All three services' (filter by resource.service.name for one).

   $Q traces '{resource.deployment.environment.name="production" && duration > 5s}'
   $Q traces '{resource.deployment.environment.name="production" && status = error}'
   $Q trace <id>
   

   Scans (scan) are long by nature, since they fetch many feeds one after another: look for one feed or fetch span holding most of it, or a download far slower than its size explains. A web request over a second is worth a look; the trace shows whether the time is in the handler or a scan it waited on.

   Steps are step spans named by of (#187): an item list's entries_in, sanitize, deadair, count_in; the feed list's catalogue, feed_list and its queries; a reap's missing_files, prune_abandoned_failures, reap_candidates, clean_directory (once a day since #193, so a reap is about a second), trim_artwork, with what it removed on the reap span. DeadAir's work on a file is a deadair trace (feed, enclosure, cuts) with probe, silencedetect, frame_bytes, store_plans: seconds each, more is a file ffmpeg struggles with. A trace with no name in $Q traces started in the page (#190, #192): its root is the tap in the browser, which sends no span, so it holds the requests that tap made (API calls, the playlist and audio of a play, position saves, the page's own report) side by side.

6. The daemon's state, from Prometheus. Pressure and leaks that requests and traces only hint at.

   $Q prom 'up{job="ipx"}'
   $Q prom 'max(max_over_time(ipx_db_connections{job="ipx",state="in_use"}[$range]))' 24h
   $Q prom 'max(max_over_time(ipx_db_connections_max{job="ipx"}[$range]))' 24h
   $Q prom 'max(max_over_time(ipx_worker_queue{job="ipx"}[$range]))' 24h
   $Q prom 'max(max_over_time(ipx_runtime_queue{job="ipx"}[$range]))' 24h
   $Q prom 'max(resets(ipx_uptime_seconds{job="ipx"}[$range]))' 24h
   $Q prom 'ipx_process_resident_bytes{job="ipx"} - ipx_process_resident_bytes{job="ipx"} offset $range' 24h
   $Q prom 'ipx_process_open_fds{job="ipx"} - ipx_process_open_fds{job="ipx"} offset $range' 24h
   $Q prom 'ipx_runtime_tasks{job="ipx"} - ipx_runtime_tasks{job="ipx"} offset $range' 24h
   $Q prom 'max(max_over_time(ipx_worker_job_seconds{job="ipx"}[$range])) by (kind)' 24h
   $Q prom 'max(max_over_time(ipx_last_scan_age_seconds{job="ipx"}[$range]))' 24h
   $Q prom 'sum by (kind) (increase(ipx_job_seconds_total{job="ipx"}[$range]))' 24h
   $Q prom 'max(max_over_time(ipx_http_requests_in_flight{job="ipx"}[$range]))' 24h
   $Q prom 'sum(increase(ipx_events_missed_total{job="ipx"}[$range]))' 24h
   $Q prom 'max(max_over_time(ipx_deadair_busy_seconds{job="ipx"}[$range]))' 24h
   

   up 0 means Prometheus cannot reach the port: the container is down or metrics_bind is unset. Connections in use at the pool's max (20 on Postgres) means something is waiting for one: find the slow requests or scan of that time in check 3 and 5. A worker queue that only grows means downloads are not keeping up, or the worker is stuck on one job. A runtime queue above 0 for more than a scrape or two means every thread is busy: something is blocking a thread or burning one. Memory, open files or tasks only climbing across a day with no restart is a leak (a file or socket not closed, a task never ending); a step after a deploy is not. A reset of the uptime is a restart: match it with a deploy, or it is a crash (check 4).

   The worker: one job over 30 minutes (ipx_worker_job_seconds), or a scan last finished over 15 minutes ago, is the worker stuck, which the healthcheck does not notice; its trace and the log lines of that time say on what. A kind whose share of ipx_job_seconds_total jumps is where the worker's time went (reap was most of it until #193). Web requests in flight near the pool's max mean the pool is what they wait on. Live updates missed mean a page showed stale counts until reloaded. DeadAir over a minute on one file is a file ffmpeg chokes on.

7. What the page saw. The browser's own errors and timings, which no server line shows.

   $Q metric 'sum by (ev, msg) (count_over_time({container="iPX"} |= "\"target\":\"ipx::client\"" | json | __error__="" | level="WARN" [$range]))' 7d
   $Q logs '{container="iPX"} |= "\"target\":\"ipx::client\"" | json | ev=~"play_start|stall" | line_format "{{.timestamp}} {{.user}} {{.ev}} {{.ms}}ms deadair={{.deadair}} {{.feed}}"' 24h
   

   A player_error is playback that failed in front of someone: hls.js: ... is DeadAir's playlist (its trace has the playlist and audio requests), media error 4 a file the browser cannot play. A page_error or rejection is a bug in the page; src and line point into the minified app.js, so match the message to web/src. play_start over 2000 ms, or a stall, is worth a trace: was it the server, Cloudflare, or the network.

docker logs --since 10m iPX 2>&1 | tail -5 shows the newest lines when Loki is behind.

ipx-web, the page

Caddy logs one JSON line a request (#210), logger http.log.access.log0: Loki's | json gives request_uri, request_method, request_remote_ip, status, duration (seconds, not ms), size. The request's headers and a ?token= are filtered out before it is written. Its healthcheck requests come from 127.0.0.1 every 30 s; leave them out. /api and /media pass on to ipx and are in ipx's own access log too, with ipx's time; Caddy's duration for them adds the hop.

W='{container="ipx-web"} |= "http.log.access" | json | __error__="" | request_remote_ip != "127.0.0.1"'
$Q metric "sum by (status) (count_over_time($W [\$range]))"
$Q metric "sum by (request_uri) (count_over_time($W | status >= 500 [\$range]))"
$Q metric "topk(10, quantile_over_time(0.95, $W | request_uri !~ \"/(api|media)/.*\" | unwrap duration [\$range]) by (request_uri))"
$Q logs '{container="ipx-web"} | json | __error__="" | level=~"warn|error" | msg !~ "admin endpoint disabled|automatic HTTPS is completely disabled|HTTP/[23] skipped.*|exiting; byeee.*"'

A 502 is ipx not answering Caddy: a restart (match the deploy) or ipx down; Caddy's error line names it (dial tcp ...:8099: connect: connection refused). The page itself (/, /admin) waits on /api/page-check before it is sent, so a slow page is usually a slow page-check: look it up in ipx's access log. 401 and 303 on / are people signed out being sent to /login, not a problem. Caddy's own notices at each start (no admin endpoint, no HTTPS or HTTP/3, its goodbye on SIGTERM) are by design and filtered out above.

Each access line carries the request's traceID (| json gives traceID): give it to $Q trace to see where the time went, Caddy's span with ipx's inside it. A slow page with ipx's span short is Caddy or the network; a long one is ipx.

$Q traces '{resource.service.name="ipx-web" && duration > 2s}'
$Q traces '{resource.service.name="ipx-web" && status = error}'

Who reaches it: request_remote_ip is 192.168.64.1 for the tunnel and anything else on Tower (Docker's gateway), 192.168.1.130 for Tower itself, a LAN address for the LAN. Caddy passes Cloudflare Access's headers on only from 192.168.64.1 (TUNNEL_PEER), and ipx trusts them only from Caddy, with a valid signed token. Testing sign-in from Tower proves nothing about the tunnel (#209): send it from a container on another network.

ipx-directory

It logs a line a request (target ipx_directory::http: method, path, status, duration_ms, token, the token's name and never the token), a line a snapshot (message snapshot: instance, accepted, refused_private, refused_blocked, refused_invalid), a line a refresher round (feeds checked: checked), a line a failed feed check (feed check failed: feed, the directory's id, and error), the admin's actions (feed removed and blocked, feed unblocked, snapshot dropped, each with by), and config read again when its token file changes. It has no metrics. Its traces (#213): a span a request (http), a child of ipx's when ipx called with a traceparent, and a trace a refresher round (refresh, feeds) with a feed span for each check. Its log lines carry no trace id; find the trace by time or by the feed.

$Q traces '{resource.service.name="ipx-directory" && duration > 2s}'
$Q traces '{resource.service.name="ipx-directory" && name="feed" && duration > 10s}'
$Q traces '{resource.service.name="ipx" && name="directory_share"}'

A slow Directory page in ipx is usually its http span in the directory: the trace says whether. A refresher round far over a minute is one publisher's feed holding it, the slow feed span.

D='{container="ipx-directory"} | json | __error__=""'
$Q metric "sum by (status, token) (count_over_time({container=\"ipx-directory\"} |= \"ipx_directory::http\" | json | __error__=\"\" [\$range]))"
$Q metric "topk(5, max_over_time({container=\"ipx-directory\"} |= \"ipx_directory::http\" | json | __error__=\"\" | unwrap duration_ms [\$range]) by (path))"
$Q logs "$D | level=~\"WARN|ERROR\""
$Q logs "$D | message=\"snapshot\"" 7d
$Q metric "sum(count_over_time($D | message=\"snapshot\" [\$range])) by (instance)"
$Q metric "sum by (feed, error) (count_over_time($D | message=\"feed check failed\" [\$range]))" 7d
$Q metric "sum(sum_over_time($D | message=\"feeds checked\" | unwrap checked [\$range]))" 24h

Expect a snapshot from each server when what it shares changes and every half hour, so about 48 a day each; one every half minute is ipx resending what has not changed (#211, fixed). A burst of 401s with an empty token is someone without a token trying the public address, expected now and then; a steady stream is worth an issue, since Cloudflare can rate-limit it. A snapshot whose refused_private is not 0 means an ipx server tried to send a feed that looks private: find which in that server's log, since the directory refuses it but the sender should not have sent it. Many feed check failed lines for one feed are the publisher's, not the directory's. Each feed is checked every six hours, 25 at a time, so feeds checked over a day should be about four times the feeds listed; none for over six hours means the refresher is stuck. could not store a feed check or could not read which feeds are due is its database (ipx_directory on the databases project's Postgres). A config that does not load (config changed but does not load) keeps the old tokens: worth fixing before someone's new token is needed.

Tower

The three share Tower with everything else, so a slow or failing ipx can be the host. The node exporter (job="node") has the history; twelve threads.

$Q prom 'max_over_time(node_load1{job="node"}[$range])' 24h
$Q prom 'min_over_time(node_memory_MemAvailable_bytes{job="node"}[$range]) / 1e9' 24h
$Q prom 'sum by (mode) (rate(node_cpu_seconds_total{job="node", mode=~"user|system|iowait"}[$range]))' 1h
$Q prom 'max by (device) (rate(node_disk_io_time_seconds_total{job="node", device=~"nvme.*|md.*"}[$range]))' 1h

For when, step back with offset: max_over_time(node_load1[10m] offset 60m). Load far above twelve with little CPU busy is processes waiting, on a disk or on the user shares: on 2026-10-12 debug builds linking through /mnt/user (shfs, FUSE) took load to 248 with mostly system time and stalled everything on the shares, ipx's downloads included. Available memory near nothing makes it worse. Disk busy near 100% on nvme1n1 is /mnt/fast, appdata and Postgres's data; on md* the array behind /mnt/user, the downloads. Say what was running at the time if you can tell (docker stats --no-stream); a host problem is the user's, not an ipx issue, unless ipx caused it.

The monitoring itself

A quiet dashboard can be a dead exporter. Check what Prometheus scrapes and that each container's lines are reaching Loki:

curl -s http://192.168.1.130:9090/api/v1/targets | python3 -c 'import json,sys
for t in json.load(sys.stdin)["data"]["activeTargets"]: print(t["health"], t["labels"]["job"], t["scrapeUrl"], t.get("lastError","")[:80])'
for c in iPX ipx-web ipx-directory; do echo "$c $($Q metric "sum(count_over_time({container=\"$c\"} [1h]))" 1h | tail -1)"; done
docker ps -a --filter label=com.docker.compose.project=monitoring --format '{{.Names}} {{.Status}}'

ipx and node should be up. Every monitoring container should be running; after Docker restarts, one can be left Exited (255) though its policy is unless-stopped (Prometheus, on 2026-10-12, when Docker lost its exit during an overload): docker start it and say so. blocky and nut on 192.168.2.21 are another machine, not ipx's; leave them. A container with no lines in an hour while it runs means Alloy is not shipping it; Alloy can take a few minutes to pick up a container after a deploy.

Rules

docker logs and docker exec iPX ipx ... are fine. Do not query production's Postgres directly: ask the user if a question needs the database.

What to do with what you find

Follow the repository's rules in CLAUDE.md: every problem found gets a Gitea issue, with a closed stdin and a timeout on tea. Before filing, list the open issues and do not file one twice; comment on the existing issue with the new evidence instead.

R="--login git.sdf1.net --repo rays/ipx"
t() { timeout 30 /src/tea "$@" < /dev/null; }
t issues list $R --state open
t issues create $R -t "<what is wrong, as the user would notice it>" -L bug -d "<what, where, since when, how often, the query or trace id that shows it>"

Put in each issue what would let someone pick it up cold: the LogQL or TraceQL that shows it, a trace id, the first time it was seen and how often. A publisher's dead feed is worth one issue saying so (the user may want to unsubscribe or find its new address); a single 503 is not.

Finish with a short report to the user, a line for each of ipx-web, ipx, the directory, Tower and the monitoring: what is healthy, what is wrong (with the issue numbers), and anything you could not tell from logs, traces and metrics alone. Do not fix things unless asked; the check is for finding them.

When the checks come back empty

Check that there is data before concluding all is well (the monitoring section's per-container count): iPX should have hundreds of lines an hour or more, ipx-web at least its healthchecks' 120, ipx-directory a few dozen. Nothing at all means Alloy is not shipping it, or the container is down.