rays
8dc6dd4ce6
Kept bytes against the limit was a flat line once the cache filled. Now a percentage of the limit beside the share served from disk, and the images each sweep evicts. The counter shipped minutes ago as ipx_art_trimmed_total is renamed ipx_art_evicted_total before anything came to read it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
237 lines
16 KiB
Markdown
237 lines
16 KiB
Markdown
---
|
|
name: ipx-prod-check
|
|
description: Look for problems in production ipx (the iPX container on Tower) from its logs in Loki, its traces in Tempo and its metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site.
|
|
---
|
|
|
|
# Checking production ipx
|
|
|
|
Production logs one JSON object a line (`IPX_LOG_FORMAT=json`), which Alloy ships to Loki under
|
|
`{container="iPX"}`, and sends traces to Tempo tagged
|
|
`resource.deployment.environment.name="production"`. Anything without that tag is a daemon run by
|
|
hand, not production. Loki and Tempo publish no query port, so use the script beside this file:
|
|
|
|
```sh
|
|
Q=.claude/skills/ipx-prod-check/query.py
|
|
$Q logs '<LogQL>' [since] # lines, newest first (500 at most)
|
|
$Q metric '<LogQL metric>' [since] # $range becomes `since`
|
|
$Q traces '<TraceQL>' [since] # slowest first
|
|
$Q trace <trace id> # one trace as a tree, with its log lines
|
|
$Q prom '<PromQL>' [since] # Prometheus, now; $range becomes `since`
|
|
```
|
|
|
|
`since` is `30m`, `1h`, `24h`, `7d`. Default to `24h`; widen it to see whether something is new.
|
|
"Since the last deploy" is since the container started:
|
|
|
|
```sh
|
|
S=$(docker inspect iPX -f '{{.State.StartedAt}}'); echo $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m
|
|
```
|
|
|
|
## What the log carries
|
|
|
|
Every line has `timestamp`, `level`, `message` and `target`; lines inside a span have `span` (the
|
|
innermost: `{"name":"feed","feed":"x"}`). Loki's `| json` flattens it to `span_name`, `span_feed`.
|
|
Lines inside a traced span, requests and scans, also carry `trace_id` and `span_id`: give the
|
|
`trace_id` to `$Q trace` to see the whole request or scan. (From 2026-09-29 16:30 UTC; before that,
|
|
lines had no trace id, the access log's time was `ms`, and each event was logged twice, words under
|
|
`ipx::scan` and fields under `ipx::io`.)
|
|
|
|
| target | fields | what |
|
|
|---|---|---|
|
|
| `ipx::http` | `method`, `path`, `route`, `status`, `duration_ms` | one per web request; `route` is the pattern, empty for an unrouted path |
|
|
| `ipx::scan` | `ev` and the event's own: `feed`, `new`, `downloaded`, `failed`, `bytes`, `msg`, `url`, `feeds`, `reason`, `from`, `to`; on a failure `error.type` and, from an HTTP error, `http.response.status_code` | the daemon's events, one line each, in words; warnings are feed and download failures |
|
|
| `ipx::io` | `ev`, `feeds`, `pending`, `downloaded` on the `status` reply | commands arriving (`-> {...}`) and the healthcheck's answer |
|
|
| `ipx` | message, sometimes fields | start-up, shutdown, account and config messages |
|
|
| `ipx::client` | `ev`, `user`, `feed`, `enclosure`, `deadair`; `msg`, `src`, `line` on an error; `ms` on a timing | what the page reports from the browser (#190): `page_error`, `rejection` and `player_error` as warnings; `play_start` (ms from play to sound) and `stall` (ms waiting mid-play) as info. Lines carry the trace of the action in the page: its API requests send `traceparent`, one trace id per tap or key |
|
|
|
|
Events (`ev`): `feed_start`, `feed_done` (new, downloaded, failed, torrents), `feed_skip` (not due,
|
|
routine), `feed_error` (msg), `feed_moved` (from, to: a permanent redirect followed, the address
|
|
updated), `download_done` (bytes), `download_error` (msg, url),
|
|
`torrent_deferred`, `reaped`, `scan_done` (feeds checked), `reap_done`, `status` (feeds, pending,
|
|
downloaded: the healthcheck's, every 30s), `deadair_done` (feed, enclosure, `saved_s` at the
|
|
default pause), `error` (msg). DeadAir also logs `DeadAir found the silences` (`cuts`, `saved_s`,
|
|
`took_ms`) under `ipx::deadair`, and warns `DeadAir could not read a file` when ffmpeg fails on
|
|
one, which it then passes over until the next restart.
|
|
|
|
`error.type` is the HTTP status (`404`, `503`) or one of `dns`, `redirect_loop`, `timeout`, `tls`,
|
|
`not_a_feed`, `site_message`, `connect`, `parse`, `other`; Loki's `| json` names it `error_type`.
|
|
|
|
Filter on the text before `| json` where you can (`|= "\"ev\":\"feed_error\""`): it is much
|
|
cheaper than parsing every line. Lines before 2026-09-29 14:00 UTC are text, not JSON, and
|
|
`| json | __error__=""` drops them.
|
|
|
|
## What Prometheus carries
|
|
|
|
The daemon's own state between requests, which no log line or trace holds (#194): ipx serves
|
|
`/metrics` on `[web] metrics_bind` (production: `192.168.1.130:9187`, LAN only), on a thread and
|
|
port of its own, from memory alone, and Prometheus scrapes it every 15 s as `job="ipx"`. Kept
|
|
from 2026-10-11 15:15 UTC. All gauges:
|
|
|
|
| metric | what |
|
|
|---|---|
|
|
| `ipx_db_connections{state="in_use"\|"idle"}`, `ipx_db_connections_max` | the database pool: in use reaching the max means requests and scans wait for a connection |
|
|
| `ipx_worker_queue` | commands waiting for the sequential worker (scans, downloads, adds) |
|
|
| `ipx_event_listeners` | open pages and socket clients on the live-update stream |
|
|
| `ipx_torrent_slots_free` | of 2 |
|
|
| `ipx_deadair_playing` | episodes being played through DeadAir's playlist |
|
|
| `ipx_runtime_workers`, `ipx_runtime_tasks`, `ipx_runtime_queue` | tokio: threads, live tasks, tasks waiting for a thread |
|
|
| `ipx_process_resident_bytes`, `ipx_process_open_fds` | memory and open files and sockets |
|
|
| `ipx_uptime_seconds`, `ipx_build_info{version}` | since the daemon started; what runs |
|
|
| `ipx_worker_busy`, `ipx_worker_job_seconds{kind}` | the worker's job now (`scan`, `reap`, `download`, `status`, `reload`) and for how long; the healthcheck passes while it is stuck |
|
|
| `ipx_jobs_total{kind}`, `ipx_job_seconds_total{kind}` | counters: jobs finished, and their seconds all told |
|
|
| `ipx_last_scan_age_seconds` | since a scan last finished; absent until one has since starting |
|
|
| `ipx_http_requests_in_flight`, `ipx_scan_feeds_fetching`, `ipx_downloads_active` | work in progress |
|
|
| `ipx_download_bytes_total` | counter; `rate()` is the download speed |
|
|
| `ipx_event_queue`, `ipx_events_missed_total` | live updates unread by some listener; counter of those dropped for one that fell behind |
|
|
| `ipx_deadair_backlog`, `ipx_deadair_busy_seconds` | files of shows with DeadAir on not yet looked at (episodes everyone heard through stay counted); how long it has been on its file |
|
|
| `ipx_feed_retries` | feeds waiting to retry a failed name lookup |
|
|
| `ipx_art_cache_bytes`, `ipx_art_cache_limit_bytes` | artwork kept, as of the last sweep, and the most kept: kept sits at the limit by design |
|
|
| `ipx_art_evicted_total`, `ipx_art_requests_total{result="hit"\|"miss"}` | counters: images the sweep threw out to stay under the limit, and artwork served from disk or fetched from the publisher; many evicted at every sweep with a falling hit rate means the limit is too small |
|
|
| `ipx_feeds`, `ipx_downloads_pending`, `ipx_files_downloaded` | the library, as the healthcheck last counted it (every 30 s) |
|
|
|
|
## The checks
|
|
|
|
Run these, then read the lines behind whatever stands out. Most of the time is in the reading:
|
|
a count says something happened, the lines and traces say why.
|
|
|
|
1. **Warnings and errors, grouped.** What went wrong, how often, and since when.
|
|
```
|
|
$Q metric 'sum by (target, message) (count_over_time({container="iPX"} | json | __error__="" | level=~"WARN|ERROR" [$range]))'
|
|
```
|
|
Feed and download failures name the feed in the message; group them in the next check instead.
|
|
2. **Failing feeds and downloads.**
|
|
```
|
|
$Q metric 'sum by (feed, error_type) (count_over_time({container="iPX"} |= "\"ev\":\"feed_error\"" | json | __error__="" [$range]))' 7d
|
|
$Q metric 'sum by (feed, error_type) (count_over_time({container="iPX"} |= "\"ev\":\"download_error\"" | json | __error__="" [$range]))' 7d
|
|
```
|
|
Tell the publisher's problems from ipx's. A 404, 410, DNS failure or 503 from the feed's own
|
|
server is the publisher (worth saying, since the feed may have moved; one issue for a feed
|
|
that has been dead for days, not for a 503 once). A parse error on a feed that loads in a
|
|
browser, a redirect loop ipx should follow, or the same failure on many feeds at once is ipx.
|
|
3. **Server errors and slow requests.**
|
|
```
|
|
$Q metric 'sum by (method, route, status) (count_over_time({container="iPX"} |= "\"target\":\"ipx::http\"" | json | __error__="" | status >= 500 [$range]))'
|
|
$Q metric 'topk(10, quantile_over_time(0.95, {container="iPX"} |= "\"target\":\"ipx::http\"" | json | __error__="" | route != "" | route != "/api/events" | unwrap duration_ms [$range]) by (method, route))'
|
|
```
|
|
Any 5xx is worth a look. 401s are people signing in, not a problem unless one address is
|
|
hammering. For a slow route, find its traces (check 5) and see which span holds the time.
|
|
4. **Is the worker keeping up?** Scans should finish regularly, the queue should drain, and the
|
|
daemon should not be restarting on its own.
|
|
```
|
|
$Q metric 'sum(count_over_time({container="iPX"} |= "\"ev\":\"scan_done\"" [$range]))' 6h
|
|
$Q logs '{container="iPX"} |= "\"ev\":\"status\"" | json | line_format "{{.timestamp}} pending={{.pending}} downloaded={{.downloaded}}"' 6h
|
|
$Q logs '{container="iPX"} |= "daemon started"' 7d
|
|
```
|
|
A `daemon started` not matched by a deploy (see `git log` and the image's build time) is a
|
|
crash or an OOM kill: check `docker inspect iPX -f '{{.State.OOMKilled}} {{.RestartCount}}'`
|
|
and the lines just before it. A pending count that only grows means downloads are not
|
|
keeping up or not running.
|
|
|
|
Since 2026-10-02 the daemon sleeps until the next feed is due (at most 10 minutes) instead of
|
|
scanning every minute (#114), so expect tens of scans in 6 hours, not 360, nearly all with
|
|
`feeds` above 0; none at all for over 10 minutes means the worker is stuck. And `pending` is
|
|
the real queue (#113): files a scan will download on its own. Back-catalogue files are
|
|
`held`, listed but not counted, so it is usually 0 or a handful.
|
|
5. **Slow and failed traces.**
|
|
```
|
|
$Q traces '{resource.deployment.environment.name="production" && duration > 5s}'
|
|
$Q traces '{resource.deployment.environment.name="production" && status = error}'
|
|
$Q trace <id>
|
|
```
|
|
Scans (`scan`) are long by nature, since they fetch many feeds one after another: look for one
|
|
`feed` or `fetch` span holding most of it, or a `download` far slower than its size explains.
|
|
A web request over a second is worth a look; the trace shows whether the time is in the
|
|
handler or a scan it waited on.
|
|
|
|
Steps are `step` spans named by `of` (#187): an item list's `entries_in`, `sanitize`,
|
|
`deadair`, `count_in`; the feed list's `catalogue`, `feed_list` and its queries; a `reap`'s
|
|
`missing_files`, `prune_abandoned_failures`, `reap_candidates`, `clean_directory` (once a day
|
|
since #193, so a reap is about a second), `trim_artwork`, with what it removed on the `reap`
|
|
span. DeadAir's work on a file is a `deadair` trace (feed, enclosure, cuts) with `probe`,
|
|
`silencedetect`, `frame_bytes`, `store_plans`: seconds each, more is a file ffmpeg struggles
|
|
with. A trace with no name in `$Q traces` started in the page (#190, #192): its root is the tap
|
|
in the browser, which sends no span, so it holds the requests that tap made (API calls, the
|
|
playlist and audio of a play, position saves, the page's own report) side by side.
|
|
|
|
6. **The daemon's state, from Prometheus.** Pressure and leaks that requests and traces only
|
|
hint at.
|
|
```
|
|
$Q prom 'up{job="ipx"}'
|
|
$Q prom 'max(max_over_time(ipx_db_connections{job="ipx",state="in_use"}[$range]))' 24h
|
|
$Q prom 'max(max_over_time(ipx_db_connections_max{job="ipx"}[$range]))' 24h
|
|
$Q prom 'max(max_over_time(ipx_worker_queue{job="ipx"}[$range]))' 24h
|
|
$Q prom 'max(max_over_time(ipx_runtime_queue{job="ipx"}[$range]))' 24h
|
|
$Q prom 'max(resets(ipx_uptime_seconds{job="ipx"}[$range]))' 24h
|
|
$Q prom 'ipx_process_resident_bytes{job="ipx"} - ipx_process_resident_bytes{job="ipx"} offset $range' 24h
|
|
$Q prom 'ipx_process_open_fds{job="ipx"} - ipx_process_open_fds{job="ipx"} offset $range' 24h
|
|
$Q prom 'ipx_runtime_tasks{job="ipx"} - ipx_runtime_tasks{job="ipx"} offset $range' 24h
|
|
$Q prom 'max(max_over_time(ipx_worker_job_seconds{job="ipx"}[$range])) by (kind)' 24h
|
|
$Q prom 'max(max_over_time(ipx_last_scan_age_seconds{job="ipx"}[$range]))' 24h
|
|
$Q prom 'sum by (kind) (increase(ipx_job_seconds_total{job="ipx"}[$range]))' 24h
|
|
$Q prom 'max(max_over_time(ipx_http_requests_in_flight{job="ipx"}[$range]))' 24h
|
|
$Q prom 'sum(increase(ipx_events_missed_total{job="ipx"}[$range]))' 24h
|
|
$Q prom 'max(max_over_time(ipx_deadair_busy_seconds{job="ipx"}[$range]))' 24h
|
|
```
|
|
`up` 0 means Prometheus cannot reach the port: the container is down or `metrics_bind` is
|
|
unset. Connections in use at the pool's max (10 on Postgres) means something is waiting for
|
|
one: find the slow requests or scan of that time in check 3 and 5. A worker queue that only
|
|
grows means downloads are not keeping up, or the worker is stuck on one job. A runtime queue
|
|
above 0 for more than a scrape or two means every thread is busy: something is blocking a
|
|
thread or burning one. Memory, open files or tasks only climbing across a day with no restart
|
|
is a leak (a file or socket not closed, a task never ending); a step after a deploy is not. A
|
|
reset of the uptime is a restart: match it with a deploy, or it is a crash (check 4).
|
|
|
|
The worker: one job over 30 minutes (`ipx_worker_job_seconds`), or a scan last finished over
|
|
15 minutes ago, is the worker stuck, which the healthcheck does not notice; its trace and the
|
|
log lines of that time say on what. A kind whose share of `ipx_job_seconds_total` jumps is
|
|
where the worker's time went (reap was most of it until #193). Web requests in flight near
|
|
the pool's max mean the pool is what they wait on. Live updates missed mean a page showed stale
|
|
counts until reloaded. DeadAir over a minute on one file is a file ffmpeg chokes on.
|
|
|
|
7. **What the page saw.** The browser's own errors and timings, which no server line shows.
|
|
```
|
|
$Q metric 'sum by (ev, msg) (count_over_time({container="iPX"} |= "\"target\":\"ipx::client\"" | json | __error__="" | level="WARN" [$range]))' 7d
|
|
$Q logs '{container="iPX"} |= "\"target\":\"ipx::client\"" | json | ev=~"play_start|stall" | line_format "{{.timestamp}} {{.user}} {{.ev}} {{.ms}}ms deadair={{.deadair}} {{.feed}}"' 24h
|
|
```
|
|
A `player_error` is playback that failed in front of someone: `hls.js: ...` is DeadAir's
|
|
playlist (its trace has the playlist and audio requests), `media error 4` a file the browser
|
|
cannot play. A `page_error` or `rejection` is a bug in the page; `src` and `line` point into
|
|
the minified `app.js`, so match the message to `web/src`. `play_start` over 2000 ms, or a
|
|
`stall`, is worth a trace: was it the server, Cloudflare, or the network.
|
|
|
|
Also check the container itself, since Loki cannot see a daemon that is not running:
|
|
|
|
```sh
|
|
docker ps --filter name=iPX --format '{{.Status}}'
|
|
docker logs --since 10m iPX 2>&1 | tail -5
|
|
```
|
|
|
|
`docker logs` and `docker exec iPX ipx ...` are fine. Do not query production's Postgres
|
|
directly: ask the user if a question needs the database.
|
|
|
|
## What to do with what you find
|
|
|
|
Follow the repository's rules in CLAUDE.md: **every problem found gets a Gitea issue**, with a
|
|
closed stdin and a timeout on `tea`. Before filing, list the open issues and do not file one
|
|
twice; comment on the existing issue with the new evidence instead.
|
|
|
|
```sh
|
|
R="--login git.sdf1.net --repo rays/ipx"
|
|
t() { timeout 30 /src/tea "$@" < /dev/null; }
|
|
t issues list $R --state open
|
|
t issues create $R -t "<what is wrong, as the user would notice it>" -L bug -d "<what, where, since when, how often, the query or trace id that shows it>"
|
|
```
|
|
|
|
Put in each issue what would let someone pick it up cold: the LogQL or TraceQL that shows it, a
|
|
trace id, the first time it was seen and how often. A publisher's dead feed is worth one issue
|
|
saying so (the user may want to unsubscribe or find its new address); a single 503 is not.
|
|
|
|
Finish with a short report to the user: what is healthy, what is wrong (with the issue numbers),
|
|
and anything you could not tell from logs and traces alone. Do not fix things unless asked; the
|
|
check is for finding them.
|
|
|
|
## When the checks come back empty
|
|
|
|
Check that there is data before concluding all is well: `$Q metric 'sum(count_over_time({container="iPX"} [1h]))' 1h`
|
|
should be in the hundreds or more. Nothing at all means Alloy is not shipping (it can take a few
|
|
minutes to pick up a container after a deploy), or the container is down.
|