Prod check reads the page's reports, today's trace steps and DeadAir's events

A seventh check groups the browser's errors and lists play starts and
stalls (ipx::client, #190); the trace check names the steps of an item
list, the feed list, a reap and DeadAir's per-file trace (#187), and says
a trace with no name began in the page (#192), which query.py now labels;
the events list has deadair_done and DeadAir's own lines; and "since the
last deploy" is worked out from the container's start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-11 15:17:34 +00:00
parent 29f1cbce29
commit 0d930e75ee
2 changed files with 34 additions and 3 deletions

View File

@@ -19,7 +19,12 @@ $Q trace <trace id> # one trace as a tree, with its log lin
$Q prom '<PromQL>' [since] # Prometheus, now; $range becomes `since`
```
`since` is `1h`, `24h`, `7d`. Default to `24h`; widen it to see whether something is new.
`since` is `30m`, `1h`, `24h`, `7d`. Default to `24h`; widen it to see whether something is new.
"Since the last deploy" is since the container started:
```sh
S=$(docker inspect iPX -f '{{.State.StartedAt}}'); echo $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m
```
## What the log carries
@@ -42,7 +47,10 @@ Events (`ev`): `feed_start`, `feed_done` (new, downloaded, failed, torrents), `f
routine), `feed_error` (msg), `feed_moved` (from, to: a permanent redirect followed, the address
updated), `download_done` (bytes), `download_error` (msg, url),
`torrent_deferred`, `reaped`, `scan_done` (feeds checked), `reap_done`, `status` (feeds, pending,
downloaded: the healthcheck's, every 30s), `error` (msg).
downloaded: the healthcheck's, every 30s), `deadair_done` (feed, enclosure, `saved_s` at the
default pause), `error` (msg). DeadAir also logs `DeadAir found the silences` (`cuts`, `saved_s`,
`took_ms`) under `ipx::deadair`, and warns `DeadAir could not read a file` when ffmpeg fails on
one, which it then passes over until the next restart.
`error.type` is the HTTP status (`404`, `503`) or one of `dns`, `redirect_loop`, `timeout`, `tls`,
`not_a_feed`, `site_message`, `connect`, `parse`, `other`; Loki's `| json` names it `error_type`.
@@ -133,6 +141,16 @@ a count says something happened, the lines and traces say why.
A web request over a second is worth a look; the trace shows whether the time is in the
handler or a scan it waited on.
Steps are `step` spans named by `of` (#187): an item list's `entries_in`, `sanitize`,
`deadair`, `count_in`; the feed list's `catalogue`, `feed_list` and its queries; a `reap`'s
`missing_files`, `prune_abandoned_failures`, `reap_candidates`, `clean_directory` (once a day
since #193, so a reap is about a second), `trim_artwork`, with what it removed on the `reap`
span. DeadAir's work on a file is a `deadair` trace (feed, enclosure, cuts) with `probe`,
`silencedetect`, `frame_bytes`, `store_plans`: seconds each, more is a file ffmpeg struggles
with. A trace with no name in `$Q traces` started in the page (#190, #192): its root is the tap
in the browser, which sends no span, so it holds the requests that tap made (API calls, the
playlist and audio of a play, position saves, the page's own report) side by side.
6. **The daemon's state, from Prometheus.** Pressure and leaks that requests and traces only
hint at.
```
@@ -168,6 +186,17 @@ a count says something happened, the lines and traces say why.
the pool's max mean the pool is what they wait on. Live updates missed mean a page showed stale
counts until reloaded. DeadAir over a minute on one file is a file ffmpeg chokes on.
7. **What the page saw.** The browser's own errors and timings, which no server line shows.
```
$Q metric 'sum by (ev, msg) (count_over_time({container="iPX"} |= "\"target\":\"ipx::client\"" | json | __error__="" | level="WARN" [$range]))' 7d
$Q logs '{container="iPX"} |= "\"target\":\"ipx::client\"" | json | ev=~"play_start|stall" | line_format "{{.timestamp}} {{.user}} {{.ev}} {{.ms}}ms deadair={{.deadair}} {{.feed}}"' 24h
```
A `player_error` is playback that failed in front of someone: `hls.js: ...` is DeadAir's
playlist (its trace has the playlist and audio requests), `media error 4` a file the browser
cannot play. A `page_error` or `rejection` is a bug in the page; `src` and `line` point into
the minified `app.js`, so match the message to `web/src`. `play_start` over 2000 ms, or a
`stall`, is worth a trace: was it the server, Cloudflare, or the network.
Also check the container itself, since Loki cannot see a daemon that is not running:
```sh