diff --git a/.claude/skills/ipx-prod-check/SKILL.md b/.claude/skills/ipx-prod-check/SKILL.md index 796860a..43825a3 100644 --- a/.claude/skills/ipx-prod-check/SKILL.md +++ b/.claude/skills/ipx-prod-check/SKILL.md @@ -19,7 +19,12 @@ $Q trace # one trace as a tree, with its log lin $Q prom '' [since] # Prometheus, now; $range becomes `since` ``` -`since` is `1h`, `24h`, `7d`. Default to `24h`; widen it to see whether something is new. +`since` is `30m`, `1h`, `24h`, `7d`. Default to `24h`; widen it to see whether something is new. +"Since the last deploy" is since the container started: + +```sh +S=$(docker inspect iPX -f '{{.State.StartedAt}}'); echo $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m +``` ## What the log carries @@ -42,7 +47,10 @@ Events (`ev`): `feed_start`, `feed_done` (new, downloaded, failed, torrents), `f routine), `feed_error` (msg), `feed_moved` (from, to: a permanent redirect followed, the address updated), `download_done` (bytes), `download_error` (msg, url), `torrent_deferred`, `reaped`, `scan_done` (feeds checked), `reap_done`, `status` (feeds, pending, -downloaded: the healthcheck's, every 30s), `error` (msg). +downloaded: the healthcheck's, every 30s), `deadair_done` (feed, enclosure, `saved_s` at the +default pause), `error` (msg). DeadAir also logs `DeadAir found the silences` (`cuts`, `saved_s`, +`took_ms`) under `ipx::deadair`, and warns `DeadAir could not read a file` when ffmpeg fails on +one, which it then passes over until the next restart. `error.type` is the HTTP status (`404`, `503`) or one of `dns`, `redirect_loop`, `timeout`, `tls`, `not_a_feed`, `site_message`, `connect`, `parse`, `other`; Loki's `| json` names it `error_type`. @@ -133,6 +141,16 @@ a count says something happened, the lines and traces say why. A web request over a second is worth a look; the trace shows whether the time is in the handler or a scan it waited on. + Steps are `step` spans named by `of` (#187): an item list's `entries_in`, `sanitize`, + `deadair`, `count_in`; the feed list's `catalogue`, `feed_list` and its queries; a `reap`'s + `missing_files`, `prune_abandoned_failures`, `reap_candidates`, `clean_directory` (once a day + since #193, so a reap is about a second), `trim_artwork`, with what it removed on the `reap` + span. DeadAir's work on a file is a `deadair` trace (feed, enclosure, cuts) with `probe`, + `silencedetect`, `frame_bytes`, `store_plans`: seconds each, more is a file ffmpeg struggles + with. A trace with no name in `$Q traces` started in the page (#190, #192): its root is the tap + in the browser, which sends no span, so it holds the requests that tap made (API calls, the + playlist and audio of a play, position saves, the page's own report) side by side. + 6. **The daemon's state, from Prometheus.** Pressure and leaks that requests and traces only hint at. ``` @@ -168,6 +186,17 @@ a count says something happened, the lines and traces say why. the pool's max mean the pool is what they wait on. Live updates missed mean a page showed stale counts until reloaded. DeadAir over a minute on one file is a file ffmpeg chokes on. +7. **What the page saw.** The browser's own errors and timings, which no server line shows. + ``` + $Q metric 'sum by (ev, msg) (count_over_time({container="iPX"} |= "\"target\":\"ipx::client\"" | json | __error__="" | level="WARN" [$range]))' 7d + $Q logs '{container="iPX"} |= "\"target\":\"ipx::client\"" | json | ev=~"play_start|stall" | line_format "{{.timestamp}} {{.user}} {{.ev}} {{.ms}}ms deadair={{.deadair}} {{.feed}}"' 24h + ``` + A `player_error` is playback that failed in front of someone: `hls.js: ...` is DeadAir's + playlist (its trace has the playlist and audio requests), `media error 4` a file the browser + cannot play. A `page_error` or `rejection` is a bug in the page; `src` and `line` point into + the minified `app.js`, so match the message to `web/src`. `play_start` over 2000 ms, or a + `stall`, is worth a trace: was it the server, Cloudflare, or the network. + Also check the container itself, since Loki cannot see a daemon that is not running: ```sh diff --git a/.claude/skills/ipx-prod-check/query.py b/.claude/skills/ipx-prod-check/query.py index 452c751..6b737ba 100755 --- a/.claude/skills/ipx-prod-check/query.py +++ b/.claude/skills/ipx-prod-check/query.py @@ -54,7 +54,9 @@ def main(): traces = sorted(r.get("traces", []), key=lambda t: -t.get("durationMs", 0)) for t in traces: when = time.strftime("%m-%d %H:%M:%S", time.gmtime(int(t["startTimeUnixNano"]) / 1e9)) - print(f"{t.get('durationMs', 0):>8}ms {when}Z {t['traceID']} {t.get('rootTraceName', '')}") + # No root name: its root span is a tap in the page, which the browser never sends. + name = t.get("rootTraceName") or "(from the page)" + print(f"{t.get('durationMs', 0):>8}ms {when}Z {t['traceID']} {name}") print(f"-- {len(traces)} trace(s)", file=sys.stderr) elif kind == "prom": # ipx's /metrics (#194), scraped as job ipx; through the network like the others.