Check 5 listed traces over 5 s, which were all page traces holding minutes of
live updates or audio, so a slow request never showed. query.py's new `spans`
lists the matching spans with their own time; check 5 asks for ipx requests
over a second, steps over half a second, and says to set the hours beside
Tower's load: on 2026-10-18, 120 of 130 slow requests were the host stalling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A seventh check groups the browser's errors and lists play starts and
stalls (ipx::client, #190); the trace check names the steps of an item
list, the feed list, a reap and DeadAir's per-file trace (#187), and says
a trace with no name began in the page (#192), which query.py now labels;
the events list has deadair_done and DeadAir's own lines; and "since the
last deploy" is worked out from the container's start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
query.py prom asks Prometheus as it asks Loki, and a sixth check looks for
a pool at its max, a worker or runtime queue that only grows, memory, open
files or tasks that only climb, and restarts by the uptime's resets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.claude/skills/ipx-prod-check: what production's JSON log and traces
carry, the queries that find trouble (warnings grouped, failing feeds and
downloads, 5xx and slow routes, whether the worker keeps up, slow and
failed traces), how to tell a publisher's dead feed from an ipx bug, and
filing what is found as issues per CLAUDE.md. query.py beside it runs the
LogQL and TraceQL through a throwaway container on the monitoring
network, since Loki and Tempo publish no query port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
