rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

prod check: slow requests as spans, with their route, laid beside Tower's load

Browse Source 
Check 5 listed traces over 5 s, which were all page traces holding minutes of
live updates or audio, so a slow request never showed. query.py's new `spans`
lists the matching spans with their own time; check 5 asks for ipx requests
over a second, steps over half a second, and says to set the hours beside
Tower's load: on 2026-10-18, 120 of 130 slow requests were the host stalling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-18 18:59:20 +00:00
parent 59aa00e07c
commit 3df7c974e8
2 changed files with 37 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -26,6 +26,7 @@ Q=.claude/skills/ipx-prod-check/query.py
$Q logs '<LogQL>' [since] # lines, newest first (500 at most)
$Q metric '<LogQL metric>' [since] # $range becomes `since`
$Q traces '<TraceQL>' [since] # slowest first
$Q spans '<TraceQL>' [since] # the matching spans, each with its own time, slowest first
$Q trace <trace id> # one trace as a tree, with its log lines
$Q prom '<PromQL>' [since] # Prometheus, now; $range becomes `since`
```
@@ -159,16 +160,30 @@ a count says something happened, the lines and traces say why.
`feeds` above 0; none at all for over 10 minutes means the worker is stuck. And `pending` is
the real queue (#113): files a scan will download on its own. Back-catalogue files are
`held`, listed but not counted, so it is usually 0 or a handful.
5. **Slow and failed traces.** All three services' (filter by `resource.service.name` for one).
5. **Slow and failed traces.** Look at ipx's requests as spans, not traces: a trace started by a
tap in the page also holds that page's live updates and any audio it streamed, minutes long,
so a list of traces over a few seconds is nothing but those. `spans` lists each matching span
with its own time and route, slowest first:
```
$Q traces '{resource.deployment.environment.name="production" && duration > 5s}'
R='resource.deployment.environment.name="production" && resource.service.name="ipx"'
$Q spans "{$R && span.http.route != \"\" && span.http.route != \"/api/events\" && span.http.route !~ \"/media/.*\" && duration > 1s} | select(span.http.route)"
$Q spans "{$R && name=\"step\" && duration > 500ms} | select(span.of)"
$Q traces '{resource.deployment.environment.name="production" && status = error}'
$Q trace <id>
```
Group the first by hour and by route (`awk '{print substr($3,1,2), $NF}' | sort | uniq -c`),
then lay the hours beside Tower's (the Tower section): load over 100 with processes blocked
and memory under 4 GB is the host stalling, and every request of those minutes is slow with
it, ipx's queries included. On 2026-10-18 that was 120 of 130 requests over a second, up to 43 s.
What is left at quiet hours is ipx's own: open its trace and see which step holds the time. A
request far longer than its steps added up waited outside them, for a connection or a thread.
The second line finds steps over half a second wherever they ran; a step slow only once, first
thing after hours idle, is Postgres reading from disk what it no longer had in memory (All
Subscriptions' `entries_in`, a second where it is 60 ms warm).
Scans (`scan`) are long by nature, since they fetch many feeds one after another: look for one
`feed` or `fetch` span holding most of it, or a `download` far slower than its size explains.
A web request over a second is worth a look; the trace shows whether the time is in the
handler or a scan it waited on.
`feed` or `fetch` span holding most of it (`$Q spans '{... && name="feed" && duration > 10s}'`),
or a `download` far slower than its size explains.
Steps are `step` spans named by `of` (#187): an item list's `entries_in`, `sanitize`,
`deadair`, `count_in`; the feed list's `catalogue`, `feed_list` and its queries; a `reap`'s
@@ -262,7 +277,7 @@ to see where the time went, Caddy's span with ipx's inside it. A slow page with
Caddy or the network; a long one is ipx.
```sh
$Q traces '{resource.service.name="ipx-web" && duration > 2s}'
$Q spans '{resource.service.name="ipx-web" && span.url.path != "/api/events" && span.url.path !~ "/media/.*" && duration > 2s} | select(span.url.path)'
$Q traces '{resource.service.name="ipx-web" && status = error}'
```