rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

prod check: slow requests as spans, with their route, laid beside Tower's load

Browse Source 
Check 5 listed traces over 5 s, which were all page traces holding minutes of
live updates or audio, so a slow request never showed. query.py's new `spans`
lists the matching spans with their own time; check 5 asks for ipx requests
over a second, steps over half a second, and says to set the hours beside
Tower's load: on 2026-10-18, 120 of 130 slow requests were the host stalling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-18 18:59:20 +00:00
parent 59aa00e07c
commit 3df7c974e8
2 changed files with 37 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
.claude/skills/ipx-prod-check/query.py
View File

@@ -4,6 +4,7 @@
query.py logs '<LogQL log query>' [since] lines, newest first
query.py metric '<LogQL metric query>' [since] one value per series; $range is `since`
query.py traces '<TraceQL query>' [since] matching traces, slowest first
query.py spans '<TraceQL query>' [since] the matching spans, each with its own time
query.py trace <trace id> one trace's spans, as a tree
query.py prom '<PromQL query>' [since] one value per series, now; $range is `since`
@@ -58,6 +59,21 @@ def main():
name = t.get("rootTraceName") or "(from the page)"
print(f"{t.get('durationMs', 0):>8}ms {when}Z {t['traceID']} {name}")
print(f"-- {len(traces)} trace(s)", file=sys.stderr)
elif kind == "spans":
# The spans that matched, each with its own time: a trace's length says nothing about a
# slow request in it when the trace also holds a page's live updates or an audio stream.
r = fetch(f"http://tempo:3200/api/search?q={enc(q)}&start={int(start)}&end={int(now)}&limit=200&spss=20")
rows = []
for t in r.get("traces", []):
for ss in t.get("spanSets") or [t.get("spanSet") or {}]:
for sp in ss.get("spans", []):
attrs = {a["key"]: next(iter(a["value"].values()), None) for a in sp.get("attributes", [])}
what = attrs.get("http.route") or attrs.get("of") or sp.get("name") or ""
rows.append((int(sp.get("durationNanos", 0)) / 1e6, int(sp["startTimeUnixNano"]), t["traceID"], what))
for ms, ts, tid, what in sorted(rows, reverse=True):
when = time.strftime("%m-%d %H:%M:%S", time.gmtime(ts / 1e9))
print(f"{ms:>8.0f}ms {when}Z {tid} {what}")
print(f"-- {len(rows)} span(s)", file=sys.stderr)
elif kind == "prom":
# ipx's /metrics (#194), scraped as job ipx; through the network like the others.
r = fetch(f"http://prometheus:9090/api/v1/query?query={enc(q.replace('$range', since))}&time={now}")