rays/ipx
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Prod check reads the daemon's metrics from Prometheus (#194)

Browse Source 
query.py prom asks Prometheus as it asks Loki, and a sixth check looks for
a pool at its max, a worker or runtime queue that only grows, memory, open
files or tasks that only climb, and restarts by the uptime's resets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-11 15:06:58 +00:00
parent 973eeb67c7
commit a985cfafef
2 changed files with 52 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -1,6 +1,6 @@
--- ---
name: ipx-prod-check name: ipx-prod-check
description: Look for problems in production ipx (the iPX container on Tower) from its logs in Loki and its traces in Tempo, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site. description: Look for problems in production ipx (the iPX container on Tower) from its logs in Loki, its traces in Tempo and its metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site.
--- ---
# Checking production ipx # Checking production ipx
@@ -16,6 +16,7 @@ $Q logs '<LogQL>' [since] # lines, newest first (500 at most)
$Q metric '<LogQL metric>' [since] # $range becomes `since` $Q metric '<LogQL metric>' [since] # $range becomes `since`
$Q traces '<TraceQL>' [since] # slowest first $Q traces '<TraceQL>' [since] # slowest first
$Q trace <trace id> # one trace as a tree, with its log lines $Q trace <trace id> # one trace as a tree, with its log lines
$Q prom '<PromQL>' [since] # Prometheus, now; $range becomes `since`
``` ```
`since` is `1h`, `24h`, `7d`. Default to `24h`; widen it to see whether something is new. `since` is `1h`, `24h`, `7d`. Default to `24h`; widen it to see whether something is new.
@@ -50,6 +51,24 @@ Filter on the text before `| json` where you can (`|= "\"ev\":\"feed_error\""`):
cheaper than parsing every line. Lines before 2026-09-29 14:00 UTC are text, not JSON, and cheaper than parsing every line. Lines before 2026-09-29 14:00 UTC are text, not JSON, and
`| json | __error__=""` drops them. `| json | __error__=""` drops them.
## What Prometheus carries
The daemon's own state between requests, which no log line or trace holds (#194): ipx serves
`/metrics` on `[web] metrics_bind` (production: `192.168.1.130:9187`, LAN only), on a thread and
port of its own, from memory alone, and Prometheus scrapes it every 15 s as `job="ipx"`. Kept
from 2026-10-11 15:15 UTC. All gauges:
| metric | what |
|---|---|
| `ipx_db_connections{state="in_use"\|"idle"}`, `ipx_db_connections_max` | the database pool: in use reaching the max means requests and scans wait for a connection |
| `ipx_worker_queue` | commands waiting for the sequential worker (scans, downloads, adds) |
| `ipx_event_listeners` | open pages and socket clients on the live-update stream |
| `ipx_torrent_slots_free` | of 2 |
| `ipx_deadair_playing` | episodes being played through DeadAir's playlist |
| `ipx_runtime_workers`, `ipx_runtime_tasks`, `ipx_runtime_queue` | tokio: threads, live tasks, tasks waiting for a thread |
| `ipx_process_resident_bytes`, `ipx_process_open_fds` | memory and open files and sockets |
| `ipx_uptime_seconds`, `ipx_build_info{version}` | since the daemon started; what runs |
## The checks ## The checks
Run these, then read the lines behind whatever stands out. Most of the time is in the reading: Run these, then read the lines behind whatever stands out. Most of the time is in the reading:
@@ -104,6 +123,28 @@ a count says something happened, the lines and traces say why.
A web request over a second is worth a look; the trace shows whether the time is in the A web request over a second is worth a look; the trace shows whether the time is in the
handler or a scan it waited on. handler or a scan it waited on.
6. **The daemon's state, from Prometheus.** Pressure and leaks that requests and traces only
hint at.
```
$Q prom 'up{job="ipx"}'
$Q prom 'max(max_over_time(ipx_db_connections{job="ipx",state="in_use"}[$range]))' 24h
$Q prom 'max(max_over_time(ipx_db_connections_max{job="ipx"}[$range]))' 24h
$Q prom 'max(max_over_time(ipx_worker_queue{job="ipx"}[$range]))' 24h
$Q prom 'max(max_over_time(ipx_runtime_queue{job="ipx"}[$range]))' 24h
$Q prom 'max(resets(ipx_uptime_seconds{job="ipx"}[$range]))' 24h
$Q prom 'ipx_process_resident_bytes{job="ipx"} - ipx_process_resident_bytes{job="ipx"} offset $range' 24h
$Q prom 'ipx_process_open_fds{job="ipx"} - ipx_process_open_fds{job="ipx"} offset $range' 24h
$Q prom 'ipx_runtime_tasks{job="ipx"} - ipx_runtime_tasks{job="ipx"} offset $range' 24h
```
`up` 0 means Prometheus cannot reach the port: the container is down or `metrics_bind` is
unset. Connections in use at the pool's max (10 on Postgres) means something is waiting for
one: find the slow requests or scan of that time in check 3 and 5. A worker queue that only
grows means downloads are not keeping up, or the worker is stuck on one job. A runtime queue
above 0 for more than a scrape or two means every thread is busy: something is blocking a
thread or burning one. Memory, open files or tasks only climbing across a day with no restart
is a leak (a file or socket not closed, a task never ending); a step after a deploy is not. A
reset of the uptime is a restart: match it with a deploy, or it is a crash (check 4).
Also check the container itself, since Loki cannot see a daemon that is not running: Also check the container itself, since Loki cannot see a daemon that is not running:
```sh ```sh

11
.claude/skills/ipx-prod-check/query.py
View File

@@ -1,10 +1,11 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Ask production's Loki or Tempo a question, from anywhere that can run docker on Tower. """Ask production's Loki, Tempo or Prometheus a question, from anywhere that can run docker on Tower.
query.py logs '<LogQL log query>' [since] lines, newest first query.py logs '<LogQL log query>' [since] lines, newest first
query.py metric '<LogQL metric query>' [since] one value per series; $range is `since` query.py metric '<LogQL metric query>' [since] one value per series; $range is `since`
query.py traces '<TraceQL query>' [since] matching traces, slowest first query.py traces '<TraceQL query>' [since] matching traces, slowest first
query.py trace <trace id> one trace's spans, as a tree query.py trace <trace id> one trace's spans, as a tree
query.py prom '<PromQL query>' [since] one value per series, now; $range is `since`
`since` is 1h, 24h, 7d and the like (default 24h). Loki and Tempo publish no query port on the `since` is 1h, 24h, 7d and the like (default 24h). Loki and Tempo publish no query port on the
host, so each call runs a throwaway alpine container on the monitoring project's network. host, so each call runs a throwaway alpine container on the monitoring project's network.
@@ -55,6 +56,14 @@ def main():
when = time.strftime("%m-%d %H:%M:%S", time.gmtime(int(t["startTimeUnixNano"]) / 1e9)) when = time.strftime("%m-%d %H:%M:%S", time.gmtime(int(t["startTimeUnixNano"]) / 1e9))
print(f"{t.get('durationMs', 0):>8}ms {when}Z {t['traceID']} {t.get('rootTraceName', '')}") print(f"{t.get('durationMs', 0):>8}ms {when}Z {t['traceID']} {t.get('rootTraceName', '')}")
print(f"-- {len(traces)} trace(s)", file=sys.stderr) print(f"-- {len(traces)} trace(s)", file=sys.stderr)
elif kind == "prom":
# ipx's /metrics (#194), scraped as job ipx; through the network like the others.
r = fetch(f"http://prometheus:9090/api/v1/query?query={enc(q.replace('$range', since))}&time={now}")
rows = sorted(r["data"]["result"], key=lambda s: -float(s["value"][1]))
for s in rows:
labels = {k: v for k, v in s["metric"].items() if k not in ("job", "instance")}
print(f"{float(s['value'][1]):>14g} {json.dumps(labels) if labels else ''}")
print(f"-- {len(rows)} series", file=sys.stderr)
elif kind == "trace": elif kind == "trace":
r = fetch(f"http://tempo:3200/api/traces/{q}") r = fetch(f"http://tempo:3200/api/traces/{q}")
spans = [sp for b in r.get("batches", r.get("resourceSpans", [])) spans = [sp for b in r.get("batches", r.get("resourceSpans", []))