rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Metrics and traces for what had none, and alerts on them (#365-#377)

Browse Source 
Reviewing what production's traces and metrics leave out turned up
twelve gaps; each is an issue.

Metrics. A Prometheus histogram for ipx and the directory both
(ipx_directory::metrics::Histograms), which ipx's request histogram
(#364) now uses too.
- ipx: each statement's time once it has a connection, by its first
  word, from SeaORM's metric callback; and the wait for one, for
  hand-written SQL in db.rs, measured as the call less the statement
  (the callback runs in the poll that returns the result, so the time
  read after the await is that statement's) (#367). SeaORM's own queries
  are timed but their wait is not seen: a shortcut, noted.
- ipx: counters of what the events say, counted in emit, which every
  event passes once: feed failures by error.type, new items, downloads
  saved and failed, DeadAir's seconds found (#368).
- ipx: the page's list_shown, play_start and stall timings (#369).
- directory: requests by method, route (a route layer puts the matched
  route on the response, or each feed's page was a row) and status;
  feeds checked and failed; snapshots taken (#370).
- ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from
  the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11
  calls deprecated, because the browser tests' Caddy is 2.6 (#371).

Traces.
- A list's sync is list_sync with steps, an OPML upload opml_import
  (#372). Doing so showed sync_group reading every account and its
  subscription to the list once per listed feed; read once now (#377).
- Each command the worker runs, from the socket, the CLI or the page,
  is a command span with its job inside (#373).
- The daemon's start is a trace, from opening the database through the
  catalogue's repairs to "daemon started" (#374).
- The directory's lines carry trace_id and span_id: WithTrace moves to
  ipx_directory::otel and both use it (#375).
- A tap or key is its trace's root span (#376). The page names a root
  span id in each request's traceparent, where the parent was random and
  pointed at nothing, and reports the tap (ids, start, its last answer)
  when the next begins or the page is hidden. opentelemetry 0.33 gives a
  span builder no way to take ids, so the shared tracer provider has an
  id generator that takes given ids inside with_ids, on one thread.

Alerts. grafana/alerts.py, Grafana alerting provisioning beside the
dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a
job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals
stopped, the directory's refresher stopped, Tower's load and memory, the
monitoring down, Tempo not reading its blocks, iPX's lines not reaching
Loki. No contact point yet: firing shows in Grafana. Prometheus now
scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's
config, not this repository's.

The dashboard has database and page-timing panels; the prod check knows
the new metrics, spans and alerts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-25 16:18:06 +00:00
parent c788112190
commit dd91602a49
23 changed files with 768 additions and 189 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -10,11 +10,12 @@ Production is three containers in the Arcane project `ipx` on Tower
| container | what | logs, in Loki | traces, in Tempo | metrics, in Prometheus |
|---|---|---|---|---|
| `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | service `ipx-web`: a span a request, passing `traceparent` on to ipx (#213) | none |
| `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | service `ipx-web`: a span a request, passing `traceparent` on to ipx (#213) | `job="ipx-web"`: Caddy's own `caddy_http_*`, from `METRICS_BIND`, published as `192.168.1.130:9190` (#371) |
| `iPX` | the daemon: API, scans, downloads, DeadAir | `{container="iPX"}`, JSON (`IPX_LOG_FORMAT=json`) | service `ipx`; passes `traceparent` to the directory | `job="ipx"`, its `/metrics` |
| `ipx-directory` | the directory of public podcasts, `directory.ipxreader.app` (#204, #214) | `{container="ipx-directory"}`, JSON (`DIRECTORY_LOG_FORMAT=json`) | service `ipx-directory`: a span a request, joining ipx's trace when ipx called; a trace a refresher round (#213) | `job="ipx-directory"`, its `/metrics` |
Under them, Tower itself: `job="node"` (the node exporter: load, CPU, memory, disks). The Grafana
Grafana's alert rules (`grafana/alerts.py`, #365) watch all of it between checks: start by asking
which are firing (the monitoring section). Under them, Tower itself: `job="node"` (the node exporter: load, CPU, memory, disks). The Grafana
dashboard `iPX` draws all of it in that order (`grafana/dashboard.py`). A trace without the
production tag (`resource.deployment.environment.name="production"`, on all three) is a daemon run
by hand, not production. Since #213 a request is one trace across the services it touched: a
@@ -111,6 +112,9 @@ from 2026-10-11 15:15 UTC. Gauges, but for the counters (`_total`) and the one h
| `ipx_jobs_total{kind}`, `ipx_job_seconds_total{kind}` | counters: jobs finished, and their seconds all told |
| `ipx_last_scan_age_seconds` | since a scan last finished; absent until one has since starting |
| `ipx_http_requests_in_flight`, `ipx_scan_feeds_fetching`, `ipx_downloads_active` | work in progress |
| `ipx_db_query_seconds{kind}`, `ipx_db_wait_seconds` | histograms (#367): each statement's time once it has a connection, by its first word (`select`, `insert`, ...); and the wait for a connection, measured for hand-written SQL only, about half the queries. A wait growing while queries stay flat is the pool full |
| `ipx_page_seconds{ev}` | histogram: what the page timed, `list_shown`, `play_start`, `stall`, as the `ipx::client` lines say one by one (#369) |
| `ipx_feed_failures_total{error_type}`, `ipx_new_items_total`, `ipx_downloads_total{result="saved"\|"failed"}`, `ipx_deadair_saved_seconds_total` | counters of what the events say (#368): feeds failed by kind, as the `feed_error` lines' `error.type`; new items; downloads; seconds of silence DeadAir found |
| `ipx_http_request_duration_seconds{method,route,status}` | histogram: each request answered and its seconds, buckets 5 ms to 10 s; `_count` is requests, `_sum` their time. `route` is the pattern, `""` for an unrouted path; the event stream is left out (#364). From 2026-10-25 |
| `ipx_download_bytes_total` | counter; `rate()` is the download speed |
| `ipx_event_queue`, `ipx_events_missed_total` | live updates unread by some listener; counter of those dropped for one that fell behind |
@@ -122,7 +126,7 @@ from 2026-10-11 15:15 UTC. Gauges, but for the counters (`_total`) and the one h
| `ipx_jev_requests_total{result="ok"\|"key_refused"\|"malformed"\|"rate_limited"\|"error"}`, `ipx_jev_seconds_total`, `ipx_jev_paced_seconds_total` | counters: calls to TypeSafe's Jev, Signals' and icons', by how they ended; their seconds, the wait for a turn included; and that wait alone. `malformed` is ipx's request at fault (#257); `rate_limited` is a 429 from TypeSafe, whose limits are 80 calls and 100,000 input tokens a second, which ipx paces itself under at 60 and 80,000 (#277) |
| `ipx_jev_tokens_total{kind="input"\|"output",user,key}` | counter: tokens TypeSafe counted, as its answers say, by person and whose key paid: what Signals, icon picks and key checks cost (#275, #278). A call shared on the server's key is split by each person's share of its questions |
| `ipx_signals_last_pass_age_seconds`, `ipx_signals_keys_refused` | since the Signals judge last finished a pass (at most 5 minutes when well; absent until one has); people whose key TypeSafe refused |
| `directory_process_resident_bytes`, `directory_process_open_fds`, `directory_feed_fetches_total`, `directory_feed_bytes_total`, `directory_uptime_seconds`, `directory_build_info` | the directory's, `job="ipx-directory"`, from its `metrics_bind` (production `192.168.1.130:9188`, LAN only) |
| `directory_process_resident_bytes`, `directory_process_open_fds`, `directory_feed_fetches_total`, `directory_feed_bytes_total`, `directory_uptime_seconds`, `directory_build_info`, `directory_http_request_duration_seconds{method,route,status}`, `directory_feed_checks_total`, `directory_feed_checks_failed_total`, `directory_snapshots_total` | the directory's (the last four #370), `job="ipx-directory"`, from its `metrics_bind` (production `192.168.1.130:9188`, LAN only) |
| `ipx_feeds`, `ipx_downloads_pending`, `ipx_files_downloaded` | the library, as the daemon counts it every 30 s |
### The checks
@@ -208,9 +212,18 @@ a count says something happened, the lines and traces say why.
since #193, so a reap is about a second), `trim_artwork`, with what it removed on the `reap`
span. DeadAir's work on a file is a `deadair` trace (feed, enclosure, cuts) with `probe`,
`silencedetect`, `frame_bytes`, `store_plans`: seconds each, more is a file ffmpeg struggles
with. A trace with no name in `$Q traces` started in the page (#190, #192): its root is the tap
in the browser, which sends no span, so it holds the requests that tap made (API calls, the
playlist and audio of a play, position saves, the page's own report) side by side.
with. A trace begun in the page (#190, #192) has the tap or key as its root span, `tap` or
`key` (#376), from the moment it began to its last API answer, with the requests it made under
it (API calls, the playlist and audio of a play, position saves, the page's own report): a tap
long with its requests short is the page itself, drawing. The page reports the tap when the next
one begins or it is hidden, so the root arrives last; a trace with no name in `$Q traces` is
requests no tap began (a live update's, a timer's), or a tap not reported yet.
Every command the worker runs from the socket, the CLI or the page is a `command` span (`cmd`,
#373) with its job inside; a scan on the clock is a `scan` of its own. A list's sync is
`list_sync` (`list`, `listed`) with steps `upsert_listed`, `subscribe_readers`, `drop_unlisted`,
and an OPML upload `opml_import` (#372). The daemon's start is a `start` trace (#374), from
`open_database` and `assemble_config` to `daemon started`: how long a deploy took to answer.
6. **The daemon's state, from Prometheus.** Pressure and leaks that requests and traces only
hint at.
@@ -228,6 +241,8 @@ a count says something happened, the lines and traces say why.
$Q prom 'max(max_over_time(ipx_last_scan_age_seconds{job="ipx"}[$range]))' 24h
$Q prom 'sum by (kind) (increase(ipx_job_seconds_total{job="ipx"}[$range]))' 24h
$Q prom 'max(max_over_time(ipx_http_requests_in_flight{job="ipx"}[$range]))' 24h
$Q prom 'histogram_quantile(0.99, sum by (le) (rate(ipx_db_wait_seconds_bucket{job="ipx"}[$range])))' 24h
$Q prom 'histogram_quantile(0.95, sum by (le, kind) (rate(ipx_db_query_seconds_bucket{job="ipx"}[$range])))' 24h
$Q prom 'sum(increase(ipx_events_missed_total{job="ipx"}[$range]))' 24h
$Q prom 'max(max_over_time(ipx_deadair_busy_seconds{job="ipx"}[$range]))' 24h
$Q prom 'sum by (result) (increase(ipx_jev_requests_total{job="ipx"}[$range]))' 24h
@@ -246,7 +261,7 @@ a count says something happened, the lines and traces say why.
15 minutes ago, is the worker stuck, which `ipx status` does not show; its trace and the
log lines of that time say on what. A kind whose share of `ipx_job_seconds_total` jumps is
where the worker's time went (reap was most of it until #193). Web requests in flight near
the pool's max mean the pool is what they wait on. Live updates missed mean a page showed stale
the pool's max mean the pool is what they wait on, and `ipx_db_wait_seconds` for how long. Live updates missed mean a page showed stale
counts until reloaded. DeadAir over a minute on one file is a file ffmpeg chokes on.
Any `rate_limited` Jev call means the pacing is not holding ipx under TypeSafe's limits (#277); each
`jev` span has `paced_ms`, its wait for a turn. Any `malformed` Jev call is ipx asking wrongly (#257); `key_refused` is someone's key; the Signals
@@ -273,7 +288,9 @@ Caddy logs one JSON line a request (#210), logger `http.log.access.log0`: Loki's
The request's headers and a `?token=` are filtered out before it is written. Requests from
`127.0.0.1` are local checks; leave them out. `/api` and `/media` pass on to ipx, which counts
them in `ipx_http_request_duration_seconds` with its own time; Caddy's `duration` for them adds
the hop. This is the only log with a line for every request.
the hop. This is the only log with a line for every request. Caddy's own metrics, `job="ipx-web"`
(`caddy_http_request_duration_seconds{code,handler,method}` and the like), count every request too,
502s included, which ipx never sees.
```sh
W='{container="ipx-web"} |= "http.log.access" | json | __error__="" | request_remote_ip != "127.0.0.1"'
@@ -319,8 +336,8 @@ token file changes. Its lines carry their own fields and no span (#363). Its met
`directory_process_open_fds`, `directory_feed_fetches_total`, `directory_feed_bytes_total`,
`directory_uptime_seconds`, `directory_build_info`. Its traces (#213): a span a request (`http`), a child of
ipx's when ipx called with a `traceparent`, and a trace a refresher round (`refresh`, `feeds`)
with a `feed` span for each check. Its log lines carry no trace id; find the trace by time or by
the feed.
with a `feed` span for each check. Its lines inside a span carry `trace_id` and `span_id`, as
ipx's do (#375): a `feed check failed` leads to its check's span.
```sh
$Q traces '{resource.service.name="ipx-directory" && duration > 2s}'
@@ -392,7 +409,22 @@ docker ps -a --filter label=com.docker.compose.project=monitoring --format '{{.N
docker logs --since 1h tempo 2>&1 | grep -E 'failed to poll|blocklist poll complete' | tail -2
```
`ipx`, `ipx-directory` and `node` should be `up`.
`ipx`, `ipx-directory`, `ipx-web`, `node` and the monitoring's own, `prometheus`, `tempo`, `loki`
and `alloy` (#366), should be `up`. Alloy is scraped at `alloy:12345` because its command sets
`--server.http.listen-addr`; it listens on 127.0.0.1 without it.
The alert rules (#365) are `grafana/alerts.py`, written to
`/mnt/fast/arcane/projects/monitoring/grafana-provisioning/alerting/ipx.json` and loaded by
`POST /api/admin/provisioning/alerting/reload`. There is no contact point yet, so one firing is
only in Grafana: ask it which are, at the start of every check. The password is in the
monitoring project's compose file; never print it.
```sh
PW=$(grep -oP 'GF_SECURITY_ADMIN_PASSWORD=\K\S+' /mnt/fast/arcane/projects/monitoring/compose.yaml)
curl -s -u "admin:$PW" http://192.168.1.130:3444/api/prometheus/grafana/api/v1/rules | python3 -c 'import json,sys
for g in json.load(sys.stdin)["data"]["groups"]:
for r in g["rules"]: print(r["state"], r["name"])' | sort
```
Tempo looks over its stored blocks every 5 minutes. `failed to poll or create index for tenant`
means it could not, and then searches and trace lookups see only what it still holds in memory,