rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

ipx-directory traces, and ipx's trace follows its calls there (#213)

Browse Source 
The exporter setup and the traceparent parser move from ipx into ipx_directory::otel, which ipx
already depends on, so both binaries share them. The directory makes a span a request, a child of
the caller's traceparent, and a trace a refresher round with a span a feed; ipx sends the current
span's traceparent on every call to the directory and wraps each share in a span. The dashboard's
Traces row and the prod-check skill cover all three services.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-12 16:19:32 +00:00
parent a3708769a4
commit 44e75df99e
12 changed files with 188 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -10,13 +10,15 @@ Production is three containers in the Arcane project `ipx` on Tower
| container | what | logs, in Loki | traces, in Tempo | metrics, in Prometheus |
|---|---|---|---|---|
| `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | none | none |
| `iPX` | the daemon: API, scans, downloads, DeadAir | `{container="iPX"}`, JSON (`IPX_LOG_FORMAT=json`) | yes, tagged `resource.deployment.environment.name="production"` | `job="ipx"`, its `/metrics` |
| `ipx-directory` | the directory of public podcasts, `ipx-dir.sdf1.net` (#204) | `{container="ipx-directory"}`, JSON (`DIRECTORY_LOG_FORMAT=json`) | none | none |
| `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | service `ipx-web`: a span a request, passing `traceparent` on to ipx (#213) | none |
| `iPX` | the daemon: API, scans, downloads, DeadAir | `{container="iPX"}`, JSON (`IPX_LOG_FORMAT=json`) | service `ipx`; passes `traceparent` to the directory | `job="ipx"`, its `/metrics` |
| `ipx-directory` | the directory of public podcasts, `ipx-dir.sdf1.net` (#204) | `{container="ipx-directory"}`, JSON (`DIRECTORY_LOG_FORMAT=json`) | service `ipx-directory`: a span a request, joining ipx's trace when ipx called; a trace a refresher round (#213) | none |
Under them, Tower itself: `job="node"` (the node exporter: load, CPU, memory, disks). The Grafana
dashboard `iPX` draws all of it in that order (`grafana/dashboard.py`). A trace without the
production tag is a daemon run by hand, not production. Alloy ships the logs; Loki, Tempo and
production tag (`resource.deployment.environment.name="production"`, on all three) is a daemon run
by hand, not production. Since #213 a request is one trace across the services it touched: a
Directory page is ipx-web, then ipx, then the directory, so start from the trace, not the logs. Alloy ships the logs; Loki, Tempo and
Prometheus publish no query port to this machine, so use the script beside this file:
```sh
@@ -151,7 +153,7 @@ a count says something happened, the lines and traces say why.
`feeds` above 0; none at all for over 10 minutes means the worker is stuck. And `pending` is
the real queue (#113): files a scan will download on its own. Back-catalogue files are
`held`, listed but not counted, so it is usually 0 or a handful.
5. **Slow and failed traces.**
5. **Slow and failed traces.** All three services' (filter by `resource.service.name` for one).
```
$Q traces '{resource.deployment.environment.name="production" && duration > 5s}'
$Q traces '{resource.deployment.environment.name="production" && status = error}'
@@ -243,6 +245,15 @@ up in ipx's access log. 401 and 303 on `/` are people signed out being sent to `
problem. Caddy's own notices at each start (no admin endpoint, no HTTPS or HTTP/3, its goodbye on
SIGTERM) are by design and filtered out above.
Each access line carries the request's `traceID` (`| json` gives `traceID`): give it to `$Q trace`
to see where the time went, Caddy's span with ipx's inside it. A slow page with ipx's span short is
Caddy or the network; a long one is ipx.
```sh
$Q traces '{resource.service.name="ipx-web" && duration > 2s}'
$Q traces '{resource.service.name="ipx-web" && status = error}'
```
Who reaches it: `request_remote_ip` is `192.168.64.1` for the tunnel and anything else on Tower
(Docker's gateway), `192.168.1.130` for Tower itself, a LAN address for the LAN. Caddy passes
Cloudflare Access's headers on only from `192.168.64.1` (`TUNNEL_PEER`), and ipx trusts them only
@@ -257,7 +268,19 @@ It logs a line a request (`target` `ipx_directory::http`: `method`, `path`, `sta
line a refresher round (`feeds checked`: `checked`), a line a failed feed check (`feed check
failed`: `feed`, the directory's id, and `error`), the admin's actions (`feed removed and
blocked`, `feed unblocked`, `snapshot dropped`, each with `by`), and `config read again` when its
token file changes. It sends no traces and has no metrics.
token file changes. It has no metrics. Its traces (#213): a span a request (`http`), a child of
ipx's when ipx called with a `traceparent`, and a trace a refresher round (`refresh`, `feeds`)
with a `feed` span for each check. Its log lines carry no trace id; find the trace by time or by
the feed.
```sh
$Q traces '{resource.service.name="ipx-directory" && duration > 2s}'
$Q traces '{resource.service.name="ipx-directory" && name="feed" && duration > 10s}'
$Q traces '{resource.service.name="ipx" && name="directory_share"}'
```
A slow Directory page in ipx is usually its `http` span in the directory: the trace says whether.
A refresher round far over a minute is one publisher's feed holding it, the slow `feed` span.
```sh
D='{container="ipx-directory"} | json | __error__=""'