ipx-play: metrics for Prometheus, a dashboard row and a down alert (#458)

ipx-play sent traces and logged JSON but served no /metrics, so nothing showed whether
shared links were opening or failing. It now serves them on metrics_bind (9189 in
production, off the tunnel's port), with the directory's histogram and /proc reading:
requests by method, route and status (by route, so each shared episode is not a row of its
own, and an unknown address is route=""), episodes shared, memory, open files, uptime and
version.

The iPX dashboard gets an ipx-play row, its warnings join the other containers' log panel,
and alerts.py fires when Prometheus cannot reach it. The prod-check skill and docs/play.md
name it, with the scrape job and the OTEL_RESOURCE_ATTRIBUTES tag the trace panels need.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-11-13 22:08:11 -05:00
parent 47e093e313
commit 5d0ff1ac57
10 changed files with 209 additions and 15 deletions

View File

@@ -32,10 +32,13 @@ The compose service, beside `ipx-directory`:
environment:
PLAY_LOG_FORMAT: json
OTEL_EXPORTER_OTLP_ENDPOINT: http://192.168.1.130:4318 # Tempo is not on this network
# The dashboard's trace panels show only spans with this tag, as the others' are.
OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=production
volumes:
- /mnt/fast/appdata/ipx-play:/config
ports:
- "8102:8102"
- "9189:9189" # /metrics, for Prometheus on the LAN; not in the tunnel
```
Its config, `/mnt/fast/appdata/ipx-play/play.toml`:
@@ -43,6 +46,9 @@ Its config, `/mnt/fast/appdata/ipx-play/play.toml`:
```toml
bind = "0.0.0.0:8102"
public_url = "https://play.ipxreader.app"
# Prometheus metrics, on a port of their own and not the one the tunnel publishes. Empty, or
# left out, serves none.
metrics_bind = "0.0.0.0:9189"
# One for each ipx server allowed to make links; `ipx-play token <name>` prints the lines.
[[tokens]]
@@ -60,6 +66,27 @@ The tunnel publishes `play.ipxreader.app` to `http://192.168.1.130:8102` with no
the pages are anyone's. cloudflared is not on the project's network, so not `ipx-play:8102`, as
the directory's route is the host's port too. `/v1` takes a token for everything.
## Watching it
Its traces go to Tempo as service `ipx-play`: a span a request, a child of ipx's when ipx shared
an episode, so one trace runs from the page's Share through ipx to here. Its log is a JSON line a
request (`target` `ipx_play::http`: `method`, `path`, `status`, `duration_ms`, `token`, the
server's name and never its token) and one a share (`an episode was shared`: `server`, `id`).
Its metrics (#458), on `metrics_bind`, scraped as `job="ipx-play"`, beside the directory's job in
Prometheus's config:
```yaml
- job_name: ipx-play
static_configs:
- targets: ["192.168.1.130:9189"]
```
`play_http_request_duration_seconds{method,route,status}` (a page opened is `route="/e/{id}"`, an
address no route has `route=""`), `play_shares_total`, `play_process_resident_bytes`,
`play_process_open_fds`, `play_uptime_seconds` and `play_build_info`. The iPX dashboard's
`ipx-play` row draws them, and the alert `ipx-play is down` fires when Prometheus cannot reach it.
## Setting up ipx
In Admin, Directory, Sharing episodes: the address ipx reaches it at, `http://ipx-play:8102`