ipx-play: metrics for Prometheus, a dashboard row and a down alert (#458)

ipx-play sent traces and logged JSON but served no /metrics, so nothing showed whether
shared links were opening or failing. It now serves them on metrics_bind (9189 in
production, off the tunnel's port), with the directory's histogram and /proc reading:
requests by method, route and status (by route, so each shared episode is not a row of its
own, and an unknown address is route=""), episodes shared, memory, open files, uptime and
version.

The iPX dashboard gets an ipx-play row, its warnings join the other containers' log panel,
and alerts.py fires when Prometheus cannot reach it. The prod-check skill and docs/play.md
name it, with the scrape job and the OTEL_RESOURCE_ATTRIBUTES tag the trace panels need.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.claude/skills/ipx-prod-check/SKILL.md

@@ -1,11 +1,11 @@
 ---
 name: ipx-prod-check
-description: Look for problems in production ipx (the ipx-web, iPX and ipx-directory containers on Tower, Tower itself and the monitoring that watches them) from their logs in Loki, their traces in Tempo and the metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site.
+description: Look for problems in production ipx (the ipx-web, iPX, ipx-directory and ipx-play containers on Tower, Tower itself and the monitoring that watches them) from their logs in Loki, their traces in Tempo and the metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site.
 ---
 
 # Checking production ipx
 
-Production is three containers in the Arcane project `ipx` on Tower
+Production is four containers in the Arcane project `ipx` on Tower
 (`/mnt/fast/arcane/projects/ipx/compose.yaml`), one request passing through them in this order:
 
 | container | what | logs, in Loki | traces, in Tempo | metrics, in Prometheus |
@@ -13,11 +13,12 @@ Production is three containers in the Arcane project `ipx` on Tower
 | `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | service `ipx-web`: a span a request, passing `traceparent` on to ipx (#213) | `job="ipx-web"`: Caddy's own `caddy_http_*`, from `METRICS_BIND`, published as `192.168.1.130:9190` (#371) |
 | `iPX` | the daemon: API, scans, downloads, DeadAir | `{container="iPX"}`, JSON (`IPX_LOG_FORMAT=json`) | service `ipx`; passes `traceparent` to the directory | `job="ipx"`, its `/metrics` |
 | `ipx-directory` | the directory of public podcasts, `directory.ipxreader.app` (#204, #214) | `{container="ipx-directory"}`, JSON (`DIRECTORY_LOG_FORMAT=json`) | service `ipx-directory`: a span a request, joining ipx's trace when ipx called; a trace a refresher round (#213) | `job="ipx-directory"`, its `/metrics` |
+| `ipx-play` | the page a shared episode opens, `play.ipxreader.app` (#447); ipx sends it the episodes shared | `{container="ipx-play"}`, JSON (`PLAY_LOG_FORMAT=json`): a line a request (`ipx_play::http`), one a share | service `ipx-play`: a span a request, joining ipx's trace on a share | `job="ipx-play"`, its `/metrics` on `metrics_bind` (production `192.168.1.130:9189`, LAN only, #458) |
 
 Grafana's alert rules (`grafana/alerts.py`, #365) watch all of it between checks: start by asking
 which are firing (the monitoring section). Under them, Tower itself: `job="node"` (the node exporter: load, CPU, memory, disks). The Grafana
 dashboard `iPX` draws all of it in that order (`grafana/dashboard.py`). A trace without the
-production tag (`resource.deployment.environment.name="production"`, on all three) is a daemon run
+production tag (`resource.deployment.environment.name="production"`, on all four) is a daemon run
 by hand, not production. Since #213 a request is one trace across the services it touched: a
 Directory page is ipx-web, then ipx, then the directory, so start from the trace, not the logs. Alloy ships the logs; Loki, Tempo and
 Prometheus publish no query port to this machine, so use the script beside this file:
@@ -38,7 +39,7 @@ $Q flame   '<TraceQL>'  [since] [out.svg] # where the matching spans' time went,
 "Since the last deploy" is since that container started (each deploys on its own):
 
 ```sh
-for c in iPX ipx-web ipx-directory; do S=$(docker inspect $c -f '{{.State.StartedAt}}'); echo "$c $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m"; done
+for c in iPX ipx-web ipx-directory ipx-play; do S=$(docker inspect $c -f '{{.State.StartedAt}}'); echo "$c $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m"; done
 ```
 
 Start every check by seeing that all three run, since Loki cannot see a container that is not:
@@ -422,12 +423,12 @@ container's lines are reaching Loki:
 ```sh
 curl -s http://192.168.1.130:9090/api/v1/targets | python3 -c 'import json,sys
 for t in json.load(sys.stdin)["data"]["activeTargets"]: print(t["health"], t["labels"]["job"], t["scrapeUrl"], t.get("lastError","")[:80])'
-for c in iPX ipx-web ipx-directory; do echo "$c $($Q metric "sum(count_over_time({container=\"$c\"} [24h]))" 24h | tail -1)"; done
+for c in iPX ipx-web ipx-directory ipx-play; do echo "$c $($Q metric "sum(count_over_time({container=\"$c\"} [24h]))" 24h | tail -1)"; done
 docker ps -a --filter label=com.docker.compose.project=monitoring --format '{{.Names}} {{.Status}}'
 docker logs --since 1h tempo 2>&1 | grep -E 'failed to poll|blocklist poll complete' | tail -2
 ```
 
-`ipx`, `ipx-directory`, `ipx-web`, `node` and the monitoring's own, `prometheus`, `tempo`, `loki`
+`ipx`, `ipx-directory`, `ipx-play`, `ipx-web`, `node` and the monitoring's own, `prometheus`, `tempo`, `loki`
 and `alloy` (#366), should be `up`. Alloy is scraped at `alloy:12345` because its command sets
 `--server.http.listen-addr`; it listens on 127.0.0.1 without it.