ipx-play: metrics for Prometheus, a dashboard row and a down alert (#458)

ipx-play sent traces and logged JSON but served no /metrics, so nothing showed whether
shared links were opening or failing. It now serves them on metrics_bind (9189 in
production, off the tunnel's port), with the directory's histogram and /proc reading:
requests by method, route and status (by route, so each shared episode is not a row of its
own, and an unknown address is route=""), episodes shared, memory, open files, uptime and
version.

The iPX dashboard gets an ipx-play row, its warnings join the other containers' log panel,
and alerts.py fires when Prometheus cannot reach it. The prod-check skill and docs/play.md
name it, with the scrape job and the OTEL_RESOURCE_ATTRIBUTES tag the trace panels need.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-11-13 22:08:11 -05:00
parent 47e093e313
commit 5d0ff1ac57
10 changed files with 209 additions and 15 deletions

View File

@@ -41,6 +41,8 @@ rules = [
"Prometheus cannot reach ipx's /metrics: the iPX container is down or restarting.", no_data="Alerting"),
rule("directory-down", "ipx-directory is down", 'up{job="ipx-directory"}', "lt", 1, "5m",
"Prometheus cannot reach the directory's /metrics.", no_data="Alerting"),
rule("play-down", "ipx-play is down", 'up{job="ipx-play"}', "lt", 1, "5m",
"Prometheus cannot reach ipx-play's /metrics: shared links are likely not opening.", no_data="Alerting"),
rule("web-down", "ipx-web is down", 'up{job="ipx-web"}', "lt", 1, "5m",
"Prometheus cannot reach Caddy's /metrics: the page is likely down too.", no_data="Alerting"),
# The worker: the daemon answers `ipxd status` while it is stuck, so only these say so.