ipx-play: metrics for Prometheus, a dashboard row and a down alert (#458)

ipx-play sent traces and logged JSON but served no /metrics, so nothing showed whether
shared links were opening or failing. It now serves them on metrics_bind (9189 in
production, off the tunnel's port), with the directory's histogram and /proc reading:
requests by method, route and status (by route, so each shared episode is not a row of its
own, and an unknown address is route=""), episodes shared, memory, open files, uptime and
version.

The iPX dashboard gets an ipx-play row, its warnings join the other containers' log panel,
and alerts.py fires when Prometheus cannot reach it. The prod-check skill and docs/play.md
name it, with the scrape job and the OTEL_RESOURCE_ATTRIBUTES tag the trace panels need.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-11-13 22:08:11 -05:00
parent 47e093e313
commit 5d0ff1ac57
10 changed files with 209 additions and 15 deletions

View File

@@ -1,11 +1,11 @@
--- ---
name: ipx-prod-check name: ipx-prod-check
description: Look for problems in production ipx (the ipx-web, iPX and ipx-directory containers on Tower, Tower itself and the monitoring that watches them) from their logs in Loki, their traces in Tempo and the metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site. description: Look for problems in production ipx (the ipx-web, iPX, ipx-directory and ipx-play containers on Tower, Tower itself and the monitoring that watches them) from their logs in Loki, their traces in Tempo and the metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site.
--- ---
# Checking production ipx # Checking production ipx
Production is three containers in the Arcane project `ipx` on Tower Production is four containers in the Arcane project `ipx` on Tower
(`/mnt/fast/arcane/projects/ipx/compose.yaml`), one request passing through them in this order: (`/mnt/fast/arcane/projects/ipx/compose.yaml`), one request passing through them in this order:
| container | what | logs, in Loki | traces, in Tempo | metrics, in Prometheus | | container | what | logs, in Loki | traces, in Tempo | metrics, in Prometheus |
@@ -13,11 +13,12 @@ Production is three containers in the Arcane project `ipx` on Tower
| `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | service `ipx-web`: a span a request, passing `traceparent` on to ipx (#213) | `job="ipx-web"`: Caddy's own `caddy_http_*`, from `METRICS_BIND`, published as `192.168.1.130:9190` (#371) | | `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | service `ipx-web`: a span a request, passing `traceparent` on to ipx (#213) | `job="ipx-web"`: Caddy's own `caddy_http_*`, from `METRICS_BIND`, published as `192.168.1.130:9190` (#371) |
| `iPX` | the daemon: API, scans, downloads, DeadAir | `{container="iPX"}`, JSON (`IPX_LOG_FORMAT=json`) | service `ipx`; passes `traceparent` to the directory | `job="ipx"`, its `/metrics` | | `iPX` | the daemon: API, scans, downloads, DeadAir | `{container="iPX"}`, JSON (`IPX_LOG_FORMAT=json`) | service `ipx`; passes `traceparent` to the directory | `job="ipx"`, its `/metrics` |
| `ipx-directory` | the directory of public podcasts, `directory.ipxreader.app` (#204, #214) | `{container="ipx-directory"}`, JSON (`DIRECTORY_LOG_FORMAT=json`) | service `ipx-directory`: a span a request, joining ipx's trace when ipx called; a trace a refresher round (#213) | `job="ipx-directory"`, its `/metrics` | | `ipx-directory` | the directory of public podcasts, `directory.ipxreader.app` (#204, #214) | `{container="ipx-directory"}`, JSON (`DIRECTORY_LOG_FORMAT=json`) | service `ipx-directory`: a span a request, joining ipx's trace when ipx called; a trace a refresher round (#213) | `job="ipx-directory"`, its `/metrics` |
| `ipx-play` | the page a shared episode opens, `play.ipxreader.app` (#447); ipx sends it the episodes shared | `{container="ipx-play"}`, JSON (`PLAY_LOG_FORMAT=json`): a line a request (`ipx_play::http`), one a share | service `ipx-play`: a span a request, joining ipx's trace on a share | `job="ipx-play"`, its `/metrics` on `metrics_bind` (production `192.168.1.130:9189`, LAN only, #458) |
Grafana's alert rules (`grafana/alerts.py`, #365) watch all of it between checks: start by asking Grafana's alert rules (`grafana/alerts.py`, #365) watch all of it between checks: start by asking
which are firing (the monitoring section). Under them, Tower itself: `job="node"` (the node exporter: load, CPU, memory, disks). The Grafana which are firing (the monitoring section). Under them, Tower itself: `job="node"` (the node exporter: load, CPU, memory, disks). The Grafana
dashboard `iPX` draws all of it in that order (`grafana/dashboard.py`). A trace without the dashboard `iPX` draws all of it in that order (`grafana/dashboard.py`). A trace without the
production tag (`resource.deployment.environment.name="production"`, on all three) is a daemon run production tag (`resource.deployment.environment.name="production"`, on all four) is a daemon run
by hand, not production. Since #213 a request is one trace across the services it touched: a by hand, not production. Since #213 a request is one trace across the services it touched: a
Directory page is ipx-web, then ipx, then the directory, so start from the trace, not the logs. Alloy ships the logs; Loki, Tempo and Directory page is ipx-web, then ipx, then the directory, so start from the trace, not the logs. Alloy ships the logs; Loki, Tempo and
Prometheus publish no query port to this machine, so use the script beside this file: Prometheus publish no query port to this machine, so use the script beside this file:
@@ -38,7 +39,7 @@ $Q flame '<TraceQL>' [since] [out.svg] # where the matching spans' time went,
"Since the last deploy" is since that container started (each deploys on its own): "Since the last deploy" is since that container started (each deploys on its own):
```sh ```sh
for c in iPX ipx-web ipx-directory; do S=$(docker inspect $c -f '{{.State.StartedAt}}'); echo "$c $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m"; done for c in iPX ipx-web ipx-directory ipx-play; do S=$(docker inspect $c -f '{{.State.StartedAt}}'); echo "$c $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m"; done
``` ```
Start every check by seeing that all three run, since Loki cannot see a container that is not: Start every check by seeing that all three run, since Loki cannot see a container that is not:
@@ -422,12 +423,12 @@ container's lines are reaching Loki:
```sh ```sh
curl -s http://192.168.1.130:9090/api/v1/targets | python3 -c 'import json,sys curl -s http://192.168.1.130:9090/api/v1/targets | python3 -c 'import json,sys
for t in json.load(sys.stdin)["data"]["activeTargets"]: print(t["health"], t["labels"]["job"], t["scrapeUrl"], t.get("lastError","")[:80])' for t in json.load(sys.stdin)["data"]["activeTargets"]: print(t["health"], t["labels"]["job"], t["scrapeUrl"], t.get("lastError","")[:80])'
for c in iPX ipx-web ipx-directory; do echo "$c $($Q metric "sum(count_over_time({container=\"$c\"} [24h]))" 24h | tail -1)"; done for c in iPX ipx-web ipx-directory ipx-play; do echo "$c $($Q metric "sum(count_over_time({container=\"$c\"} [24h]))" 24h | tail -1)"; done
docker ps -a --filter label=com.docker.compose.project=monitoring --format '{{.Names}} {{.Status}}' docker ps -a --filter label=com.docker.compose.project=monitoring --format '{{.Names}} {{.Status}}'
docker logs --since 1h tempo 2>&1 | grep -E 'failed to poll|blocklist poll complete' | tail -2 docker logs --since 1h tempo 2>&1 | grep -E 'failed to poll|blocklist poll complete' | tail -2
``` ```
`ipx`, `ipx-directory`, `ipx-web`, `node` and the monitoring's own, `prometheus`, `tempo`, `loki` `ipx`, `ipx-directory`, `ipx-play`, `ipx-web`, `node` and the monitoring's own, `prometheus`, `tempo`, `loki`
and `alloy` (#366), should be `up`. Alloy is scraped at `alloy:12345` because its command sets and `alloy` (#366), should be `up`. Alloy is scraped at `alloy:12345` because its command sets
`--server.http.listen-addr`; it listens on 127.0.0.1 without it. `--server.http.listen-addr`; it listens on 127.0.0.1 without it.

View File

@@ -20,7 +20,7 @@ service of the `content` project, beside audiobookshelf, immich and Plex (#208).
| Old database | `/mnt/user/ipx/state.db`, SQLite, used until the move to Postgres on 2026-09-18 and kept for rollback | `/data/state.db` | | Old database | `/mnt/user/ipx/state.db`, SQLite, used until the move to Postgres on 2026-09-18 and kept for rollback | `/data/state.db` |
| Downloads | `/mnt/user/ipx/downloads` | `/downloads` | | Downloads | `/mnt/user/ipx/downloads` | `/downloads` |
| Directory | `ipx-directory` in the same project, `192.168.1.130:8100`, public at `directory.ipxreader.app` through the tunnel with no Access in front: its pages are anyone's to browse, read only (#215), and its API (`/v1`) takes a token for everything. Until 2026-10-12 it was `ipx-dir.sdf1.net` (#214). Config `/mnt/fast/appdata/ipx-directory/directory.toml` (token hashes; Tower's own token in `tower.token` beside it, and entered in ipx's Admin, Directory, which keeps it in the database's `settings`, with the address ipx reaches it at: `http://ipx-directory:8100`, inside Docker, not the public address, which went out through Cloudflare and the tunnel, 170-340 ms a call where the directory answers in 16 ms), database `ipx_directory` on the same Postgres, URL in `directory.env` beside the compose file. See [docs/directory.md](docs/directory.md) | `/config/directory.toml` | | Directory | `ipx-directory` in the same project, `192.168.1.130:8100`, public at `directory.ipxreader.app` through the tunnel with no Access in front: its pages are anyone's to browse, read only (#215), and its API (`/v1`) takes a token for everything. Until 2026-10-12 it was `ipx-dir.sdf1.net` (#214). Config `/mnt/fast/appdata/ipx-directory/directory.toml` (token hashes; Tower's own token in `tower.token` beside it, and entered in ipx's Admin, Directory, which keeps it in the database's `settings`, with the address ipx reaches it at: `http://ipx-directory:8100`, inside Docker, not the public address, which went out through Cloudflare and the tunnel, 170-340 ms a call where the directory answers in 16 ms), database `ipx_directory` on the same Postgres, URL in `directory.env` beside the compose file. See [docs/directory.md](docs/directory.md) | `/config/directory.toml` |
| Shared episodes | `ipx-play` in the same project, `192.168.1.130:8102`, public at `play.ipxreader.app` through the tunnel with no Access in front: the page a shared episode opens, anyone's to see (#447). `/v1` takes a token. Config `/mnt/fast/appdata/ipx-play/play.toml` (token hashes; Tower's token entered in ipx's Admin, Directory, Sharing episodes, with the address `http://ipx-play:8102`), database `ipx_play` on the same Postgres, URL in `play.env` beside the compose file. See [docs/play.md](docs/play.md) | `/config/play.toml` | | Shared episodes | `ipx-play` in the same project, `192.168.1.130:8102`, public at `play.ipxreader.app` through the tunnel with no Access in front: the page a shared episode opens, anyone's to see (#447). `/v1` takes a token. Config `/mnt/fast/appdata/ipx-play/play.toml` (token hashes; Tower's token entered in ipx's Admin, Directory, Sharing episodes, with the address `http://ipx-play:8102`), database `ipx_play` on the same Postgres, URL in `play.env` beside the compose file. Metrics for Prometheus on `192.168.1.130:9189`, LAN only, as `job="ipx-play"` (#458). See [docs/play.md](docs/play.md) | `/config/play.toml` |
| Home page | `ipxreader.app` (and `www`): a static page from `site/`, on the Cloudflare Pages project `ipxreader`, not on Tower. See below | | | Home page | `ipxreader.app` (and `www`): a static page from `site/`, on the Cloudflare Pages project `ipxreader`, not on Tower. See below | |
| Web UI | `ipx-web` (Caddy) in the same project, at `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel. It serves the page and passes `/api` and `/media` to ipx, whose own port is published nowhere; its address on `ipx_default` is pinned to `192.168.64.10` (#203) | ipx `0.0.0.0:8099`, Caddy `:8080` | | Web UI | `ipx-web` (Caddy) in the same project, at `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel. It serves the page and passes `/api` and `/media` to ipx, whose own port is published nowhere; its address on `ipx_default` is pinned to `192.168.64.10` (#203) | ipx `0.0.0.0:8099`, Caddy `:8080` |
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | Caddy passes `Cf-Access-*` only from `192.168.64.1`, the `ipx_default` gateway cloudflared arrives from (`TUNNEL_PEER`; the subnet is pinned in the compose file); ipx trusts them only from Caddy, `192.168.64.10`, and only with a valid `Cf-Access-Jwt-Assertion` (`access_team`, `access_aud`) | | Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | Caddy passes `Cf-Access-*` only from `192.168.64.1`, the `ipx_default` gateway cloudflared arrives from (`TUNNEL_PEER`; the subnet is pinned in the compose file); ipx trusts them only from Caddy, `192.168.64.10`, and only with a valid `Cf-Access-Jwt-Assertion` (`access_team`, `access_aud`) |

View File

@@ -14,5 +14,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- A page for each shared episode, at `/e/<id>`: its show, title, date, length and notes, a player that plays the publisher's own file from the time it was shared at (`?t=754` or `?t=12:34`), buttons to subscribe in Apple Podcasts, Overcast, Pocket Casts or Castro, the feed's address, and a preview where the link is pasted (#447). - A page for each shared episode, at `/e/<id>`: its show, title, date, length and notes, a player that plays the publisher's own file from the time it was shared at (`?t=754` or `?t=12:34`), buttons to subscribe in Apple Podcasts, Overcast, Pocket Casts or Castro, the feed's address, and a preview where the link is pasted (#447).
- `POST /v1/shares`, for an ipx server with a token from `ipx-play token <name>`: an episode in, its short link out, the same link however often and from whichever server it is shared. An address that looks like a private or paid feed's is refused (#447). - `POST /v1/shares`, for an ipx server with a token from `ipx-play token <name>`: an episode in, its short link out, the same link however often and from whichever server it is shared. An address that looks like a private or paid feed's is refused (#447).
- Metrics for Prometheus on a port of its own, `metrics_bind`: requests by route and status and their time, episodes shared, memory (#458).
[unreleased]: https://git.sdf1.net/rays/ipx/commits/main/crates/play [unreleased]: https://git.sdf1.net/rays/ipx/commits/main/crates/play

View File

@@ -15,6 +15,10 @@ pub struct Config {
/// The environment variable holding the database URL, so the file holds no secret. /// The environment variable holding the database URL, so the file holds no secret.
#[serde(default = "default_database_url_env")] #[serde(default = "default_database_url_env")]
pub database_url_env: String, pub database_url_env: String,
/// Where to serve /metrics for Prometheus (#458), apart from `bind`, which the tunnel
/// publishes: `0.0.0.0:9189`. Empty serves none.
#[serde(default)]
pub metrics_bind: String,
/// The address people open, which the links ipx is handed begin with. /// The address people open, which the links ipx is handed begin with.
#[serde(default = "default_public_url")] #[serde(default = "default_public_url")]
pub public_url: String, pub public_url: String,

View File

@@ -9,10 +9,11 @@
//! its file is, nothing more. //! its file is, nothing more.
pub mod config; pub mod config;
pub mod metrics;
mod page; mod page;
pub mod store; pub mod store;
use axum::extract::{Path, Query, Request, State}; use axum::extract::{MatchedPath, Path, Query, Request, State};
use axum::http::{StatusCode, header}; use axum::http::{StatusCode, header};
use axum::middleware::Next; use axum::middleware::Next;
use axum::response::{IntoResponse, Response}; use axum::response::{IntoResponse, Response};
@@ -160,6 +161,7 @@ async fn share(State(st): State<St>, Extension(who): Extension<Who>, Json(mut e)
} }
match st.store.put(&who.0, &e).await { match st.store.put(&who.0, &e).await {
Ok(id) => { Ok(id) => {
metrics::SHARES.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
tracing::info!(server = %who.0, id = %id, "an episode was shared"); tracing::info!(server = %who.0, id = %id, "an episode was shared");
Json(Shared { url: format!("{}/e/{id}", st.base), id }).into_response() Json(Shared { url: format!("{}/e/{id}", st.base), id }).into_response()
} }
@@ -198,6 +200,9 @@ async fn healthz(State(st): State<St>) -> Response {
/// calls here; and a line, with the server's name and never its token. /// calls here; and a line, with the server's name and never its token.
async fn access_log(req: Request, next: Next) -> Response { async fn access_log(req: Request, next: Next) -> Response {
let (method, path) = (req.method().clone(), req.uri().path().to_owned()); let (method, path) = (req.method().clone(), req.uri().path().to_owned());
// By route, not path: by path each shared episode would be a row of its own. A path no route
// has is "", which someone trying addresses cannot multiply.
let route = req.extensions().get::<MatchedPath>().map_or("", |r| r.as_str()).to_owned();
let start = Instant::now(); let start = Instant::now();
let span = if path == "/healthz" { let span = if path == "/healthz" {
tracing::Span::none() tracing::Span::none()
@@ -218,6 +223,14 @@ async fn access_log(req: Request, next: Next) -> Response {
let status = resp.status().as_u16(); let status = resp.status().as_u16();
span.record("http.response.status_code", status); span.record("http.response.status_code", status);
if path != "/healthz" { if path != "/healthz" {
metrics::REQUESTS.observe(
&format!(
"method=\"{}\",route=\"{}\",status=\"{status}\"",
ipx_directory::metrics::method(method.as_str()),
ipx_directory::metrics::label(&route)
),
start.elapsed().as_secs_f64(),
);
let _in = span.enter(); let _in = span.enter();
let who = resp.extensions().get::<Who>().map_or("", |w| w.0.as_str()).to_owned(); let who = resp.extensions().get::<Who>().map_or("", |w| w.0.as_str()).to_owned();
tracing::info!( tracing::info!(
@@ -301,6 +314,18 @@ mod tests {
assert!(!html.contains("javascript:"), "a link that is not a web address is dropped"); assert!(!html.contains("javascript:"), "a link that is not a web address is dropped");
assert_eq!(get(&app, "/e/nothere").await.0, StatusCode::NOT_FOUND); assert_eq!(get(&app, "/e/nothere").await.0, StatusCode::NOT_FOUND);
get(&app, "/no/such/page").await;
let mut out = String::new();
metrics::REQUESTS.render(&mut out, "r", "");
for row in [
r#"r_count{method="GET",route="/e/{id}",status="200"}"#,
r#"r_count{method="POST",route="/v1/shares",status="200"}"#,
r#"r_count{method="GET",route="",status="404"}"#,
] {
assert!(out.contains(row), "requests are counted by route, not path: {row} in {out}");
}
assert!(!out.contains("nothere") && !out.contains("/no/such"), "{out}");
} }
#[tokio::test] #[tokio::test]

View File

@@ -98,6 +98,13 @@ async fn serve(path: &Path) -> Result<()> {
.with_context(|| format!("{} is not set: ipx-play needs a database URL", cfg.database_url_env))?; .with_context(|| format!("{} is not set: ipx-play needs a database URL", cfg.database_url_env))?;
let store = Store::open(&url, true).await.context("opening the database")?; let store = Store::open(&url, true).await.context("opening the database")?;
let listener = TcpListener::bind(&cfg.bind).await.with_context(|| format!("listening on {}", cfg.bind))?; let listener = TcpListener::bind(&cfg.bind).await.with_context(|| format!("listening on {}", cfg.bind))?;
if !cfg.metrics_bind.is_empty() {
let metrics = TcpListener::bind(&cfg.metrics_bind)
.await
.with_context(|| format!("listening for metrics on {}", cfg.metrics_bind))?;
tracing::info!(bind = %cfg.metrics_bind, "serving metrics for Prometheus at /metrics");
tokio::spawn(ipx_play::metrics::serve(metrics));
}
tracing::info!(version = env!("CARGO_PKG_VERSION"), bind = %cfg.bind, tokens = tokens.len(), "ipx-play started"); tracing::info!(version = env!("CARGO_PKG_VERSION"), bind = %cfg.bind, tokens = tokens.len(), "ipx-play started");
let app = ipx_play::router(Play { store, tokens, base: cfg.base().to_owned() }); let app = ipx_play::router(Play { store, tokens, base: cfg.base().to_owned() });
axum::serve(listener, app) axum::serve(listener, app)

View File

@@ -0,0 +1,99 @@
//! ipx-play's /metrics (#458), in Prometheus's text format: memory, open files, the requests it
//! answered and the episodes it was given to share. The histogram and /proc reading are the
//! directory's, so the two read alike on the dashboard.
use axum::Router;
use ipx_directory::metrics::{Histograms, SECONDS, process};
use std::sync::atomic::AtomicU64;
use std::sync::atomic::Ordering::Relaxed;
use std::time::Instant;
use tokio::net::TcpListener;
/// Requests answered, by method, route and status: a page opened is `route="/e/{id}"`.
pub static REQUESTS: Histograms = Histograms::new(SECONDS);
/// Episodes an ipx server sent to be shared, the same one again included.
pub static SHARES: AtomicU64 = AtomicU64::new(0);
/// Nothing from the database, so a scrape costs nothing and answers while it is down.
fn render(started: Instant) -> String {
let mut out = String::new();
let mut gauge = |name: &str, kind: &str, help: &str, labels: &str, v: f64| {
out.push_str(&format!("# HELP {name} {help}\n# TYPE {name} {kind}\n{name}{labels} {v}\n"));
};
let (rss, fds) = process();
if let Some(v) = rss {
gauge("play_process_resident_bytes", "gauge", "Memory the process holds.", "", v);
}
if let Some(v) = fds {
gauge("play_process_open_fds", "gauge", "Files and sockets the process has open.", "", v);
}
gauge("play_uptime_seconds", "gauge", "Seconds since ipx-play started.", "", started.elapsed().as_secs() as f64);
gauge(
"play_build_info",
"gauge",
"The version running, as a label.",
&format!("{{version=\"{}\"}}", env!("CARGO_PKG_VERSION")),
1.0,
);
gauge("play_shares_total", "counter", "Episodes ipx servers sent to be shared.", "", SHARES.load(Relaxed) as f64);
REQUESTS.render(
&mut out,
"play_http_request_duration_seconds",
"Requests answered and the seconds each took, by method, route and status.",
);
out
}
/// Serves /metrics on a listener of its own: the public port is behind the tunnel with no Access
/// in front, and these are for the LAN.
// ponytail: no gzip, unlike the directory's (#253): a few dozen lines a scrape. Add tower-http's
// CompressionLayer if the routes grow.
pub async fn serve(listener: TcpListener) {
let started = Instant::now();
let app = Router::new().route(
"/metrics",
axum::routing::get(move || async move {
([(axum::http::header::CONTENT_TYPE, "text/plain; version=0.0.4")], render(started))
}),
);
if let Err(e) = axum::serve(listener, app).await {
tracing::error!(error = %e, "metrics stopped");
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn every_play_metric_grafana_queries_is_served() {
// A row for each name, so a histogram's lines are there to find.
REQUESTS.observe("method=\"GET\",route=\"/e/{id}\",status=\"200\"", 0.01);
let l = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let at = l.local_addr().unwrap();
tokio::spawn(serve(l));
let text = reqwest::get(format!("http://{at}/metrics")).await.unwrap().text().await.unwrap();
let src = [include_str!("../../../grafana/dashboard.py"), include_str!("../../../grafana/alerts.py")].concat();
let names: std::collections::BTreeSet<&str> = src
.match_indices("play_")
.filter(|(i, _)| !src[..*i].ends_with(|c: char| c.is_ascii_alphanumeric() || c == '_'))
.filter_map(|(i, _)| {
let rest = &src[i..];
let end = rest.find(|c: char| !(c.is_ascii_lowercase() || c == '_')).unwrap_or(rest.len());
// A query names one with its labels, or the dashboard keeps it in a string; ipx's
// events, such as play_start in a panel's description, are neither.
rest[end..].starts_with(['{', '"']).then(|| &rest[..end])
})
// A histogram is queried by its series' names and served under its own.
.map(|n| n.trim_end_matches("_bucket").trim_end_matches("_count").trim_end_matches("_sum"))
.collect();
assert!(names.len() >= 3, "{names:?}");
for n in names {
// Memory comes from /proc, which a Mac has not.
if n.starts_with("play_process_") && process().0.is_none() {
continue;
}
assert!(text.contains(&format!("# TYPE {n} ")), "{n} is queried by grafana/ but not served: {text}");
}
}
}

View File

@@ -32,10 +32,13 @@ The compose service, beside `ipx-directory`:
environment: environment:
PLAY_LOG_FORMAT: json PLAY_LOG_FORMAT: json
OTEL_EXPORTER_OTLP_ENDPOINT: http://192.168.1.130:4318 # Tempo is not on this network OTEL_EXPORTER_OTLP_ENDPOINT: http://192.168.1.130:4318 # Tempo is not on this network
# The dashboard's trace panels show only spans with this tag, as the others' are.
OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=production
volumes: volumes:
- /mnt/fast/appdata/ipx-play:/config - /mnt/fast/appdata/ipx-play:/config
ports: ports:
- "8102:8102" - "8102:8102"
- "9189:9189" # /metrics, for Prometheus on the LAN; not in the tunnel
``` ```
Its config, `/mnt/fast/appdata/ipx-play/play.toml`: Its config, `/mnt/fast/appdata/ipx-play/play.toml`:
@@ -43,6 +46,9 @@ Its config, `/mnt/fast/appdata/ipx-play/play.toml`:
```toml ```toml
bind = "0.0.0.0:8102" bind = "0.0.0.0:8102"
public_url = "https://play.ipxreader.app" public_url = "https://play.ipxreader.app"
# Prometheus metrics, on a port of their own and not the one the tunnel publishes. Empty, or
# left out, serves none.
metrics_bind = "0.0.0.0:9189"
# One for each ipx server allowed to make links; `ipx-play token <name>` prints the lines. # One for each ipx server allowed to make links; `ipx-play token <name>` prints the lines.
[[tokens]] [[tokens]]
@@ -60,6 +66,27 @@ The tunnel publishes `play.ipxreader.app` to `http://192.168.1.130:8102` with no
the pages are anyone's. cloudflared is not on the project's network, so not `ipx-play:8102`, as the pages are anyone's. cloudflared is not on the project's network, so not `ipx-play:8102`, as
the directory's route is the host's port too. `/v1` takes a token for everything. the directory's route is the host's port too. `/v1` takes a token for everything.
## Watching it
Its traces go to Tempo as service `ipx-play`: a span a request, a child of ipx's when ipx shared
an episode, so one trace runs from the page's Share through ipx to here. Its log is a JSON line a
request (`target` `ipx_play::http`: `method`, `path`, `status`, `duration_ms`, `token`, the
server's name and never its token) and one a share (`an episode was shared`: `server`, `id`).
Its metrics (#458), on `metrics_bind`, scraped as `job="ipx-play"`, beside the directory's job in
Prometheus's config:
```yaml
- job_name: ipx-play
static_configs:
- targets: ["192.168.1.130:9189"]
```
`play_http_request_duration_seconds{method,route,status}` (a page opened is `route="/e/{id}"`, an
address no route has `route=""`), `play_shares_total`, `play_process_resident_bytes`,
`play_process_open_fds`, `play_uptime_seconds` and `play_build_info`. The iPX dashboard's
`ipx-play` row draws them, and the alert `ipx-play is down` fires when Prometheus cannot reach it.
## Setting up ipx ## Setting up ipx
In Admin, Directory, Sharing episodes: the address ipx reaches it at, `http://ipx-play:8102` In Admin, Directory, Sharing episodes: the address ipx reaches it at, `http://ipx-play:8102`

View File

@@ -41,6 +41,8 @@ rules = [
"Prometheus cannot reach ipx's /metrics: the iPX container is down or restarting.", no_data="Alerting"), "Prometheus cannot reach ipx's /metrics: the iPX container is down or restarting.", no_data="Alerting"),
rule("directory-down", "ipx-directory is down", 'up{job="ipx-directory"}', "lt", 1, "5m", rule("directory-down", "ipx-directory is down", 'up{job="ipx-directory"}', "lt", 1, "5m",
"Prometheus cannot reach the directory's /metrics.", no_data="Alerting"), "Prometheus cannot reach the directory's /metrics.", no_data="Alerting"),
rule("play-down", "ipx-play is down", 'up{job="ipx-play"}', "lt", 1, "5m",
"Prometheus cannot reach ipx-play's /metrics: shared links are likely not opening.", no_data="Alerting"),
rule("web-down", "ipx-web is down", 'up{job="ipx-web"}', "lt", 1, "5m", rule("web-down", "ipx-web is down", 'up{job="ipx-web"}', "lt", 1, "5m",
"Prometheus cannot reach Caddy's /metrics: the page is likely down too.", no_data="Alerting"), "Prometheus cannot reach Caddy's /metrics: the page is likely down too.", no_data="Alerting"),
# The worker: the daemon answers `ipxd status` while it is stuck, so only these say so. # The worker: the daemon answers `ipxd status` while it is stuck, so only these say so.

View File

@@ -1,11 +1,11 @@
"""The iPX dashboard in Grafana: the three containers of the Arcane project ipx (ipx, ipx-web, """The iPX dashboard in Grafana: the four containers of the Arcane project ipx (ipx, ipx-web,
ipx-directory) and Tower under them, from Prometheus, Loki (their logs, shipped by Alloy) and Tempo. ipx-directory, ipx-play) and Tower under them, from Prometheus, Loki (their logs, shipped by Alloy) and Tempo.
python3 grafana/dashboard.py > /mnt/fast/arcane/projects/monitoring/grafana-provisioning/dashboards/ipx.json python3 grafana/dashboard.py > /mnt/fast/arcane/projects/monitoring/grafana-provisioning/dashboards/ipx.json
Grafana reads that file on its own within a minute; edits made in Grafana are refused. Rows go Grafana reads that file on its own within a minute; edits made in Grafana are refused. Rows go
top-down as a request does: an overview, Tower, then each container (ipx-web, ipx's API, its top-down as a request does: an overview, Tower, then each container (ipx-web, ipx's API, its
worker and its process, the directory), then traces and logs. Prometheus scrapes ipx's /metrics worker and its process, the directory, ipx-play), then traces and logs. Prometheus scrapes ipx's /metrics
(job ipx, #194) and Tower's node exporter (job node); nothing scrapes the containers themselves, (job ipx, #194) and Tower's node exporter (job node); nothing scrapes the containers themselves,
so ipx-web and the directory are read from their logs. The log panels read fields: ipx's so ipx-web and the directory are read from their logs. The log panels read fields: ipx's
(ipc.rs log_event), ipx's metrics (metrics.rs), the directory's and Caddy's; renaming one has to be (ipc.rs log_event), ipx's metrics (metrics.rs), the directory's and Caddy's; renaming one has to be
@@ -41,7 +41,9 @@ WEB = ('{container="ipx-web"} |= "http.log.access" | json | __error__=""'
' | request_uri !~ "/(api|media)/.*" | request_remote_ip != "127.0.0.1"') # not its healthcheck ' | request_uri !~ "/(api|media)/.*" | request_remote_ip != "127.0.0.1"') # not its healthcheck
# Less what Caddy says at every start, by design: no admin endpoint, and no HTTPS or HTTP/3 # Less what Caddy says at every start, by design: no admin endpoint, and no HTTPS or HTTP/3
# behind the tunnel; and its goodbye on a restart. # behind the tunnel; and its goodbye on a restart.
OTHERS_BAD = ('{container=~"ipx-web|ipx-directory"} | json | __error__="" | level =~ "(?i)warn|error"' # ipx-play's (#447): a line a request (method, path, status, duration_ms, token), and one a share.
PLAY_REQ = "play_http_request_duration_seconds"
OTHERS_BAD = ('{container=~"ipx-web|ipx-directory|ipx-play"} | json | __error__="" | level =~ "(?i)warn|error"'
' | msg !~ "admin endpoint disabled|automatic HTTPS is completely disabled|HTTP/[23] skipped.*|exiting; byeee.*"') ' | msg !~ "admin endpoint disabled|automatic HTTPS is completely disabled|HTTP/[23] skipped.*|exiting; byeee.*"')
@@ -149,7 +151,7 @@ stat("Page and API errors", loki(f'sum(count_over_time({WEB_ALL} | status >= 500
thresholds=GOOD_BAD, desc="Answers of 500 and up from ipx-web, for the page or passed on from ipx: 502 is ipx not answering.") thresholds=GOOD_BAD, desc="Answers of 500 and up from ipx-web, for the page or passed on from ipx: 502 is ipx not answering.")
stat("Warnings and errors", loki(f'sum(count_over_time({BAD} [$__range])) + sum(count_over_time({OTHERS_BAD} [$__range])) or vector(0)', instant=True), 20, stat("Warnings and errors", loki(f'sum(count_over_time({BAD} [$__range])) + sum(count_over_time({OTHERS_BAD} [$__range])) or vector(0)', instant=True), 20,
thresholds=[{"color": "green", "value": None}, {"color": "orange", "value": 1}, {"color": "red", "value": 50}], thresholds=[{"color": "green", "value": None}, {"color": "orange", "value": 1}, {"color": "red", "value": 50}],
desc="All three containers, in the time range.") desc="All four containers, in the time range.")
y += 4 y += 4
stat("Feed failures", loki(f'sum(count_over_time({EV} | ev="feed_error" [$__range])) or vector(0)', instant=True), 0, stat("Feed failures", loki(f'sum(count_over_time({EV} | ev="feed_error" [$__range])) or vector(0)', instant=True), 0,
thresholds=GOOD_BAD, desc="Failed feed checks in the time range.") thresholds=GOOD_BAD, desc="Failed feed checks in the time range.")
@@ -394,7 +396,33 @@ ts("Feed bytes fetched", [prom('rate(ipx_feed_bytes_total{job="ipx"}[$__rate_int
12, unit="Bps", desc="Feeds are read whole, so this is what scans and checks allocate; set it beside memory.") 12, unit="Bps", desc="Feeds are read whole, so this is what scans and checks allocate; set it beside memory.")
y += 8 y += 8
# ---- Traces: all three send them to Tempo (#213) # ---- ipx-play: the page a shared episode opens, public at play.ipxreader.app (#447), from its
# /metrics on a port apart from the tunnel's (#458)
row("ipx-play")
stat("ipx-play", prom('max(up{job="ipx-play"})'), 0, thresholds=[{"color": "red", "value": None}, {"color": "green", "value": 1}],
desc="Prometheus reaching ipx-play's /metrics: 1 is up.")["fieldConfig"]["defaults"]["mappings"] = [
{"type": "value", "options": {"0": {"text": "down"}, "1": {"text": "up"}}}]
stat("Episodes shared", prom('sum(increase(play_shares_total{job="ipx-play"}[$__range])) or vector(0)', instant=True), 4,
desc="Links ipx servers asked for, the same episode again included.")
stat("Pages opened", prom(f'sum(increase({PLAY_REQ}_count{{job="ipx-play",route="/e/{{id}}",status="200"}}[$__range])) or vector(0)', instant=True), 8,
desc="Shared episodes someone opened, a link preview's fetch included.")
stat("Links not found", prom(f'sum(increase({PLAY_REQ}_count{{job="ipx-play",status="404"}}[$__range])) or vector(0)', instant=True), 12,
desc="A link mistyped, or someone trying addresses.")
stat("Server errors", prom(f'sum(increase({PLAY_REQ}_count{{job="ipx-play",status=~"5.."}}[$__range])) or vector(0)', instant=True), 16,
thresholds=GOOD_BAD, desc="Mostly its database not answering; its log has an error for each.")
stat("Requests refused", prom(f'sum(increase({PLAY_REQ}_count{{job="ipx-play",status=~"401|422"}}[$__range])) or vector(0)', instant=True), 20,
thresholds=GOOD_BAD, desc="A share without a valid token (401), or of an episode that looked private (422): ipx's own check should have kept it.")
y += 4
ts("Requests by route", [prom(f'sum by (route, status) (increase({PLAY_REQ}_count{{job="ipx-play"}}[$__interval]))', legend="{{status}} {{route}}")],
0, w=8, bars=True, stack=True, desc="An empty route is an address no route has.")
ts("Response time", [prom(f'histogram_quantile(0.5, sum by (le) (rate({PLAY_REQ}_bucket{{job="ipx-play"}}[$__rate_interval])))', legend="median"),
prom(f'histogram_quantile(0.95, sum by (le) (rate({PLAY_REQ}_bucket{{job="ipx-play"}}[$__rate_interval])))', ref="B", legend="95th percentile")],
8, w=8, unit="s")
ts("ipx-play memory", [prom('play_process_resident_bytes{job="ipx-play"}', legend="resident")], 16, w=8, unit="bytes",
desc="mimalloc, as the directory's: a rise that never comes down is a leak.")
y += 8
# ---- Traces: all four send them to Tempo (#213)
row("Traces") row("Traces")
for x, title, q in [(0, "Recent", TRACES + "}"), for x, title, q in [(0, "Recent", TRACES + "}"),
(8, "Slow (over 2s)", TRACES + " && duration > 2s}"), (8, "Slow (over 2s)", TRACES + " && duration > 2s}"),
@@ -408,7 +436,7 @@ y += 10
# ---- Logs # ---- Logs
row("Logs") row("Logs")
panel("logs", "ipx: warnings and errors", 12, 10, 0, [loki(BAD + TEXT)], options=LOG_OPTS) panel("logs", "ipx: warnings and errors", 12, 10, 0, [loki(BAD + TEXT)], options=LOG_OPTS)
panel("logs", "ipx-web and ipx-directory: warnings and errors", 12, 10, 12, panel("logs", "ipx-web, ipx-directory and ipx-play: warnings and errors", 12, 10, 12,
[loki(OTHERS_BAD + ' | line_format "{{.container}} {{.level}}: {{.message}}{{.msg}}"')], options=LOG_OPTS) [loki(OTHERS_BAD + ' | line_format "{{.container}} {{.level}}: {{.message}}{{.msg}}"')], options=LOG_OPTS)
y += 10 y += 10
panel("logs", "ipx log", 24, 12, 0, panel("logs", "ipx log", 24, 12, 0,