diff --git a/.claude/skills/ipx-prod-check/SKILL.md b/.claude/skills/ipx-prod-check/SKILL.md index 06b6ea6..a7361e5 100644 --- a/.claude/skills/ipx-prod-check/SKILL.md +++ b/.claude/skills/ipx-prod-check/SKILL.md @@ -1,11 +1,11 @@ --- name: ipx-prod-check -description: Look for problems in production ipx (the ipx-web, iPX and ipx-directory containers on Tower, Tower itself and the monitoring that watches them) from their logs in Loki, their traces in Tempo and the metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site. +description: Look for problems in production ipx (the ipx-web, iPX, ipx-directory and ipx-play containers on Tower, Tower itself and the monitoring that watches them) from their logs in Loki, their traces in Tempo and the metrics in Prometheus, and file what is found as Gitea issues. Use when asked to check on production, look for issues or errors in ipx, see why something is slow or failing in production, review the logs, or investigate a report about the live site. --- # Checking production ipx -Production is three containers in the Arcane project `ipx` on Tower +Production is four containers in the Arcane project `ipx` on Tower (`/mnt/fast/arcane/projects/ipx/compose.yaml`), one request passing through them in this order: | container | what | logs, in Loki | traces, in Tempo | metrics, in Prometheus | @@ -13,11 +13,12 @@ Production is three containers in the Arcane project `ipx` on Tower | `ipx-web` | Caddy: serves the page, passes `/api` and `/media` to ipx (#203). Port 8099, the LAN and the tunnel | `{container="ipx-web"}`, JSON: a line a request (`http.log.access`), its own errors | service `ipx-web`: a span a request, passing `traceparent` on to ipx (#213) | `job="ipx-web"`: Caddy's own `caddy_http_*`, from `METRICS_BIND`, published as `192.168.1.130:9190` (#371) | | `iPX` | the daemon: API, scans, downloads, DeadAir | `{container="iPX"}`, JSON (`IPX_LOG_FORMAT=json`) | service `ipx`; passes `traceparent` to the directory | `job="ipx"`, its `/metrics` | | `ipx-directory` | the directory of public podcasts, `directory.ipxreader.app` (#204, #214) | `{container="ipx-directory"}`, JSON (`DIRECTORY_LOG_FORMAT=json`) | service `ipx-directory`: a span a request, joining ipx's trace when ipx called; a trace a refresher round (#213) | `job="ipx-directory"`, its `/metrics` | +| `ipx-play` | the page a shared episode opens, `play.ipxreader.app` (#447); ipx sends it the episodes shared | `{container="ipx-play"}`, JSON (`PLAY_LOG_FORMAT=json`): a line a request (`ipx_play::http`), one a share | service `ipx-play`: a span a request, joining ipx's trace on a share | `job="ipx-play"`, its `/metrics` on `metrics_bind` (production `192.168.1.130:9189`, LAN only, #458) | Grafana's alert rules (`grafana/alerts.py`, #365) watch all of it between checks: start by asking which are firing (the monitoring section). Under them, Tower itself: `job="node"` (the node exporter: load, CPU, memory, disks). The Grafana dashboard `iPX` draws all of it in that order (`grafana/dashboard.py`). A trace without the -production tag (`resource.deployment.environment.name="production"`, on all three) is a daemon run +production tag (`resource.deployment.environment.name="production"`, on all four) is a daemon run by hand, not production. Since #213 a request is one trace across the services it touched: a Directory page is ipx-web, then ipx, then the directory, so start from the trace, not the logs. Alloy ships the logs; Loki, Tempo and Prometheus publish no query port to this machine, so use the script beside this file: @@ -38,7 +39,7 @@ $Q flame '' [since] [out.svg] # where the matching spans' time went, "Since the last deploy" is since that container started (each deploys on its own): ```sh -for c in iPX ipx-web ipx-directory; do S=$(docker inspect $c -f '{{.State.StartedAt}}'); echo "$c $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m"; done +for c in iPX ipx-web ipx-directory ipx-play; do S=$(docker inspect $c -f '{{.State.StartedAt}}'); echo "$c $(( ($(date +%s) - $(date -d "$S" +%s)) / 60 + 1 ))m"; done ``` Start every check by seeing that all three run, since Loki cannot see a container that is not: @@ -422,12 +423,12 @@ container's lines are reaching Loki: ```sh curl -s http://192.168.1.130:9090/api/v1/targets | python3 -c 'import json,sys for t in json.load(sys.stdin)["data"]["activeTargets"]: print(t["health"], t["labels"]["job"], t["scrapeUrl"], t.get("lastError","")[:80])' -for c in iPX ipx-web ipx-directory; do echo "$c $($Q metric "sum(count_over_time({container=\"$c\"} [24h]))" 24h | tail -1)"; done +for c in iPX ipx-web ipx-directory ipx-play; do echo "$c $($Q metric "sum(count_over_time({container=\"$c\"} [24h]))" 24h | tail -1)"; done docker ps -a --filter label=com.docker.compose.project=monitoring --format '{{.Names}} {{.Status}}' docker logs --since 1h tempo 2>&1 | grep -E 'failed to poll|blocklist poll complete' | tail -2 ``` -`ipx`, `ipx-directory`, `ipx-web`, `node` and the monitoring's own, `prometheus`, `tempo`, `loki` +`ipx`, `ipx-directory`, `ipx-play`, `ipx-web`, `node` and the monitoring's own, `prometheus`, `tempo`, `loki` and `alloy` (#366), should be `up`. Alloy is scraped at `alloy:12345` because its command sets `--server.http.listen-addr`; it listens on 127.0.0.1 without it. diff --git a/CLAUDE.md b/CLAUDE.md index 4a86185..772307a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,7 +20,7 @@ service of the `content` project, beside audiobookshelf, immich and Plex (#208). | Old database | `/mnt/user/ipx/state.db`, SQLite, used until the move to Postgres on 2026-09-18 and kept for rollback | `/data/state.db` | | Downloads | `/mnt/user/ipx/downloads` | `/downloads` | | Directory | `ipx-directory` in the same project, `192.168.1.130:8100`, public at `directory.ipxreader.app` through the tunnel with no Access in front: its pages are anyone's to browse, read only (#215), and its API (`/v1`) takes a token for everything. Until 2026-10-12 it was `ipx-dir.sdf1.net` (#214). Config `/mnt/fast/appdata/ipx-directory/directory.toml` (token hashes; Tower's own token in `tower.token` beside it, and entered in ipx's Admin, Directory, which keeps it in the database's `settings`, with the address ipx reaches it at: `http://ipx-directory:8100`, inside Docker, not the public address, which went out through Cloudflare and the tunnel, 170-340 ms a call where the directory answers in 16 ms), database `ipx_directory` on the same Postgres, URL in `directory.env` beside the compose file. See [docs/directory.md](docs/directory.md) | `/config/directory.toml` | -| Shared episodes | `ipx-play` in the same project, `192.168.1.130:8102`, public at `play.ipxreader.app` through the tunnel with no Access in front: the page a shared episode opens, anyone's to see (#447). `/v1` takes a token. Config `/mnt/fast/appdata/ipx-play/play.toml` (token hashes; Tower's token entered in ipx's Admin, Directory, Sharing episodes, with the address `http://ipx-play:8102`), database `ipx_play` on the same Postgres, URL in `play.env` beside the compose file. See [docs/play.md](docs/play.md) | `/config/play.toml` | +| Shared episodes | `ipx-play` in the same project, `192.168.1.130:8102`, public at `play.ipxreader.app` through the tunnel with no Access in front: the page a shared episode opens, anyone's to see (#447). `/v1` takes a token. Config `/mnt/fast/appdata/ipx-play/play.toml` (token hashes; Tower's token entered in ipx's Admin, Directory, Sharing episodes, with the address `http://ipx-play:8102`), database `ipx_play` on the same Postgres, URL in `play.env` beside the compose file. Metrics for Prometheus on `192.168.1.130:9189`, LAN only, as `job="ipx-play"` (#458). See [docs/play.md](docs/play.md) | `/config/play.toml` | | Home page | `ipxreader.app` (and `www`): a static page from `site/`, on the Cloudflare Pages project `ipxreader`, not on Tower. See below | | | Web UI | `ipx-web` (Caddy) in the same project, at `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel. It serves the page and passes `/api` and `/media` to ipx, whose own port is published nowhere; its address on `ipx_default` is pinned to `192.168.64.10` (#203) | ipx `0.0.0.0:8099`, Caddy `:8080` | | Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | Caddy passes `Cf-Access-*` only from `192.168.64.1`, the `ipx_default` gateway cloudflared arrives from (`TUNNEL_PEER`; the subnet is pinned in the compose file); ipx trusts them only from Caddy, `192.168.64.10`, and only with a valid `Cf-Access-Jwt-Assertion` (`access_team`, `access_aud`) | diff --git a/crates/play/CHANGELOG.md b/crates/play/CHANGELOG.md index 5f34df2..b8764b9 100644 --- a/crates/play/CHANGELOG.md +++ b/crates/play/CHANGELOG.md @@ -14,5 +14,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - A page for each shared episode, at `/e/`: its show, title, date, length and notes, a player that plays the publisher's own file from the time it was shared at (`?t=754` or `?t=12:34`), buttons to subscribe in Apple Podcasts, Overcast, Pocket Casts or Castro, the feed's address, and a preview where the link is pasted (#447). - `POST /v1/shares`, for an ipx server with a token from `ipx-play token `: an episode in, its short link out, the same link however often and from whichever server it is shared. An address that looks like a private or paid feed's is refused (#447). +- Metrics for Prometheus on a port of its own, `metrics_bind`: requests by route and status and their time, episodes shared, memory (#458). [unreleased]: https://git.sdf1.net/rays/ipx/commits/main/crates/play diff --git a/crates/play/src/config.rs b/crates/play/src/config.rs index 989df7b..63a583f 100644 --- a/crates/play/src/config.rs +++ b/crates/play/src/config.rs @@ -15,6 +15,10 @@ pub struct Config { /// The environment variable holding the database URL, so the file holds no secret. #[serde(default = "default_database_url_env")] pub database_url_env: String, + /// Where to serve /metrics for Prometheus (#458), apart from `bind`, which the tunnel + /// publishes: `0.0.0.0:9189`. Empty serves none. + #[serde(default)] + pub metrics_bind: String, /// The address people open, which the links ipx is handed begin with. #[serde(default = "default_public_url")] pub public_url: String, diff --git a/crates/play/src/lib.rs b/crates/play/src/lib.rs index 5735369..4b32dce 100644 --- a/crates/play/src/lib.rs +++ b/crates/play/src/lib.rs @@ -9,10 +9,11 @@ //! its file is, nothing more. pub mod config; +pub mod metrics; mod page; pub mod store; -use axum::extract::{Path, Query, Request, State}; +use axum::extract::{MatchedPath, Path, Query, Request, State}; use axum::http::{StatusCode, header}; use axum::middleware::Next; use axum::response::{IntoResponse, Response}; @@ -160,6 +161,7 @@ async fn share(State(st): State, Extension(who): Extension, Json(mut e) } match st.store.put(&who.0, &e).await { Ok(id) => { + metrics::SHARES.fetch_add(1, std::sync::atomic::Ordering::Relaxed); tracing::info!(server = %who.0, id = %id, "an episode was shared"); Json(Shared { url: format!("{}/e/{id}", st.base), id }).into_response() } @@ -198,6 +200,9 @@ async fn healthz(State(st): State) -> Response { /// calls here; and a line, with the server's name and never its token. async fn access_log(req: Request, next: Next) -> Response { let (method, path) = (req.method().clone(), req.uri().path().to_owned()); + // By route, not path: by path each shared episode would be a row of its own. A path no route + // has is "", which someone trying addresses cannot multiply. + let route = req.extensions().get::().map_or("", |r| r.as_str()).to_owned(); let start = Instant::now(); let span = if path == "/healthz" { tracing::Span::none() @@ -218,6 +223,14 @@ async fn access_log(req: Request, next: Next) -> Response { let status = resp.status().as_u16(); span.record("http.response.status_code", status); if path != "/healthz" { + metrics::REQUESTS.observe( + &format!( + "method=\"{}\",route=\"{}\",status=\"{status}\"", + ipx_directory::metrics::method(method.as_str()), + ipx_directory::metrics::label(&route) + ), + start.elapsed().as_secs_f64(), + ); let _in = span.enter(); let who = resp.extensions().get::().map_or("", |w| w.0.as_str()).to_owned(); tracing::info!( @@ -301,6 +314,18 @@ mod tests { assert!(!html.contains("javascript:"), "a link that is not a web address is dropped"); assert_eq!(get(&app, "/e/nothere").await.0, StatusCode::NOT_FOUND); + get(&app, "/no/such/page").await; + + let mut out = String::new(); + metrics::REQUESTS.render(&mut out, "r", ""); + for row in [ + r#"r_count{method="GET",route="/e/{id}",status="200"}"#, + r#"r_count{method="POST",route="/v1/shares",status="200"}"#, + r#"r_count{method="GET",route="",status="404"}"#, + ] { + assert!(out.contains(row), "requests are counted by route, not path: {row} in {out}"); + } + assert!(!out.contains("nothere") && !out.contains("/no/such"), "{out}"); } #[tokio::test] diff --git a/crates/play/src/main.rs b/crates/play/src/main.rs index 48a7fa9..1b7e358 100644 --- a/crates/play/src/main.rs +++ b/crates/play/src/main.rs @@ -98,6 +98,13 @@ async fn serve(path: &Path) -> Result<()> { .with_context(|| format!("{} is not set: ipx-play needs a database URL", cfg.database_url_env))?; let store = Store::open(&url, true).await.context("opening the database")?; let listener = TcpListener::bind(&cfg.bind).await.with_context(|| format!("listening on {}", cfg.bind))?; + if !cfg.metrics_bind.is_empty() { + let metrics = TcpListener::bind(&cfg.metrics_bind) + .await + .with_context(|| format!("listening for metrics on {}", cfg.metrics_bind))?; + tracing::info!(bind = %cfg.metrics_bind, "serving metrics for Prometheus at /metrics"); + tokio::spawn(ipx_play::metrics::serve(metrics)); + } tracing::info!(version = env!("CARGO_PKG_VERSION"), bind = %cfg.bind, tokens = tokens.len(), "ipx-play started"); let app = ipx_play::router(Play { store, tokens, base: cfg.base().to_owned() }); axum::serve(listener, app) diff --git a/crates/play/src/metrics.rs b/crates/play/src/metrics.rs new file mode 100644 index 0000000..2e014e8 --- /dev/null +++ b/crates/play/src/metrics.rs @@ -0,0 +1,99 @@ +//! ipx-play's /metrics (#458), in Prometheus's text format: memory, open files, the requests it +//! answered and the episodes it was given to share. The histogram and /proc reading are the +//! directory's, so the two read alike on the dashboard. + +use axum::Router; +use ipx_directory::metrics::{Histograms, SECONDS, process}; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering::Relaxed; +use std::time::Instant; +use tokio::net::TcpListener; + +/// Requests answered, by method, route and status: a page opened is `route="/e/{id}"`. +pub static REQUESTS: Histograms = Histograms::new(SECONDS); +/// Episodes an ipx server sent to be shared, the same one again included. +pub static SHARES: AtomicU64 = AtomicU64::new(0); + +/// Nothing from the database, so a scrape costs nothing and answers while it is down. +fn render(started: Instant) -> String { + let mut out = String::new(); + let mut gauge = |name: &str, kind: &str, help: &str, labels: &str, v: f64| { + out.push_str(&format!("# HELP {name} {help}\n# TYPE {name} {kind}\n{name}{labels} {v}\n")); + }; + let (rss, fds) = process(); + if let Some(v) = rss { + gauge("play_process_resident_bytes", "gauge", "Memory the process holds.", "", v); + } + if let Some(v) = fds { + gauge("play_process_open_fds", "gauge", "Files and sockets the process has open.", "", v); + } + gauge("play_uptime_seconds", "gauge", "Seconds since ipx-play started.", "", started.elapsed().as_secs() as f64); + gauge( + "play_build_info", + "gauge", + "The version running, as a label.", + &format!("{{version=\"{}\"}}", env!("CARGO_PKG_VERSION")), + 1.0, + ); + gauge("play_shares_total", "counter", "Episodes ipx servers sent to be shared.", "", SHARES.load(Relaxed) as f64); + REQUESTS.render( + &mut out, + "play_http_request_duration_seconds", + "Requests answered and the seconds each took, by method, route and status.", + ); + out +} + +/// Serves /metrics on a listener of its own: the public port is behind the tunnel with no Access +/// in front, and these are for the LAN. +// ponytail: no gzip, unlike the directory's (#253): a few dozen lines a scrape. Add tower-http's +// CompressionLayer if the routes grow. +pub async fn serve(listener: TcpListener) { + let started = Instant::now(); + let app = Router::new().route( + "/metrics", + axum::routing::get(move || async move { + ([(axum::http::header::CONTENT_TYPE, "text/plain; version=0.0.4")], render(started)) + }), + ); + if let Err(e) = axum::serve(listener, app).await { + tracing::error!(error = %e, "metrics stopped"); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn every_play_metric_grafana_queries_is_served() { + // A row for each name, so a histogram's lines are there to find. + REQUESTS.observe("method=\"GET\",route=\"/e/{id}\",status=\"200\"", 0.01); + let l = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let at = l.local_addr().unwrap(); + tokio::spawn(serve(l)); + let text = reqwest::get(format!("http://{at}/metrics")).await.unwrap().text().await.unwrap(); + let src = [include_str!("../../../grafana/dashboard.py"), include_str!("../../../grafana/alerts.py")].concat(); + let names: std::collections::BTreeSet<&str> = src + .match_indices("play_") + .filter(|(i, _)| !src[..*i].ends_with(|c: char| c.is_ascii_alphanumeric() || c == '_')) + .filter_map(|(i, _)| { + let rest = &src[i..]; + let end = rest.find(|c: char| !(c.is_ascii_lowercase() || c == '_')).unwrap_or(rest.len()); + // A query names one with its labels, or the dashboard keeps it in a string; ipx's + // events, such as play_start in a panel's description, are neither. + rest[end..].starts_with(['{', '"']).then(|| &rest[..end]) + }) + // A histogram is queried by its series' names and served under its own. + .map(|n| n.trim_end_matches("_bucket").trim_end_matches("_count").trim_end_matches("_sum")) + .collect(); + assert!(names.len() >= 3, "{names:?}"); + for n in names { + // Memory comes from /proc, which a Mac has not. + if n.starts_with("play_process_") && process().0.is_none() { + continue; + } + assert!(text.contains(&format!("# TYPE {n} ")), "{n} is queried by grafana/ but not served: {text}"); + } + } +} diff --git a/docs/play.md b/docs/play.md index 4415474..2490b6d 100644 --- a/docs/play.md +++ b/docs/play.md @@ -32,10 +32,13 @@ The compose service, beside `ipx-directory`: environment: PLAY_LOG_FORMAT: json OTEL_EXPORTER_OTLP_ENDPOINT: http://192.168.1.130:4318 # Tempo is not on this network + # The dashboard's trace panels show only spans with this tag, as the others' are. + OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=production volumes: - /mnt/fast/appdata/ipx-play:/config ports: - "8102:8102" + - "9189:9189" # /metrics, for Prometheus on the LAN; not in the tunnel ``` Its config, `/mnt/fast/appdata/ipx-play/play.toml`: @@ -43,6 +46,9 @@ Its config, `/mnt/fast/appdata/ipx-play/play.toml`: ```toml bind = "0.0.0.0:8102" public_url = "https://play.ipxreader.app" +# Prometheus metrics, on a port of their own and not the one the tunnel publishes. Empty, or +# left out, serves none. +metrics_bind = "0.0.0.0:9189" # One for each ipx server allowed to make links; `ipx-play token ` prints the lines. [[tokens]] @@ -60,6 +66,27 @@ The tunnel publishes `play.ipxreader.app` to `http://192.168.1.130:8102` with no the pages are anyone's. cloudflared is not on the project's network, so not `ipx-play:8102`, as the directory's route is the host's port too. `/v1` takes a token for everything. +## Watching it + +Its traces go to Tempo as service `ipx-play`: a span a request, a child of ipx's when ipx shared +an episode, so one trace runs from the page's Share through ipx to here. Its log is a JSON line a +request (`target` `ipx_play::http`: `method`, `path`, `status`, `duration_ms`, `token`, the +server's name and never its token) and one a share (`an episode was shared`: `server`, `id`). + +Its metrics (#458), on `metrics_bind`, scraped as `job="ipx-play"`, beside the directory's job in +Prometheus's config: + +```yaml + - job_name: ipx-play + static_configs: + - targets: ["192.168.1.130:9189"] +``` + +`play_http_request_duration_seconds{method,route,status}` (a page opened is `route="/e/{id}"`, an +address no route has `route=""`), `play_shares_total`, `play_process_resident_bytes`, +`play_process_open_fds`, `play_uptime_seconds` and `play_build_info`. The iPX dashboard's +`ipx-play` row draws them, and the alert `ipx-play is down` fires when Prometheus cannot reach it. + ## Setting up ipx In Admin, Directory, Sharing episodes: the address ipx reaches it at, `http://ipx-play:8102` diff --git a/grafana/alerts.py b/grafana/alerts.py index ae46c15..3baf5f1 100644 --- a/grafana/alerts.py +++ b/grafana/alerts.py @@ -41,6 +41,8 @@ rules = [ "Prometheus cannot reach ipx's /metrics: the iPX container is down or restarting.", no_data="Alerting"), rule("directory-down", "ipx-directory is down", 'up{job="ipx-directory"}', "lt", 1, "5m", "Prometheus cannot reach the directory's /metrics.", no_data="Alerting"), + rule("play-down", "ipx-play is down", 'up{job="ipx-play"}', "lt", 1, "5m", + "Prometheus cannot reach ipx-play's /metrics: shared links are likely not opening.", no_data="Alerting"), rule("web-down", "ipx-web is down", 'up{job="ipx-web"}', "lt", 1, "5m", "Prometheus cannot reach Caddy's /metrics: the page is likely down too.", no_data="Alerting"), # The worker: the daemon answers `ipxd status` while it is stuck, so only these say so. diff --git a/grafana/dashboard.py b/grafana/dashboard.py index 2a0f487..a9d4b7b 100644 --- a/grafana/dashboard.py +++ b/grafana/dashboard.py @@ -1,11 +1,11 @@ -"""The iPX dashboard in Grafana: the three containers of the Arcane project ipx (ipx, ipx-web, -ipx-directory) and Tower under them, from Prometheus, Loki (their logs, shipped by Alloy) and Tempo. +"""The iPX dashboard in Grafana: the four containers of the Arcane project ipx (ipx, ipx-web, +ipx-directory, ipx-play) and Tower under them, from Prometheus, Loki (their logs, shipped by Alloy) and Tempo. python3 grafana/dashboard.py > /mnt/fast/arcane/projects/monitoring/grafana-provisioning/dashboards/ipx.json Grafana reads that file on its own within a minute; edits made in Grafana are refused. Rows go top-down as a request does: an overview, Tower, then each container (ipx-web, ipx's API, its -worker and its process, the directory), then traces and logs. Prometheus scrapes ipx's /metrics +worker and its process, the directory, ipx-play), then traces and logs. Prometheus scrapes ipx's /metrics (job ipx, #194) and Tower's node exporter (job node); nothing scrapes the containers themselves, so ipx-web and the directory are read from their logs. The log panels read fields: ipx's (ipc.rs log_event), ipx's metrics (metrics.rs), the directory's and Caddy's; renaming one has to be @@ -41,7 +41,9 @@ WEB = ('{container="ipx-web"} |= "http.log.access" | json | __error__=""' ' | request_uri !~ "/(api|media)/.*" | request_remote_ip != "127.0.0.1"') # not its healthcheck # Less what Caddy says at every start, by design: no admin endpoint, and no HTTPS or HTTP/3 # behind the tunnel; and its goodbye on a restart. -OTHERS_BAD = ('{container=~"ipx-web|ipx-directory"} | json | __error__="" | level =~ "(?i)warn|error"' +# ipx-play's (#447): a line a request (method, path, status, duration_ms, token), and one a share. +PLAY_REQ = "play_http_request_duration_seconds" +OTHERS_BAD = ('{container=~"ipx-web|ipx-directory|ipx-play"} | json | __error__="" | level =~ "(?i)warn|error"' ' | msg !~ "admin endpoint disabled|automatic HTTPS is completely disabled|HTTP/[23] skipped.*|exiting; byeee.*"') @@ -149,7 +151,7 @@ stat("Page and API errors", loki(f'sum(count_over_time({WEB_ALL} | status >= 500 thresholds=GOOD_BAD, desc="Answers of 500 and up from ipx-web, for the page or passed on from ipx: 502 is ipx not answering.") stat("Warnings and errors", loki(f'sum(count_over_time({BAD} [$__range])) + sum(count_over_time({OTHERS_BAD} [$__range])) or vector(0)', instant=True), 20, thresholds=[{"color": "green", "value": None}, {"color": "orange", "value": 1}, {"color": "red", "value": 50}], - desc="All three containers, in the time range.") + desc="All four containers, in the time range.") y += 4 stat("Feed failures", loki(f'sum(count_over_time({EV} | ev="feed_error" [$__range])) or vector(0)', instant=True), 0, thresholds=GOOD_BAD, desc="Failed feed checks in the time range.") @@ -394,7 +396,33 @@ ts("Feed bytes fetched", [prom('rate(ipx_feed_bytes_total{job="ipx"}[$__rate_int 12, unit="Bps", desc="Feeds are read whole, so this is what scans and checks allocate; set it beside memory.") y += 8 -# ---- Traces: all three send them to Tempo (#213) +# ---- ipx-play: the page a shared episode opens, public at play.ipxreader.app (#447), from its +# /metrics on a port apart from the tunnel's (#458) +row("ipx-play") +stat("ipx-play", prom('max(up{job="ipx-play"})'), 0, thresholds=[{"color": "red", "value": None}, {"color": "green", "value": 1}], + desc="Prometheus reaching ipx-play's /metrics: 1 is up.")["fieldConfig"]["defaults"]["mappings"] = [ + {"type": "value", "options": {"0": {"text": "down"}, "1": {"text": "up"}}}] +stat("Episodes shared", prom('sum(increase(play_shares_total{job="ipx-play"}[$__range])) or vector(0)', instant=True), 4, + desc="Links ipx servers asked for, the same episode again included.") +stat("Pages opened", prom(f'sum(increase({PLAY_REQ}_count{{job="ipx-play",route="/e/{{id}}",status="200"}}[$__range])) or vector(0)', instant=True), 8, + desc="Shared episodes someone opened, a link preview's fetch included.") +stat("Links not found", prom(f'sum(increase({PLAY_REQ}_count{{job="ipx-play",status="404"}}[$__range])) or vector(0)', instant=True), 12, + desc="A link mistyped, or someone trying addresses.") +stat("Server errors", prom(f'sum(increase({PLAY_REQ}_count{{job="ipx-play",status=~"5.."}}[$__range])) or vector(0)', instant=True), 16, + thresholds=GOOD_BAD, desc="Mostly its database not answering; its log has an error for each.") +stat("Requests refused", prom(f'sum(increase({PLAY_REQ}_count{{job="ipx-play",status=~"401|422"}}[$__range])) or vector(0)', instant=True), 20, + thresholds=GOOD_BAD, desc="A share without a valid token (401), or of an episode that looked private (422): ipx's own check should have kept it.") +y += 4 +ts("Requests by route", [prom(f'sum by (route, status) (increase({PLAY_REQ}_count{{job="ipx-play"}}[$__interval]))', legend="{{status}} {{route}}")], + 0, w=8, bars=True, stack=True, desc="An empty route is an address no route has.") +ts("Response time", [prom(f'histogram_quantile(0.5, sum by (le) (rate({PLAY_REQ}_bucket{{job="ipx-play"}}[$__rate_interval])))', legend="median"), + prom(f'histogram_quantile(0.95, sum by (le) (rate({PLAY_REQ}_bucket{{job="ipx-play"}}[$__rate_interval])))', ref="B", legend="95th percentile")], + 8, w=8, unit="s") +ts("ipx-play memory", [prom('play_process_resident_bytes{job="ipx-play"}', legend="resident")], 16, w=8, unit="bytes", + desc="mimalloc, as the directory's: a rise that never comes down is a leak.") +y += 8 + +# ---- Traces: all four send them to Tempo (#213) row("Traces") for x, title, q in [(0, "Recent", TRACES + "}"), (8, "Slow (over 2s)", TRACES + " && duration > 2s}"), @@ -408,7 +436,7 @@ y += 10 # ---- Logs row("Logs") panel("logs", "ipx: warnings and errors", 12, 10, 0, [loki(BAD + TEXT)], options=LOG_OPTS) -panel("logs", "ipx-web and ipx-directory: warnings and errors", 12, 10, 12, +panel("logs", "ipx-web, ipx-directory and ipx-play: warnings and errors", 12, 10, 12, [loki(OTHERS_BAD + ' | line_format "{{.container}} {{.level}}: {{.message}}{{.msg}}"')], options=LOG_OPTS) y += 10 panel("logs", "ipx log", 24, 12, 0,