rays
dd91602a49
Reviewing what production's traces and metrics leave out turned up twelve gaps; each is an issue. Metrics. A Prometheus histogram for ipx and the directory both (ipx_directory::metrics::Histograms), which ipx's request histogram (#364) now uses too. - ipx: each statement's time once it has a connection, by its first word, from SeaORM's metric callback; and the wait for one, for hand-written SQL in db.rs, measured as the call less the statement (the callback runs in the poll that returns the result, so the time read after the await is that statement's) (#367). SeaORM's own queries are timed but their wait is not seen: a shortcut, noted. - ipx: counters of what the events say, counted in emit, which every event passes once: feed failures by error.type, new items, downloads saved and failed, DeadAir's seconds found (#368). - ipx: the page's list_shown, play_start and stall timings (#369). - directory: requests by method, route (a route layer puts the matched route on the response, or each feed's page was a row) and status; feeds checked and failed; snapshots taken (#370). - ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11 calls deprecated, because the browser tests' Caddy is 2.6 (#371). Traces. - A list's sync is list_sync with steps, an OPML upload opml_import (#372). Doing so showed sync_group reading every account and its subscription to the list once per listed feed; read once now (#377). - Each command the worker runs, from the socket, the CLI or the page, is a command span with its job inside (#373). - The daemon's start is a trace, from opening the database through the catalogue's repairs to "daemon started" (#374). - The directory's lines carry trace_id and span_id: WithTrace moves to ipx_directory::otel and both use it (#375). - A tap or key is its trace's root span (#376). The page names a root span id in each request's traceparent, where the parent was random and pointed at nothing, and reports the tap (ids, start, its last answer) when the next begins or the page is hidden. opentelemetry 0.33 gives a span builder no way to take ids, so the shared tracer provider has an id generator that takes given ids inside with_ids, on one thread. Alerts. grafana/alerts.py, Grafana alerting provisioning beside the dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals stopped, the directory's refresher stopped, Tower's load and memory, the monitoring down, Tempo not reading its blocks, iPX's lines not reaching Loki. No contact point yet: firing shows in Grafana. Prometheus now scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's config, not this repository's. The dashboard has database and page-timing panels; the prod check knows the new metrics, spans and alerts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6.1 KiB
6.1 KiB