Metrics and traces for what had none, and alerts on them (#365-#377)

Reviewing what production's traces and metrics leave out turned up
twelve gaps; each is an issue.

Metrics. A Prometheus histogram for ipx and the directory both
(ipx_directory::metrics::Histograms), which ipx's request histogram
(#364) now uses too.
- ipx: each statement's time once it has a connection, by its first
  word, from SeaORM's metric callback; and the wait for one, for
  hand-written SQL in db.rs, measured as the call less the statement
  (the callback runs in the poll that returns the result, so the time
  read after the await is that statement's) (#367). SeaORM's own queries
  are timed but their wait is not seen: a shortcut, noted.
- ipx: counters of what the events say, counted in emit, which every
  event passes once: feed failures by error.type, new items, downloads
  saved and failed, DeadAir's seconds found (#368).
- ipx: the page's list_shown, play_start and stall timings (#369).
- directory: requests by method, route (a route layer puts the matched
  route on the response, or each feed's page was a row) and status;
  feeds checked and failed; snapshots taken (#370).
- ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from
  the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11
  calls deprecated, because the browser tests' Caddy is 2.6 (#371).

Traces.
- A list's sync is list_sync with steps, an OPML upload opml_import
  (#372). Doing so showed sync_group reading every account and its
  subscription to the list once per listed feed; read once now (#377).
- Each command the worker runs, from the socket, the CLI or the page,
  is a command span with its job inside (#373).
- The daemon's start is a trace, from opening the database through the
  catalogue's repairs to "daemon started" (#374).
- The directory's lines carry trace_id and span_id: WithTrace moves to
  ipx_directory::otel and both use it (#375).
- A tap or key is its trace's root span (#376). The page names a root
  span id in each request's traceparent, where the parent was random and
  pointed at nothing, and reports the tap (ids, start, its last answer)
  when the next begins or the page is hidden. opentelemetry 0.33 gives a
  span builder no way to take ids, so the shared tracer provider has an
  id generator that takes given ids inside with_ids, on one thread.

Alerts. grafana/alerts.py, Grafana alerting provisioning beside the
dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a
job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals
stopped, the directory's refresher stopped, Tower's load and memory, the
monitoring down, Tempo not reading its blocks, iPX's lines not reaching
Loki. No contact point yet: firing shows in Grafana. Prometheus now
scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's
config, not this repository's.

The dashboard has database and page-timing panels; the prod check knows
the new metrics, spans and alerts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/directory/CHANGELOG.md

@@ -10,6 +10,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- `/metrics` counts its requests and their times by route and status, the feeds it checks and those that failed, and the snapshots it takes (#370).
+- Its log lines inside a trace carry the trace's and span's ids, as ipx's do, so a failed feed check leads to its trace (#375).
+
 ### Changed
 
 - Its JSON log lines no longer carry their span, once on its own and again in a list, which said nothing the line's own fields did not (#363).

crates/directory/src/api.rs

@@ -36,6 +36,7 @@ pub fn router(dir: Shared) -> Router {
         .merge(v1)
         // Anyone may browse, no token, read only (#215).
         .merge(site::router())
+        .route_layer(axum::middleware::from_fn(remember_route))
         // Brotli, zstd or gzip as the caller takes them: the feed list went to ipx as 603 KB of
         // JSON, most of a Directory page's time (#253). Inside the access log, which records what
         // was sent.
@@ -65,6 +66,14 @@ impl From<anyhow::Error> for ApiError {
 #[derive(Clone)]
 pub(crate) struct Who(String);
 
+/// Puts the route matched on the response for `access_log`, which runs outside routing: by path,
+/// each feed's page was a row of its own in the request metric.
+async fn remember_route(route: axum::extract::MatchedPath, req: Request, next: Next) -> Response {
+    let mut resp = next.run(req).await;
+    resp.extensions_mut().insert(route);
+    resp
+}
+
 /// One line a request, with the token's name and never the token: what the Grafana panels read.
 pub(crate) async fn access_log(req: Request, next: Next) -> Response {
     let (method, path) = (req.method().clone(), req.uri().path().to_owned());
@@ -92,6 +101,15 @@ pub(crate) async fn access_log(req: Request, next: Next) -> Response {
     let who = resp.extensions().get::<Who>().map_or("", |w| w.0.as_str()).to_owned();
     let status = resp.status().as_u16();
     if path != "/healthz" {
+        let route = resp.extensions().get::<axum::extract::MatchedPath>().map_or("", |r| r.as_str());
+        crate::metrics::REQUESTS.observe(
+            &format!(
+                "method=\"{}\",route=\"{}\",status=\"{status}\"",
+                crate::metrics::method(method.as_str()),
+                crate::metrics::label(route)
+            ),
+            start.elapsed().as_secs_f64(),
+        );
         tracing::info!(
             target: "ipx_directory::http",
             method = %method, path = %path, status, duration_ms = start.elapsed().as_millis() as u64, token = %who,
@@ -174,6 +192,7 @@ pub(crate) async fn snapshot(
     }
     let r = dir.store.snapshot(&me.name, &entries).await?;
     dir.wake.notify_one();
+    crate::metrics::SNAPSHOTS.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
     tracing::info!(
         instance = %me.name, accepted = r.accepted, refused_private = r.refused_private,
         refused_blocked = r.refused_blocked, refused_invalid = r.refused_invalid, "snapshot"

crates/directory/src/apple.rs

@@ -96,6 +96,7 @@ async fn sync(dir: &Shared, client: &reqwest::Client, country: &str) -> Result<(
         tokio::time::sleep(PAUSE).await;
     }
     let result = dir.store.snapshot(INSTANCE, &entries).await?;
+    crate::metrics::SNAPSHOTS.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
     tracing::info!(
         shows = ids.len(),
         accepted = result.accepted,

crates/directory/src/directory.rs

@@ -112,6 +112,7 @@ impl Directory {
             Err(e) => Refreshed::Failed(format!("{e:#}")),
         };
         if let Refreshed::Failed(e) = &got {
+            crate::metrics::CHECKS_FAILED.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
             tracing::info!(feed = %due.id, error = %ipx_feed::redact_text(e), "feed check failed");
         }
         // ponytail: a feed that moved for good is still asked for at its old address, which
@@ -143,6 +144,7 @@ pub async fn refresher(dir: Shared) {
                             .await;
                             // One line a round, for the dashboard's refresher panel (#210); failures have their own.
                             tracing::info!(checked = due.len(), "feeds checked");
+                            crate::metrics::CHECKED.fetch_add(due.len() as u64, std::sync::atomic::Ordering::Relaxed);
                         },
                         round,
                     )

crates/directory/src/main.rs

@@ -87,11 +87,16 @@ async fn serve(path: &Path) -> Result<()> {
         .with((!json).then(|| tracing_subscriber::fmt::layer().with_writer(std::io::stderr).with_filter(filter())))
         .with(json.then(|| {
             tracing_subscriber::fmt::layer()
-                .json()
-                .flatten_event(true)
-                // Each line's own fields say all its span did, and the list said it again (#363).
-                .with_current_span(false)
-                .with_span_list(false)
+                .fmt_fields(tracing_subscriber::fmt::format::JsonFields::new())
+                // Each line's own fields say all its span did, and the list said it again (#363);
+                // the trace and span ids lead to the trace instead, as ipx's do (#375).
+                .event_format(ipx_directory::otel::WithTrace(
+                    tracing_subscriber::fmt::format()
+                        .json()
+                        .flatten_event(true)
+                        .with_current_span(false)
+                        .with_span_list(false),
+                ))
                 .with_writer(std::io::stderr)
                 .with_filter(filter())
         }))

crates/directory/src/metrics.rs

@@ -7,6 +7,10 @@
 //! does not come down again is a leak. Not jemalloc, whose upstream was archived in 2025.
 
 use axum::Router;
+use std::collections::BTreeMap;
+use std::fmt::Write as _;
+use std::sync::Mutex;
+use std::sync::atomic::AtomicU64;
 use std::sync::atomic::Ordering::Relaxed;
 use std::time::Instant;
 use tokio::net::TcpListener;
@@ -27,8 +31,75 @@ pub fn process() -> (Option<f64>, Option<f64>) {
     (rss, fds)
 }
 
-/// The directory's /metrics, in Prometheus's text format: memory, open files, and the feeds it
-/// has fetched. Nothing from the database, so a scrape costs nothing and answers while it is down.
+/// Upper bounds for a request's or a query's time, in seconds: most answer in milliseconds, and a
+/// host stalling takes seconds.
+pub const SECONDS: &[f64] = &[0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0];
+
+/// Prometheus's histogram, for ipx and the directory both: for each set of labels, how many
+/// observations fell under each bound, how many all told, and their sum.
+pub struct Histograms {
+    bounds: &'static [f64],
+    rows: Mutex<BTreeMap<String, Row>>,
+}
+
+/// Under each bound, all told, and the sum.
+type Row = (Vec<u64>, u64, f64);
+
+impl Histograms {
+    #[must_use]
+    pub const fn new(bounds: &'static [f64]) -> Self {
+        Self { bounds, rows: Mutex::new(BTreeMap::new()) }
+    }
+
+    /// Counts `v` under `labels`, written as Prometheus reads them (`a="x",b="y"`, or "" for
+    /// none); `label` escapes a value that came from outside.
+    pub fn observe(&self, labels: &str, v: f64) {
+        let mut rows = self.rows.lock().expect("a panic elsewhere poisoned this lock");
+        let (buckets, n, sum) = rows.entry(labels.to_owned()).or_insert_with(|| (vec![0; self.bounds.len()], 0, 0.0));
+        for (b, le) in buckets.iter_mut().zip(self.bounds) {
+            *b += u64::from(v <= *le);
+        }
+        *n += 1;
+        *sum += v;
+    }
+
+    pub fn render(&self, out: &mut String, name: &str, help: &str) {
+        let rows = self.rows.lock().expect("a panic elsewhere poisoned this lock").clone();
+        let _ = writeln!(out, "# HELP {name} {help}\n# TYPE {name} histogram");
+        for (labels, (buckets, n, sum)) in &rows {
+            let sep = if labels.is_empty() { "" } else { "," };
+            for (le, b) in self.bounds.iter().zip(buckets) {
+                let _ = writeln!(out, "{name}_bucket{{{labels}{sep}le=\"{le}\"}} {b}");
+            }
+            let _ = writeln!(out, "{name}_bucket{{{labels}{sep}le=\"+Inf\"}} {n}");
+            let braces = if labels.is_empty() { String::new() } else { format!("{{{labels}}}") };
+            let _ = writeln!(out, "{name}_sum{braces} {sum}\n{name}_count{braces} {n}");
+        }
+    }
+}
+
+/// A request's method as a label: one outside the usual is "other", so a caller cannot make rows.
+#[must_use]
+pub fn method(m: &str) -> &'static str {
+    ["GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"].into_iter().find(|x| *x == m).unwrap_or("other")
+}
+
+/// A label's value as Prometheus's text format reads it.
+#[must_use]
+pub fn label(v: &str) -> String {
+    v.replace('\\', "\\\\").replace('"', "\\\"").replace('\n', "\\n")
+}
+
+/// Requests answered, by method, route and status (#370).
+pub static REQUESTS: Histograms = Histograms::new(SECONDS);
+/// Feeds the refresher has checked, and of them those that failed (#370).
+pub static CHECKED: AtomicU64 = AtomicU64::new(0);
+pub static CHECKS_FAILED: AtomicU64 = AtomicU64::new(0);
+/// Snapshots taken from ipx servers and the Apple sync (#370).
+pub static SNAPSHOTS: AtomicU64 = AtomicU64::new(0);
+
+/// The directory's /metrics, in Prometheus's text format: memory, open files, the feeds it has
+/// fetched and checked, snapshots and requests. Nothing from the database, so a scrape costs nothing and answers while it is down.
 fn render(started: Instant) -> String {
     let mut out = String::new();
     let mut gauge = |name: &str, kind: &str, help: &str, labels: &str, v: f64| {
@@ -69,6 +140,20 @@ fn render(started: Instant) -> String {
         &format!("{{version=\"{}\"}}", env!("CARGO_PKG_VERSION")),
         1.0,
     );
+    gauge("directory_feed_checks_total", "counter", "Feeds the refresher checked.", "", CHECKED.load(Relaxed) as f64);
+    gauge(
+        "directory_feed_checks_failed_total",
+        "counter",
+        "Feed checks that failed: the publisher's error, or a feed that would not parse.",
+        "",
+        CHECKS_FAILED.load(Relaxed) as f64,
+    );
+    gauge("directory_snapshots_total", "counter", "Snapshots taken.", "", SNAPSHOTS.load(Relaxed) as f64);
+    REQUESTS.render(
+        &mut out,
+        "directory_http_request_duration_seconds",
+        "Requests answered and the seconds each took, by method, route and status.",
+    );
     out
 }
 
@@ -92,6 +177,31 @@ pub async fn serve(listener: TcpListener) {
 
 #[cfg(test)]
 mod tests {
+    use super::*;
+
+    #[test]
+    fn a_histogram_is_written_as_prometheus_reads_it() {
+        static H: Histograms = Histograms::new(&[0.1, 1.0]);
+        for v in [0.05, 0.5, 2.0] {
+            H.observe("route=\"/a\"", v);
+        }
+        H.observe("", 0.01);
+        let mut out = String::new();
+        H.render(&mut out, "t_seconds", "Help.");
+        for line in [
+            "# TYPE t_seconds histogram\n",
+            "t_seconds_bucket{route=\"/a\",le=\"0.1\"} 1\n",
+            "t_seconds_bucket{route=\"/a\",le=\"1\"} 2\n",
+            "t_seconds_bucket{route=\"/a\",le=\"+Inf\"} 3\n",
+            "t_seconds_count{route=\"/a\"} 3\n",
+            "t_seconds_bucket{le=\"0.1\"} 1\n",
+            "t_seconds_sum 0.01\n",
+        ] {
+            assert!(out.contains(line), "{line} in {out}");
+        }
+        assert_eq!(label("a \"b\"\\"), "a \\\"b\\\"\\\\");
+    }
+
     #[test]
     fn metrics_say_memory_and_feeds() {
         let text = super::render(std::time::Instant::now());

crates/directory/src/otel.rs

@@ -17,6 +17,7 @@ pub fn provider(service: &'static str) -> anyhow::Result<Option<opentelemetry_sd
         opentelemetry_sdk::trace::SdkTracerProvider::builder()
             .with_batch_exporter(exporter)
             .with_resource(resource)
+            .with_id_generator(Ids::default())
             .build(),
     ))
 }
@@ -58,6 +59,72 @@ pub fn traceparent() -> Option<String> {
     sc.is_valid().then(|| format!("00-{}-{}-{:02x}", sc.trace_id(), sc.span_id(), sc.trace_flags().to_u8()))
 }
 
+/// Random ids, as the SDK's own, but for a span built inside `with_ids`, which takes the ids it is
+/// given: the page names its tap's span before ipx makes it (#376), and the SDK gives a span
+/// builder no other way to be told them.
+#[derive(Debug, Default)]
+struct Ids(opentelemetry_sdk::trace::RandomIdGenerator);
+
+thread_local! {
+    static GIVEN: std::cell::Cell<Option<(opentelemetry::trace::TraceId, opentelemetry::trace::SpanId)>> =
+        const { std::cell::Cell::new(None) };
+}
+
+impl opentelemetry_sdk::trace::IdGenerator for Ids {
+    fn new_trace_id(&self) -> opentelemetry::trace::TraceId {
+        GIVEN.get().map_or_else(|| self.0.new_trace_id(), |(t, _)| t)
+    }
+    fn new_span_id(&self) -> opentelemetry::trace::SpanId {
+        GIVEN.get().map_or_else(|| self.0.new_span_id(), |(_, s)| s)
+    }
+}
+
+/// Runs `f`, which builds one root span, with these ids for it. Synchronous, so nothing else on
+/// this thread makes a span in between.
+pub fn with_ids<R>(
+    trace: opentelemetry::trace::TraceId,
+    span: opentelemetry::trace::SpanId,
+    f: impl FnOnce() -> R,
+) -> R {
+    GIVEN.set(Some((trace, span)));
+    let r = f();
+    GIVEN.set(None);
+    r
+}
+
+/// The JSON log line with the trace and span it belongs to (#91), so a line in Loki leads to its
+/// trace in Tempo. The JSON formatter cannot take a field of its own, so the ids go on the end of
+/// the object it writes; the directory's lines carry them too (#375). A line outside any traced span, or with no trace exporter, is unchanged.
+pub struct WithTrace<F>(pub F);
+
+impl<S, N, F> tracing_subscriber::fmt::FormatEvent<S, N> for WithTrace<F>
+where
+    F: tracing_subscriber::fmt::FormatEvent<S, N>,
+    S: tracing::Subscriber + for<'a> tracing_subscriber::registry::LookupSpan<'a>,
+    N: for<'w> tracing_subscriber::fmt::FormatFields<'w> + 'static,
+{
+    fn format_event(
+        &self,
+        ctx: &tracing_subscriber::fmt::FmtContext<'_, S, N>,
+        mut w: tracing_subscriber::fmt::format::Writer<'_>,
+        ev: &tracing::Event<'_>,
+    ) -> std::fmt::Result {
+        use opentelemetry::trace::TraceContextExt;
+        use tracing_opentelemetry::OpenTelemetrySpanExt;
+        let current = tracing::Span::current();
+        let sc = if current.is_none() { None } else { Some(current.context().span().span_context().clone()) };
+        let Some(sc) = sc.filter(|c| c.is_valid()) else {
+            return self.0.format_event(ctx, w, ev);
+        };
+        let mut line = String::new();
+        self.0.format_event(ctx, tracing_subscriber::fmt::format::Writer::new(&mut line), ev)?;
+        match line.trim_end().strip_suffix('}') {
+            Some(body) => writeln!(w, r#"{body},"trace_id":"{}","span_id":"{}"}}"#, sc.trace_id(), sc.span_id()),
+            None => w.write_str(&line),
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -92,4 +159,21 @@ mod tests {
         assert_eq!(q.unwrap().span().span_context().trace_id().to_string(), "4bf92f3577b34da6a3ce929d0e0e4736");
         assert!(remote_parent(&axum::http::HeaderMap::new(), Some("traceparent=nonsense")).is_none());
     }
+
+    #[test]
+    fn a_span_built_with_given_ids_takes_them_and_the_next_does_not() {
+        use opentelemetry::trace::{Span as _, SpanId, TraceId, Tracer as _, TracerProvider as _};
+        let tracer = opentelemetry_sdk::trace::SdkTracerProvider::builder()
+            .with_id_generator(super::Ids::default())
+            .build()
+            .tracer("t");
+        let (t, s) = (
+            TraceId::from_hex("4bf92f3577b34da6a3ce929d0e0e4736").unwrap(),
+            SpanId::from_hex("00f067aa0ba902b7").unwrap(),
+        );
+        let given = super::with_ids(t, s, || tracer.start("tap"));
+        assert_eq!((given.span_context().trace_id(), given.span_context().span_id()), (t, s));
+        let next = tracer.start("other");
+        assert_ne!(next.span_context().trace_id(), t);
+    }
 }