Requests are counted in /metrics, and only a failed one is logged (#364)

ipx logged a line for every request: 2,306 a day, over half its log by
size, repeating Caddy's access line and the request's span. It stayed
because ipx had no request metric and the dashboard's API row was built
on the line. /metrics now has ipx_http_request_duration_seconds, a
histogram by method, route and status (5 ms to 10 s), the event stream
left out as the panels left it out. The route is the pattern, "" for an
unrouted path, and an unusual method is "other", so a scanner cannot make
a row a request. A request that went well is logged at debug, which the
admin page's log view keeps; a failed one keeps its warning in Loki.

The dashboard's API row reads the histogram (the slowest-request series
becomes the 99th percentile, a bucket's bound; the slowest themselves are
in Tempo). prom() takes instant queries and table() a unit for it.

The prod-check skill, reviewed whole: check 3 reads the histogram; the
directory has metrics, which it said it had not; "all gauges" was half
counters; the healthcheck removed in #218 was still named three times;
Caddy's /api/events aborts (600 a week, a page closing its stream) are
filtered as by design; the monitoring section checks Tempo's block poll,
which failed on 2026-10-25 on one empty block and left every search with
only the last 15 minutes; and the lines to expect from each container
are counted over a day, now that iPX can be quiet for an hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/ipx/CHANGELOG.md

@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- The log no longer has a line for every request that went well, over half its size: `/metrics` counts each request and its time by route and status as `ipx_http_request_duration_seconds`, a request that failed keeps its warning, and the admin page's log view still shows them all (#364).
 - The log no longer has a line for every scan that finishes, a fifth of its lines; the metrics count scans, and the page's log view still shows them (#363).
 
 ### Fixed

crates/ipx/src/metrics.rs

@@ -62,8 +62,17 @@ pub struct Stats {
     job: std::sync::Mutex<Option<(&'static str, Instant)>>,
     /// Jobs finished, by kind: how many, and their seconds all told.
     jobs: std::sync::Mutex<BTreeMap<&'static str, (u64, f64)>>,
+    /// Requests answered, by method, route and status: how many within each of `HTTP_BUCKETS`,
+    /// how many all told, and their seconds. What the access log's lines were read for (#364).
+    http: std::sync::Mutex<BTreeMap<(&'static str, String, u16), Requests>>,
 }
 
+/// The upper bounds, in seconds, a request's time is counted under: most of ipx's answer in tens
+/// of milliseconds, and a slow one, a host stalling, in seconds.
+const HTTP_BUCKETS: [f64; 11] = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0];
+
+type Requests = ([u64; HTTP_BUCKETS.len()], u64, f64);
+
 impl Stats {
     pub const fn new() -> Self {
         Self {
@@ -89,9 +98,26 @@ impl Stats {
             people: std::sync::Mutex::new(BTreeMap::new()),
             job: std::sync::Mutex::new(None),
             jobs: std::sync::Mutex::new(BTreeMap::new()),
+            http: std::sync::Mutex::new(BTreeMap::new()),
         }
     }
 
+    /// Counts a request answered. `route` is the pattern matched, "" for none, so a scanner's
+    /// made-up paths are one row; a method outside the usual is "other", for the same reason.
+    pub fn request(&self, method: &str, route: &str, status: u16, secs: f64) {
+        let method = ["GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"]
+            .into_iter()
+            .find(|m| *m == method)
+            .unwrap_or("other");
+        let mut http = self.http.lock().expect("a panic elsewhere poisoned this lock");
+        let (buckets, n, sum) = http.entry((method, route.to_owned(), status)).or_default();
+        for (b, le) in buckets.iter_mut().zip(HTTP_BUCKETS) {
+            *b += u64::from(secs <= le);
+        }
+        *n += 1;
+        *sum += secs;
+    }
+
     /// Adds to what `who` used on `key` ("server" or "own"): items judged, fits, input and output
     /// tokens.
     pub fn bill(&self, who: &str, key: &'static str, used: [f64; 4]) {
@@ -308,6 +334,19 @@ fn work(out: &mut String, st: &Stats, event_queue: usize, retries: usize, art_li
         "Web requests being answered.",
         &[("", st.http_in_flight.load(Relaxed) as f64)],
     );
+    let http = st.http.lock().expect("a panic elsewhere poisoned this lock").clone();
+    let name = "ipx_http_request_duration_seconds";
+    let _ = writeln!(
+        out,
+        "# HELP {name} Web requests answered and the seconds each took, by method, route and status.\n# TYPE {name} histogram"
+    );
+    for ((method, route, status), (buckets, n, sum)) in &http {
+        let l = format!("method=\"{method}\",route=\"{}\",status=\"{status}\"", label(route));
+        for (le, b) in HTTP_BUCKETS.iter().zip(buckets) {
+            let _ = writeln!(out, "{name}_bucket{{{l},le=\"{le}\"}} {b}");
+        }
+        let _ = writeln!(out, "{name}_bucket{{{l},le=\"+Inf\"}} {n}\n{name}_sum{{{l}}} {sum}\n{name}_count{{{l}}} {n}");
+    }
     gauge(
         out,
         "ipx_scan_feeds_fetching",
@@ -512,6 +551,24 @@ mod tests {
         assert!(out.contains("ipx_jev_tokens_total{kind=\"output\",user=\"a \\\"b\\\"\",key=\"own\"} 1\n"), "{out}");
         // The library's counts are left out until the daemon has counted it.
         assert!(!out.contains("ipx_feeds "));
+
+        for secs in [0.03, 0.04, 2.0] {
+            T.request("GET", "/api/feeds", 200, secs);
+        }
+        T.request("PROPFIND", "", 404, 0.001);
+        let mut out = String::new();
+        work(&mut out, &T, 0, 0, 0);
+        let l = "method=\"GET\",route=\"/api/feeds\",status=\"200\"";
+        for line in [
+            format!("ipx_http_request_duration_seconds_bucket{{{l},le=\"0.025\"}} 0\n"),
+            format!("ipx_http_request_duration_seconds_bucket{{{l},le=\"0.05\"}} 2\n"),
+            format!("ipx_http_request_duration_seconds_bucket{{{l},le=\"2.5\"}} 3\n"),
+            format!("ipx_http_request_duration_seconds_bucket{{{l},le=\"+Inf\"}} 3\n"),
+            format!("ipx_http_request_duration_seconds_count{{{l}}} 3\n"),
+            "ipx_http_request_duration_seconds_count{method=\"other\",route=\"\",status=\"404\"} 1\n".into(),
+        ] {
+            assert!(out.contains(&line), "{line} in {out}");
+        }
     }
 
     #[test]

crates/ipx/src/web.rs

@@ -2859,10 +2859,17 @@ async fn access_log(req: Request, next: Next) -> Response {
         // The same as fields, for the JSON log (#91). Unrouted, a request has no route.
         let route = resp.extensions().get::<MatchedPath>().map(|r| r.as_str().to_owned());
         let method = method.as_str();
+        // The event stream's time is its first byte, not the minutes it stays open.
+        if path != "/api/events" {
+            let secs = started.elapsed().as_secs_f64();
+            crate::metrics::STATS.request(method, route.as_deref().unwrap_or(""), status, secs);
+        }
         // Inside the request's span, so the line carries its trace id and leads to its trace.
         let _in = span.enter();
         if resp.status().is_success() || resp.status().is_redirection() {
-            tracing::info!(target: "ipx::http", method, path, route, status, duration_ms, "{method} {path} -> {status} in {duration_ms}ms");
+            // Debug: the metric above counts it, Caddy's access log and the span have it too,
+            // and it was over half the log (#364). The page's log view keeps debug.
+            tracing::debug!(target: "ipx::http", method, path, route, status, duration_ms, "{method} {path} -> {status} in {duration_ms}ms");
         } else {
             tracing::warn!(target: "ipx::http", method, path, route, status, duration_ms, "{method} {path} -> {status} in {duration_ms}ms");
         }