Every API request is logged three times, and ipx has no request metric to stand in for its line #364

Closed
opened 2026-10-25 08:07:48 -07:00 by rays · 1 comment
Owner

Found in #363's review of production's logs (2026-10-25). ipx logs a line for each request (ipx::http: 2,306 a day, 1.1 MB, 58% of its log's bytes), which repeats Caddy's access line for the same request and the request's span in Tempo. It cannot simply go: ipx has no request-rate or request-time metric, only ipx_http_requests_in_flight, so the dashboard's ipx: API row (requests by status, response time, slowest and busiest routes) and the prod check's checks 3 and 5 are built on that line.

Wanted: a histogram in /metrics of each request's time by method, route and status, the dashboard and prod check reading it, and the line for a request that went well at debug (the page's log view keeps it). A request that failed (4xx, 5xx) keeps its warning in Loki.

Found in #363's review of production's logs (2026-10-25). ipx logs a line for each request (`ipx::http`: 2,306 a day, 1.1 MB, 58% of its log's bytes), which repeats Caddy's access line for the same request and the request's span in Tempo. It cannot simply go: ipx has no request-rate or request-time metric, only `ipx_http_requests_in_flight`, so the dashboard's ipx: API row (requests by status, response time, slowest and busiest routes) and the prod check's checks 3 and 5 are built on that line. Wanted: a histogram in /metrics of each request's time by method, route and status, the dashboard and prod check reading it, and the line for a request that went well at debug (the page's log view keeps it). A request that failed (4xx, 5xx) keeps its warning in Loki.
rays added the enhancement label 2026-10-25 08:07:48 -07:00
Author
Owner

Fixed in c788112: /metrics has ipx_http_request_duration_seconds{method,route,status}, a histogram from 5 ms to 10 s (the event stream left out, unrouted paths as route="", unusual methods as "other"). A request that went well is logged at debug (the admin page's log view keeps it); a failed one keeps its warning. The dashboard's ipx: API row and the prod check's check 3 read the histogram. Deployed 2026-10-25.

Fixed in c788112: /metrics has ipx_http_request_duration_seconds{method,route,status}, a histogram from 5 ms to 10 s (the event stream left out, unrouted paths as route="", unusual methods as "other"). A request that went well is logged at debug (the admin page's log view keeps it); a failed one keeps its warning. The dashboard's ipx: API row and the prod check's check 3 read the histogram. Deployed 2026-10-25.
rays closed this issue 2026-10-25 08:12:55 -07:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rays/ipx#364