rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

Requests are counted in /metrics, and only a failed one is logged (#364)

Browse Source 
ipx logged a line for every request: 2,306 a day, over half its log by
size, repeating Caddy's access line and the request's span. It stayed
because ipx had no request metric and the dashboard's API row was built
on the line. /metrics now has ipx_http_request_duration_seconds, a
histogram by method, route and status (5 ms to 10 s), the event stream
left out as the panels left it out. The route is the pattern, "" for an
unrouted path, and an unusual method is "other", so a scanner cannot make
a row a request. A request that went well is logged at debug, which the
admin page's log view keeps; a failed one keeps its warning in Loki.

The dashboard's API row reads the histogram (the slowest-request series
becomes the 99th percentile, a bucket's bound; the slowest themselves are
in Tempo). prom() takes instant queries and table() a unit for it.

The prod-check skill, reviewed whole: check 3 reads the histogram; the
directory has metrics, which it said it had not; "all gauges" was half
counters; the healthcheck removed in #218 was still named three times;
Caddy's /api/events aborts (600 a week, a page closing its stream) are
filtered as by design; the monitoring section checks Tempo's block poll,
which failed on 2026-10-25 on one empty block and left every search with
only the last 15 minutes; and the lines to expect from each container
are counted over a day, now that iPX can be quiet for an hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-25 15:11:20 +00:00
parent 6352324bcb
commit c788112190
6 changed files with 154 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
CLAUDE.md
View File

@@ -318,10 +318,12 @@ ms in ipx, the difference being new connections, 80-330 ms each to open (#241).
watch the shutdown channel itself; the daemon ignored SIGTERM for exactly this reason.
* Only one daemon per socket. Removing the socket file defeats the guard and you get two daemons
fighting over the database, with the stale one still holding the port.
* **The Grafana dashboard reads the log's fields** (`grafana/dashboard.py`). Production logs JSON
(`IPX_LOG_FORMAT=json`); the access log's `method`, `path`, `route`, `status`, `duration_ms` and
the events' `ev`, `feed`, `new`, `bytes`, `msg` (`log_event` in ipc.rs) are what the panels query.
Rename one and its panels go blank without an error; regenerate the dashboard to match.
* **The Grafana dashboard reads the log's fields and the metrics' names** (`grafana/dashboard.py`).
Production logs JSON (`IPX_LOG_FORMAT=json`); the events' `ev`, `feed`, `new`, `bytes`, `msg`
(`log_event` in ipc.rs) and the metrics in metrics.rs (requests are
`ipx_http_request_duration_seconds`, #364) are what the panels query. Rename one and its panels
go blank without an error; regenerate the dashboard to match. Routine lines (a request that went
well, a scan's end) are debug: the admin page's Log view has them, Loki does not.
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
different task. `ipx status` checks the control socket and the database; to see the worker
getting through its jobs, watch for `scan complete` in the log.