The sweep trims the cache to its limit, so the kept bytes it reports are the
limit every time once the cache has filled: the panel was two lines flat at
500 MiB. A counter of images thrown out says whether the limit is too small.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The healthcheck answers while the worker is stuck on one job, and nothing
showed which job or for how long. STATS, one static of atomics, is kept as
the work happens: the worker's job and since when (a guard, so a failed or
stopped job is counted too), jobs and their seconds by kind, when a scan
last finished, web requests, feed fetches and downloads in progress, bytes
downloaded, live updates dropped for a listener that fell behind, DeadAir's
backlog and how long it has been on its file, and the artwork kept. With
the event queue, the DNS retries and the healthcheck's last counts, /metrics
still reads memory only. The dashboard's Daemon row and the prod-check
skill take them up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Logs and traces show requests, scans and downloads; nothing showed what
the daemon holds meanwhile. /metrics, behind sign-in, gives database
connections in use and idle against the pool's maximum, commands waiting
for the worker, live-update listeners, free torrent slots, episodes
playing through DeadAir, the library's counts, the tokio runtime's
workers, tasks and queue, and the process's memory and open files, in
Prometheus's text format, written by hand rather than with a client
library. Not logged or traced: Prometheus asks every 15 s. The iPX
dashboard gains a Daemon row of them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From Dash0's structured logging guide, what applies here:
- Each JSON line inside a traced span ends with its trace_id and span_id, so a line in Loki leads
to its trace in Tempo; the access log is written inside its request's span so it has one too.
The JSON formatter takes no extra fields, so WithTrace appends them to the object it writes.
- A feed or download failure carries error.type (the HTTP status, or dns, redirect_loop,
timeout, ...) and http.response.status_code, from failure_kind beside explain_failure, so
failures group by kind without a regex over msg.
- Each event was logged twice: words under ipx::scan and fields under ipx::io. It is now one
line under ipx::scan with both; the wire copy is at debug, for the admin page's Daemon I/O tab,
and out of production's log. The healthcheck's status reply stays under ipx::io.
- The access log's ms is duration_ms. The dashboard and the prod-check skill follow.
- error fields are Display with the anyhow chain everywhere, not a mix of Debug and Display.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The log was text, so the Grafana dashboard picked lines apart with
regular expressions, and a change of wording would have blanked its
panels. With IPX_LOG_FORMAT=json each line is one JSON object: the
access log carries method, path, route, status and ms as fields (the
route passed from the routing layer in the response's extensions), and
each wire event its ev, feed, new, downloaded, failed, bytes, msg and
the rest (log_wire), beside the old message. The two startup lines that
were println! are logged, so no line breaks the JSON. Text stays the
default, for a terminal. The dashboard reads the fields with Loki's json
parser, and groups requests by route rather than path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Built on what the monitoring project already collects: the container's
log in Loki (through Alloy) and the traces in Tempo. It parses the
access log and the event log's wire JSON, so there are no metrics to
add to ipx: what is waiting and downloaded (from the healthcheck's
status), new items, downloads and bytes, failing feeds and downloads,
requests by status and response time, the slowest and busiest paths,
recent and slow traces, and the log. Provisioned from a file, so it is
regenerated here, not edited in Grafana.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
