The weekly Apple Podcasts sync runs unattended and ran in no test. Its
Apple address is now an argument (run passes Apple's), and its pause
between calls is none under test, so a test runs it against a stand-in
with one category: the top chart and the category's are taken each
show once in order, looked up, and filed as the apple-podcasts
snapshot with a feed that has a key in it refused as private; a feed an
admin blocked is left out of the next; and a sync whose lookup fails
is an error that leaves the last good snapshot as it was.
/metrics serves every directory_ metric the dashboard and alerts
query, read out of them. Dropping a server's snapshot through the API
is for admins only, and takes the feeds only it listed, keeping one
another server lists too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three tests for what the dashboard and the alert rules read and nothing
checked: every ipx_ metric named in grafana/dashboard.py and alerts.py
(read out of them at compile time) is served by render, with the ones
written only once known (counts, the last scan, a job under way) set
first; ipx's JSON log lines for a scan's events carry ev, feed, new,
bytes, msg and error.type, and inside a span trace_id and span_id; the
directory's snapshot line carries instance, accepted and the refused_
counts.
The JSON layer was built the same way in both services' setup; it is
otel::json_layer now, so the tests run the layer production does.
The log tests set the global subscriber, as production does: under a
scoped one WithTrace finds no current span (#408).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Crawlers asked for both about 70 times a day and got 404. robots.txt
keeps them off /search, which is any query string at all, and the token-only
API, and names the sitemap; the sitemap lists /, /all, each category and each
feed. Both need whole addresses, so they are written for the host asked for:
directory.ipxreader.app through the tunnel, whose cloudflared says https, or
the LAN address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewing what production's traces and metrics leave out turned up
twelve gaps; each is an issue.
Metrics. A Prometheus histogram for ipx and the directory both
(ipx_directory::metrics::Histograms), which ipx's request histogram
(#364) now uses too.
- ipx: each statement's time once it has a connection, by its first
word, from SeaORM's metric callback; and the wait for one, for
hand-written SQL in db.rs, measured as the call less the statement
(the callback runs in the poll that returns the result, so the time
read after the await is that statement's) (#367). SeaORM's own queries
are timed but their wait is not seen: a shortcut, noted.
- ipx: counters of what the events say, counted in emit, which every
event passes once: feed failures by error.type, new items, downloads
saved and failed, DeadAir's seconds found (#368).
- ipx: the page's list_shown, play_start and stall timings (#369).
- directory: requests by method, route (a route layer puts the matched
route on the response, or each feed's page was a row) and status;
feeds checked and failed; snapshots taken (#370).
- ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from
the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11
calls deprecated, because the browser tests' Caddy is 2.6 (#371).
Traces.
- A list's sync is list_sync with steps, an OPML upload opml_import
(#372). Doing so showed sync_group reading every account and its
subscription to the list once per listed feed; read once now (#377).
- Each command the worker runs, from the socket, the CLI or the page,
is a command span with its job inside (#373).
- The daemon's start is a trace, from opening the database through the
catalogue's repairs to "daemon started" (#374).
- The directory's lines carry trace_id and span_id: WithTrace moves to
ipx_directory::otel and both use it (#375).
- A tap or key is its trace's root span (#376). The page names a root
span id in each request's traceparent, where the parent was random and
pointed at nothing, and reports the tap (ids, start, its last answer)
when the next begins or the page is hidden. opentelemetry 0.33 gives a
span builder no way to take ids, so the shared tracer provider has an
id generator that takes given ids inside with_ids, on one thread.
Alerts. grafana/alerts.py, Grafana alerting provisioning beside the
dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a
job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals
stopped, the directory's refresher stopped, Tower's load and memory, the
monitoring down, Tempo not reading its blocks, iPX's lines not reaching
Loki. No contact point yet: firing shows in Grafana. Prometheus now
scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's
config, not this repository's.
The dashboard has database and page-timing panels; the prod check knows
the new metrics, spans and alerts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The crate root held the wire types, Directory with its refresher and
reloader, and every /v1 handler. It is now the module list and the
re-exports ipx and the directory's main use, so their paths are as
before:
wire.rs Listing, FeedPage, SnapshotEntry, ... and MAX_SNAPSHOT
directory.rs Directory, Shared, the refresher and the reloader
api.rs the router, the token check and the /v1 handlers
Directory's fields and methods are pub(crate) for the handlers beside
it. Its two tests that read those fields went with it, on a scratch
Directory of their own; the API tests stay with the API. Every
definition and doc comment line of the old lib.rs is in exactly one of
the new files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
