rays
dd91602a49
Reviewing what production's traces and metrics leave out turned up twelve gaps; each is an issue. Metrics. A Prometheus histogram for ipx and the directory both (ipx_directory::metrics::Histograms), which ipx's request histogram (#364) now uses too. - ipx: each statement's time once it has a connection, by its first word, from SeaORM's metric callback; and the wait for one, for hand-written SQL in db.rs, measured as the call less the statement (the callback runs in the poll that returns the result, so the time read after the await is that statement's) (#367). SeaORM's own queries are timed but their wait is not seen: a shortcut, noted. - ipx: counters of what the events say, counted in emit, which every event passes once: feed failures by error.type, new items, downloads saved and failed, DeadAir's seconds found (#368). - ipx: the page's list_shown, play_start and stall timings (#369). - directory: requests by method, route (a route layer puts the matched route on the response, or each feed's page was a row) and status; feeds checked and failed; snapshots taken (#370). - ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11 calls deprecated, because the browser tests' Caddy is 2.6 (#371). Traces. - A list's sync is list_sync with steps, an OPML upload opml_import (#372). Doing so showed sync_group reading every account and its subscription to the list once per listed feed; read once now (#377). - Each command the worker runs, from the socket, the CLI or the page, is a command span with its job inside (#373). - The daemon's start is a trace, from opening the database through the catalogue's repairs to "daemon started" (#374). - The directory's lines carry trace_id and span_id: WithTrace moves to ipx_directory::otel and both use it (#375). - A tap or key is its trace's root span (#376). The page names a root span id in each request's traceparent, where the parent was random and pointed at nothing, and reports the tap (ids, start, its last answer) when the next begins or the page is hidden. opentelemetry 0.33 gives a span builder no way to take ids, so the shared tracer provider has an id generator that takes given ids inside with_ids, on one thread. Alerts. grafana/alerts.py, Grafana alerting provisioning beside the dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals stopped, the directory's refresher stopped, Tower's load and memory, the monitoring down, Tempo not reading its blocks, iPX's lines not reaching Loki. No contact point yet: firing shows in Grafana. Prometheus now scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's config, not this repository's. The dashboard has database and page-timing panels; the prod check knows the new metrics, spans and alerts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.9 KiB
5.9 KiB
Changelog: ipx-directory
The directory of public podcasts that ipx servers share (docs/directory.md). ipx's changelog is crates/ipx/CHANGELOG.md, the page's web/CHANGELOG.md.
Its version is in crates/directory/Cargo.toml, and a release is tagged directory-vX.Y.Z.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Unreleased
Added
/metricscounts its requests and their times by route and status, the feeds it checks and those that failed, and the snapshots it takes (#370).- Its log lines inside a trace carry the trace's and span's ids, as ipx's do, so a failed feed check leads to its trace (#375).
Changed
- Its JSON log lines no longer carry their span, once on its own and again in a list, which said nothing the line's own fields did not (#363).
1.4.1 - 2026-10-24
Fixed
- A feed larger than 50 MB is checked again, up to 200 MB, where it was marked as failing (#357).
1.4.0 - 2026-10-23
Fixed
- A WordPress.com site that answers its feed with a browser check (HTTP 403) is read from WordPress.com's API instead of failing (#317).
Security
- HTTPS connections no longer trust certificates from emSign's distrusted intermediates, and certificate checks use constant-time comparisons: rustls-webpki, webpki-roots and ctutils are updated (#351).
- The public pages read the list from the database at most once a minute, however many people ask. Each page asked for the whole catalogue, so a flood of requests with query strings of their own reached the database every time (#341).
- The directory no longer fetches feeds at internal addresses (loopback, the LAN, Docker's networks), named in a snapshot or reached by a redirect, unless directory.toml's
fetch_internallists them. It fetched them, and stored what answered as the feed's title and description (#329). - A server's count of a feed's subscribers is taken up to a million. Two servers sending the largest number there is made the sum overflow, and every listing and page failed until one was dropped (#342).
1.3.0 - 2026-10-19
Added
- A feed's items written in Markdown show formatted on its page, from a Markdown description or RSS's
<source:markdown>(#284).
Fixed
- A listed feed whose address serves something larger than 50 MB is refused as too large, instead of read whole into memory at every check (#295).
Security
- ammonia, which cleans publishers' HTML, updated to 4.2.4 for three advisories, among them XSS through SVG animation tags; rustls to 0.23.46 for five and h2 to 0.4.20 for one (#281).
1.2.0 - 2026-10-16
Added
- Answers and pages go out compressed, brotli, zstd or gzip as the caller takes them: the feed list ipx asks for was 603 KB of JSON. A snapshot may come gzipped,
/metricsanswers gzipped, and withOTEL_EXPORTER_OTLP_COMPRESSION=gziptraces go to the collector compressed (#253). - With
apple_countryset, Apple Podcasts' top charts for that country (the top 100 and the top 50 of each category) come into the directory every week, beside what servers share, without counting as subscribers; and a warning says when Apple's categories no longer match ipx's (#252).
Changed
- Each feed is checked every twelve hours instead of six (#243).
- Feeds are fetched accepting brotli and zstd as well as gzip (#244).
- On Postgres the server keeps all ten of its database connections open, so a request does not wait for a new one (#241).
1.1.1 - 2026-10-15
Changed
- The directory checks five feeds at a time, so a publisher that does not answer no longer holds up the rest of its round (#227).
- A feed's server that takes 10 s to connect, or sends nothing for 15 s, is given up on (#230).
Security
- A failed feed check is logged without its address's query or password (#238).
1.1.0 - 2026-10-13
Added
- With
metrics_bindset, the directory serves Prometheus metrics on a port of its own: memory, open files, and the feeds it has fetched and their bytes. A feed check's trace says how large the feed was (#224).
Changed
- The directory uses mimalloc instead of glibc's allocator, so it gives back memory it has freed instead of holding over a gigabyte while idle (#224).
- The example in docs/directory.md has no healthcheck;
ipx-directory healthis still there to run by hand (#218).
Security
- A feed check's trace keeps the feed's address without its query or password (#225).
1.0.0 - 2026-10-12
Added
- Anyone can browse the directory in a web browser, read only: the most subscribed, categories, every feed A to Z, a search by name, and each feed's page with its latest items and its address to copy into any podcast app (#215).
GET /v1/instanceslists the servers holding a snapshot, with their feeds and subscribers, for an admin token (#206).ipx-directory, a directory of public podcasts that ipx servers share: each sends the public feeds its people subscribe to, and any server with a token browses them, each feed listed once it has been read. Tokens are in its config file, each read-only, read-write or admin, and revoked one at a time (#204, #205).- Traces: a span a request, joining the trace of the ipx server that called, and a trace a refresher round with a span a feed (#213).