Measured 2026-10-16, uncompressed until now: the directory's answers (the feed list ipx reads,
603 KB of JSON, ~134 ms of a 159 ms Directory request), ipx's snapshot upload (~100 KB each half
hour), both /metrics (4.7 and 0.9 KB every 15 s) and every trace export to Tempo. Already
compressed: ipx-web's answers (zstd, #239), the home page and the directory's public pages
through Cloudflare (brotli, zstd).
- The directory compresses with brotli, zstd or gzip as asked, and unpacks a gzipped request.
- ipx sends its snapshot gzipped; the directory is deployed first.
- Both /metrics answer gzipped, as Prometheus asks.
- opentelemetry-otlp's gzip-http: OTEL_EXPORTER_OTLP_COMPRESSION=gzip compresses the export, set
for all three in production's compose file; Caddy's exporter reads the same variable.
- web/build.mjs writes .br (quality 11) and .gz (level 9) beside each text file, and Caddy serves
them in place (file_server precompressed): app.js 30 KB where zstd on the fly gave 37, app.css
11 where 14. A .br or .gz asked for by name is refused, or /admin.js.br would be the admin
page's script without the check that keeps it to admins; in a handle of its own, since a bare
respond runs after the handles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With apple_country set, the directory fetches Apple's top 100 podcasts and the top 50 of each of
its 19 categories for that country (itunes.apple.com/{cc}/rss/toppodcasts), looks up each
show's RSS address (/lookup, 100 ids a call, three seconds apart, as Apple asks for no more than
about twenty a minute), and stores them as the snapshot apple-podcasts with no subscribers. The
refresher fetches them like any feed; the most-subscribed chart is unchanged. Apple's primary
genre goes in as the snapshot's category, used only while a feed names none.
The same weekly run reads Apple's genre tree (ws/genres?id=26) and warns when it differs from
ipx_feed's CATEGORIES; on 2026-10-15 they were the same, all 19 and their subcategories.
A syncs table keeps when it last ran: at start if a week has passed, again weekly, an hour after
a failure. keep_instances spares apple-podcasts, which no token holds, and no token may take
the name. A trace a sync, a span each answer.
CLAUDE.md: builds one cargo command at a time, the load looked at before and after, after a
build chained with a test compile took Tower past 200.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#241: on production, All Subscriptions' query runs in 31 ms, unread oldest-first in 30, the
feed list's count in 5, generic plans included (EXPLAIN ANALYZE, BUFFERS, read only), against
0.3-1.6 s in ipx. A new Postgres connection takes 80-330 ms to open, and a page load asks for
more than are idle: four of six slow loads came as the pool grew (3 to 6, 2 to 6, 3 to 5, 6 to
8), and 20 requests at once took a median 414 ms while the pool grew from 8 to 20. The daemon
and the directory's server now keep their whole pool open (min = max); a command such as `ipx
status`, and the tests, which share the server's 100 connections, open as they need.
#246: the page reports list_shown, the ms from asking for a list to drawing it, with the view
and filter, under the click's trace; background refreshes are not timed. Timings skip report()'s
once-a-minute limit, which is for errors in a loop. A dashboard panel and table, and the prod
check, read it.
#245: the load test's browse scenario subscribes by OPML and leaves the Directory, which is
ipx-directory's since #204, to it.
CLAUDE.md records the owner's leave to EXPLAIN (ANALYZE, BUFFERS) any query on production,
read only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.
- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
something is missing a read would fill or the check is forced; and a feed without a schedule
of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).
#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fetch that failed logged reqwest's words, which name the whole address: Patreon's auth key
went to Loki on 2026-10-15, and, since scan events go to every open page, to everyone's
browser too. #225 had taken queries out of the traces' address fields only.
Every event is redacted where it is emitted, for the log, the live stream and a terminal
alike: address fields lose their query and password, and so does any address in an error's
words (feed::redact_text). The artwork proxy's warnings and the directory's failed checks
the same. The error kept on the feed, shown to its own subscribers, keeps the detail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the slow traces of 2026-10-13: the directory checked a round's 25 feeds one after
another, so two libsyn feeds timing out at 30 s made a round 70 s; a forced scan looked up
62 site icons in turn, 22 s of 62; mjg59.dreamwidth.org takes the connection and never
answers, so it ran the full 30 s in every scan it was in; and since #225 strips queries,
every YouTube fetch span read the same address, outside any span naming its feed.
The directory now checks five at a time (SQLite stays on one connection, so its writes
queue). A forced scan looks up the icon in the fetch-ahead task, which already runs six
deep, and hands it to scan_one with the body. Both feed clients give up after 15 s of
silence (no successful fetch took over 8 s in a day of traces); FEED_TIMEOUT stays 30 s
for a feed still sending. The directory's client had no connect timeout; it has ipx's 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fetch span and the download span recorded the whole address, so a private feed's
credential (Patreon's ?auth=) went to Tempo with every scan, readable by anyone with
Grafana. feed::redact drops the query, fragment and user:password; a token in the path
would still go, noted where it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx's resident memory climbed from about 40 MB to 500-850 MB each night while scans ran five
times as often, and stayed there until a restart; the directory sat at 1.25 GiB idle. glibc's
allocator keeps what each thread's arena freed. mimalloc gives freed pages back, so resident
memory now follows what is in use. jemalloc was the first choice for its in-use figures, but its
upstream was archived in 2025.
Declared once in the directory crate, which ipx builds in, so both binaries get it.
What was missing to see it: the directory served no /metrics (now on metrics_bind, a port apart
from the tunnel's), nothing counted the feed bytes fetched, and the fetch span had no body size.
The feed crate's spans never reached Tempo either: both trace exporters filtered to their own
crate, so ipx_feed's fetch and parse spans were dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker runs a healthcheck with exec, and containerd puts each exec's state in XDG_RUNTIME_DIR,
/run/user/0 on Tower, which elogind removes when root's last session ends. Every check then
failed and every container showed unhealthy while all of them answered. The checks are gone from
the sample compose file and production's; ipx's library counts for /metrics, which only the
healthcheck's status call refreshed, are counted every 30 s by the daemon.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each release is now a Gitea release as well as an image: scripts/release.sh takes the binary (or
the page's files) out of the deployed image, checks its version, and attaches it as a tarball with
the changelog section as the notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The directory renders its own pages from the store: the most subscribed, categories, A to Z,
search, and a page per feed with its address and a Copy button. No token, nothing that writes; /v1
keeps its token check. The publisher's HTML is shown as text, item links only when http(s), with
a strict Content-Security-Policy; covers load from their publishers with no referrer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Directory feed's page offers Check now and Remove to an admin whose token is rwa (the page
learns it from can_admin on the feed's page, asked of the directory only for admins). Admin,
Directory lists the servers sharing, from a new GET /v1/instances, and the blocked feeds by
title: a feed is often blocked because its address carries a key, so the address stays in the
directory. Unblocking or dropping a snapshot sends this server's snapshot again at once; since
directory's 403 reaches the page as 403, not 502.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workspace stops holding one version. ipx keeps 0.12.1-dev and its vX.Y.Z tags; the directory
starts at 0.1.0-dev, tagged directory-vX.Y.Z, and the page at 0.1.0-dev in package.json, tagged
web-vX.Y.Z, which web/build.mjs gives the script for the logo's tooltip beside ipx's. crates/feed
is a library, not released. CLAUDE.md's release steps are for one service at a time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CHANGELOG.md moves to crates/ipx/CHANGELOG.md with its history, which until 0.12.0 was all of
ipx's; web/CHANGELOG.md and crates/directory/CHANGELOG.md start with this release's lines for the
page and the directory. The dashboard's lines go: monitoring is not a service anyone uses.
CLAUDE.md says which file a change goes in, and that a release covers each.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
