Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.
- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
something is missing a read would fill or the check is forced; and a feed without a schedule
of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).
#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fetch that failed logged reqwest's words, which name the whole address: Patreon's auth key
went to Loki on 2026-10-15, and, since scan events go to every open page, to everyone's
browser too. #225 had taken queries out of the traces' address fields only.
Every event is redacted where it is emitted, for the log, the live stream and a terminal
alike: address fields lose their query and password, and so does any address in an error's
words (feed::redact_text). The artwork proxy's warnings and the directory's failed checks
the same. The error kept on the feed, shown to its own subscribers, keeps the detail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the slow traces of 2026-10-13: the directory checked a round's 25 feeds one after
another, so two libsyn feeds timing out at 30 s made a round 70 s; a forced scan looked up
62 site icons in turn, 22 s of 62; mjg59.dreamwidth.org takes the connection and never
answers, so it ran the full 30 s in every scan it was in; and since #225 strips queries,
every YouTube fetch span read the same address, outside any span naming its feed.
The directory now checks five at a time (SQLite stays on one connection, so its writes
queue). A forced scan looks up the icon in the fetch-ahead task, which already runs six
deep, and hands it to scan_one with the body. Both feed clients give up after 15 s of
silence (no successful fetch took over 8 s in a day of traces); FEED_TIMEOUT stays 30 s
for a feed still sending. The directory's client had no connect timeout; it has ipx's 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fetch span and the download span recorded the whole address, so a private feed's
credential (Patreon's ?auth=) went to Tempo with every scan, readable by anyone with
Grafana. feed::redact drops the query, fragment and user:password; a token in the path
would still go, noted where it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx's resident memory climbed from about 40 MB to 500-850 MB each night while scans ran five
times as often, and stayed there until a restart; the directory sat at 1.25 GiB idle. glibc's
allocator keeps what each thread's arena freed. mimalloc gives freed pages back, so resident
memory now follows what is in use. jemalloc was the first choice for its in-use figures, but its
upstream was archived in 2025.
Declared once in the directory crate, which ipx builds in, so both binaries get it.
What was missing to see it: the directory served no /metrics (now on metrics_bind, a port apart
from the tunnel's), nothing counted the feed bytes fetched, and the fetch span had no body size.
The feed crate's spans never reached Tempo either: both trace exporters filtered to their own
crate, so ipx_feed's fetch and parse spans were dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docker runs a healthcheck with exec, and containerd puts each exec's state in XDG_RUNTIME_DIR,
/run/user/0 on Tower, which elogind removes when root's last session ends. Every check then
failed and every container showed unhealthy while all of them answered. The checks are gone from
the sample compose file and production's; ipx's library counts for /metrics, which only the
healthcheck's status call refreshed, are counted every 30 s by the daemon.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each release is now a Gitea release as well as an image: scripts/release.sh takes the binary (or
the page's files) out of the deployed image, checks its version, and attaches it as a tarball with
the changelog section as the notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The directory renders its own pages from the store: the most subscribed, categories, A to Z,
search, and a page per feed with its address and a Copy button. No token, nothing that writes; /v1
keeps its token check. The publisher's HTML is shown as text, item links only when http(s), with
a strict Content-Security-Policy; covers load from their publishers with no referrer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Directory feed's page offers Check now and Remove to an admin whose token is rwa (the page
learns it from can_admin on the feed's page, asked of the directory only for admins). Admin,
Directory lists the servers sharing, from a new GET /v1/instances, and the blocked feeds by
title: a feed is often blocked because its address carries a key, so the address stays in the
directory. Unblocking or dropping a snapshot sends this server's snapshot again at once; since
directory's 403 reaches the page as 403, not 502.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workspace stops holding one version. ipx keeps 0.12.1-dev and its vX.Y.Z tags; the directory
starts at 0.1.0-dev, tagged directory-vX.Y.Z, and the page at 0.1.0-dev in package.json, tagged
web-vX.Y.Z, which web/build.mjs gives the script for the logo's tooltip beside ipx's. crates/feed
is a library, not released. CLAUDE.md's release steps are for one service at a time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CHANGELOG.md moves to crates/ipx/CHANGELOG.md with its history, which until 0.12.0 was all of
ipx's; web/CHANGELOG.md and crates/directory/CHANGELOG.md start with this release's lines for the
page and the directory. The dashboard's lines go: monitoring is not a service anyone uses.
CLAUDE.md says which file a change goes in, and that a release covers each.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The exporter setup and the traceparent parser move from ipx into ipx_directory::otel, which ipx
already depends on, so both binaries share them. The directory makes a span a request, a child of
the caller's traceparent, and a trace a refresher round with a span a feed; ipx sends the current
span's traceparent on every call to the directory and wraps each share in a span. The dashboard's
Traces row and the prod-check skill cover all three services.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy logs one JSON line a request, filtered: no ?token= in the address and no request
headers, which hold the session cookie and Access's token. Its default log, errors included,
has the same filter, since an error line carries the request too. The directory logs a line a
refresher round, so the dashboard can show its progress.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx asks ipx-directory on its people's behalf, with the token an admin set
in the new Admin, Directory tab, so the page never talks to it and never
sees the token or a feed's address. /api/directory, /api/popular and a
feed's Directory page answer from it; subscribing takes the address from
the directory and goes through Add feed's steps. With no directory set
up, the Directory says so. The local directory code is gone; its privacy
checks are the snapshot builder's now, and their tests moved with them.
With sharing on, ipx sends its snapshot after every scan, whenever the
config changes (set_cfg) and every half hour: each public feed and how
many here subscribe, never an OPML or Patreon creator itself or anything
private. A feed can be marked private in its settings, for a paid feed
whose address does not give it away; anyone subscribed may mark it, only
an admin unmark it. The admin's category for a feed that names none goes
with it, which the directory uses only while the feed names none, so the
category an admin gives a blog still reaches the Directory.
The directory lists only feeds it has read as a feed: the first snapshot
after a start went before the first scan, when an OPML was not yet known
to be one, and the directory listed the OPML. It now fetches what a
snapshot brings in at once instead of within the minute.
The browser suite runs its own ipx-directory on SQLite; the Directory's
tests match feeds by title, since the directory names them by address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new crate and service, crates/directory, live at ipx-dir.sdf1.net. ipx
servers send the public feeds their people subscribe to (POST /v1/snapshot,
replacing what that token sent before); the directory fetches each feed's
title, artwork, category, description and latest twenty items every six
hours with ipx-feed, and lists them A to Z and by subscribers summed over
every server. It holds metadata only, never a file or a file's address.
Tokens are in its config file as SHA-256 hashes, each ro, rw or rwa, and
the file is read again within a minute of a change, so adding and revoking
one is an edit and nothing else; a revoked token takes its server's counts.
At first they were to come from environment variables, which would have
meant recreating the container to add one. A file that does not load keeps
the tokens as they were.
Every address sent is checked with ipx_feed::url_looks_private, the check
ipx makes before sending, so one server's miss does not publish a paid feed
to every other; an rwa token can remove and block what both miss.
Deployed: ipx-directory in the ipx Arcane project, database ipx_directory
(its own login) on the databases project's Postgres, which db-backup
already covers, and a tunnel route with no Access in front. The store's
tests run on Postgres as well as SQLite when IPX_TEST_DATABASE_URL is set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
