Compressed answers, a directory twice as patient, fewer whole re-reads (#239, #242, #243, #244)

Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.

- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
  back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
  something is missing a read would fill or the check is forced; and a feed without a schedule
  of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
  failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).

#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-15 13:54:00 +00:00
parent dbfe7918ac
commit 8f19d88fc9
17 changed files with 221 additions and 18 deletions

View File

@@ -302,8 +302,8 @@ then; a steady stream is worth an issue, since Cloudflare can rate-limit it. A s
`refused_private` is not 0 means an ipx server tried to send a feed that looks private: find which
in that server's log, since the directory refuses it but the sender should not have sent it. Many
`feed check failed` lines for one feed are the publisher's, not the directory's. Each feed is
checked every six hours, 25 at a time, so `feeds checked` over a day should be about four times
the feeds listed; none for over six hours means the refresher is stuck. `could not store a feed
checked every twelve hours (#243), 25 at a time, so `feeds checked` over a day should be about
twice the feeds listed; none for over twelve hours means the refresher is stuck. `could not store a feed
check` or `could not read which feeds are due` is its database (`ipx_directory` on the
`databases` project's Postgres). A config that does not load (`config changed but does not
load`) keeps the old tokens: worth fixing before someone's new token is needed.