Compressed answers, a directory twice as patient, fewer whole re-reads (#239, #242, #243, #244)

Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.

- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
  back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
  something is missing a read would fill or the check is forced; and a feed without a schedule
  of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
  failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).

#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-15 13:54:00 +00:00
parent dbfe7918ac
commit 8f19d88fc9
17 changed files with 221 additions and 18 deletions

View File

@@ -10,6 +10,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Changed
- Each feed is checked every twelve hours instead of six (#243).
- Feeds are fetched accepting brotli and zstd as well as gzip (#244).
## [1.1.1] - 2026-10-15
### Changed

View File

@@ -117,8 +117,8 @@ pub struct Blocked {
pub const MAX_SNAPSHOT: usize = 20_000;
/// How often each feed is fetched again. Metadata changes slowly, and every server subscribed
/// to a feed fetches it itself anyway.
pub const REFRESH_EVERY: i64 = 6 * 3600;
/// to a feed fetches it itself anyway: every six hours was 3.9 GB of feeds a day (#243).
pub const REFRESH_EVERY: i64 = 12 * 3600;
pub struct Directory {
pub store: Store,