Compressed answers, a directory twice as patient, fewer whole re-reads (#239, #242, #243, #244)

Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.

- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
  back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
  something is missing a read would fill or the check is forced; and a feed without a schedule
  of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
  failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).

#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-15 13:54:00 +00:00
parent dbfe7918ac
commit 8f19d88fc9
17 changed files with 221 additions and 18 deletions

View File

@@ -12,6 +12,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Changed
- A feed that sends the same body again, as Patreon and Substack do whatever they are asked, is taken as unchanged instead of read again, and, without a schedule of its own, checked less often the longer it stays the same, up to every four hours, back on schedule when it changes (#242).
- Feeds are fetched accepting brotli and zstd as well as gzip (#244).
## [1.2.0] - 2026-10-15
### Added