Compressed answers, a directory twice as patient, fewer whole re-reads (#239, #242, #243, #244)

Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.

- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
  back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
  something is missing a read would fill or the check is forced; and a feed without a schedule
  of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
  failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).

#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-15 13:54:00 +00:00
parent dbfe7918ac
commit 8f19d88fc9
17 changed files with 221 additions and 18 deletions

View File

@@ -52,7 +52,10 @@ art_cache_mb = 500 # artwork kept on disk. 0 = none
database as described above; `download_dir`, `socket` and `organize` stay in config.toml.
* **`schedule`** — how often feeds are re-checked. A feed's own `<ttl>` still wins when it asks to
be polled *less* often, and a per-feed `schedule` overrides both. A feed that keeps failing
be polled *less* often, and a per-feed `schedule` overrides both. Without a schedule of its own,
a feed that sends the same body check after check (Patreon, Substack, which ignore "not
modified") is checked less often the longer it stays the same, up to every four hours, and is
back on schedule once it changes. A feed that keeps failing
waits as long as it has been failing before the next try, up to a day, and is back on schedule
after its first success. A moment's failure to look its host up does not count: the network, not
the feed, failed, and it is tried again in ten minutes. Admin-only from the UI.