Feeds that never answer 'not modified' are downloaded whole every hour #242
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
ipx fetched 2.6 GB of feed bodies (decoded) in 24 h to 2026-10-15. Patreon ignores conditional requests: 35 Glass Cannon show feeds, checked hourly, send about 300 KB each every time (about 250 MB a day); Substack feeds about 1 MB each check. An unchanged body should be recognised (skip parsing and storing) and a feed that keeps sending the same body checked less often, back to its schedule when it changes.
Fixed in
8f19d88, deployed in ipx 1.2.1-dev: feeds.body_hash (SHA-256 of the last body) and same_since, added by create_missing. The same body again is taken as a 304 unless the feed has no items or artwork yet or the check is forced; without a schedule of its own, a feed the same for a while waits a twelfth of that, up to four hours, and is back on schedule when it changes (quiet(), beside the failure backoff). Tests: a_feed_long_the_same_is_checked_less_and_a_schedule_set_wins, the_same_body_again_is_not_modified_unless_a_check_is_forced, a_database_from_before_body_hashes_gains_the_columns. Every feed is read whole once more to record its hash; the saving shows in ipx_feed_bytes_total from tomorrow.