Every scan downloads three more old episodes of each show, working back through its whole history #97
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
max_new_per_check (3) is documented as "the cap that stops a new subscription pulling a whole back catalogue", but it only slows it down. pending() takes the newest 3 still-pending files each time a feed is read in full, so once the latest three are downloaded, every later full read takes the next three older ones. 4525 files are pending, most of them back catalogue.
Seen 2026-09-29 in the download_done events: Ken and Robin 691, 690, 689 in the 14:12 scan, then 682, 681, 680 in the full scan at 18:09; Clarkesworld 04_26, then 03_26, then 02_26; Pivot's November 2025 episodes, then October's. Nothing was downloaded twice (177 downloads, all distinct enclosures). The user saw files downloading but not in the list, since old episodes sit far down it.
It happens on every full read: a forced scan, or a feed that changed or sends no validators. Left alone it downloads every show's whole history, 3 files at a time. When max_total_gb is reached the reaper deletes by oldest downloaded_at, which would be the newest episodes, the first ones taken.
Query: {container="iPX"} |= ""ev":"download_done"" | json | line_format "{{.timestamp}} {{.feed}} {{.url}}"
Decided 2026-09-29: keep the current behaviour for now (working back through each show's history, 3 files per full check). Left open so it can be picked up again. If it is: the fix considered was limiting pending() to the feed's newest max_new_per_check items with a file, so older episodes are only fetched by hand. The reaper deleting by oldest downloaded_at once max_total_gb is reached, which would take the newest episodes first, is part of the same question.
With max_total_gb at 0 (unlimited, production's setting per the user on 2026-09-29), the reaper never deletes for space, so the newest-deleted-first concern does not apply. What does: nothing bounds the back-catalogue downloads. 4420 files pending, today's downloads averaging 69 MB, about 300 GB in all, against 323 GB free on /mnt/user (the whole array; ipx's downloads are 60 GB now). Growth is 3 files per show per full read of its feed.
Working as intended: the user wants each show's whole back catalogue kept as an archive (2026-09-29). With max_new_per_check at 0 a feed takes everything at once; with a limit, it works back through the history that many files per full read. Not to be changed to newest-only.
Reopened 2026-09-29: the 4420 pending files all belong to 12 shows on the server default limit of 3, which the user means as 'the latest three'. The archive shows have limits of their own (100 and 10000) and nothing pending. So a limit of N means the newest N episodes; 0 (unlimited) is the whole archive.
Fixed in
527777e: pending() takes only from a feed's newest max_new_per_check items with a file, so a limit of 3 means the latest three. 0 still takes everything, for an archive. The feed and admin settings now call it 'Newest episodes to download' and say what 0 does. The 4420 back-catalogue files stay listed and pending, to download by hand; nothing downloaded was deleted. Tested on SQLite and Postgres. Deployed.