diff --git a/CHANGELOG.md b/CHANGELOG.md index 6faeb08..6227361 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,6 +48,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- A limit on new downloads per check means a show's newest episodes: with it set to 3, ipx no longer works back through the show's history three at a time. Set to 0, it still takes the whole back catalogue. - With no limit on new downloads per check (`max_new_per_check = 0`), downloads work on Postgres; they failed. - Checking a feed with a long history is much quicker: items already stored are no longer written again on every check. - A scan no longer asks a feed's website for its icon every time; it asks again when the artwork changes or you refresh the feed. diff --git a/README.md b/README.md index 7fc3a43..473e4f6 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,8 @@ behind iPodderX (2004-2008, Ray Slakinski & August Trometer). sign in with a password or through a proxy (Cloudflare Zero Trust or Authentik), and admins manage accounts and settings. - **Scanning.** Feeds are checked on a schedule, globally or per feed, and a feed's own TTL is - honoured. Keyword, explicit-content and media-type filters decide what is downloaded, with a cap - on new downloads per scan. + honoured. Keyword, explicit-content and media-type filters decide what is downloaded, with a limit + on how many of a feed's newest episodes are downloaded. - **Downloads.** Files come over HTTP or BitTorrent and are filed into a folder per feed. Retention deletes the oldest files to stay under a disk quota or an age limit, and never touches an item someone has pinned. diff --git a/docs/configuration.md b/docs/configuration.md index 34fb3de..f24e803 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -52,8 +52,10 @@ database as described above; `download_dir`, `socket` and `organize` stay in con read it. `0` disables it entirely. * **`max_age_days`** — items older than this with no file on disk are pruned from the database. Kept ones stay. `0` disables it. -* **`max_new_per_check`** — the cap that stops a new subscription pulling a whole back catalogue. - `0` means unlimited, which is rarely what you want: subscribing to an OPML of 80 feeds with no cap +* **`max_new_per_check`** — how many of a feed's newest episodes are downloaded; older ones stay + listed to download by hand. It stops a new subscription pulling a whole back catalogue. + `0` means every episode, for an archive; set it on the feeds you want archived, since on the + global default it applies to every feed: subscribing to an OPML of 80 feeds with no limit fetched 216 files and 22 GB in one scan. * **`media_types`** — top-level MIME types taken automatically. Anything else is still listed and can be fetched by hand; blog feeds put each article's header image in an ``, and diff --git a/src/db.rs b/src/db.rs index dfd48a5..efb62de 100644 --- a/src/db.rs +++ b/src/db.rs @@ -623,16 +623,26 @@ pub struct Pending { } impl Db { - /// The download queue is the table, not the parse result: an enclosure held back by - /// `max_new_per_check` is simply picked up by the next scan, in feed order. + /// The download queue is the table, not the parse result: what a scan could not finish is + /// picked up by the next. Only from the feed's `limit` newest items with a file, though: a + /// limit of 3 means the three latest episodes. Taking the newest three still pending, each + /// full read of a feed took the three before the ones already downloaded, working back + /// through every show's history: 4420 files queued across 12 shows on the default of 3 + /// (#97). Unlimited, 0 in the settings, is the whole back catalogue, for an archive. An item + /// whose files were all skipped by a filter does not hold one of the places. #[tracing::instrument(skip_all)] pub async fn pending(&self, feed_id: &str, limit: usize) -> Result> { - // Newest first: a cap of 3 should mean the three latest episodes, not the three that - // happen to have been recorded first. self.rows( "SELECT x.id, x.url, x.mime FROM enclosures x JOIN entries e ON e.feed_id = x.feed_id AND e.guid = x.guid WHERE x.feed_id = $1 AND x.state = 'pending' + AND e.guid IN (SELECT n.guid FROM entries n + WHERE n.feed_id = $1 + AND EXISTS (SELECT 1 FROM enclosures y + WHERE y.feed_id = n.feed_id AND y.guid = n.guid + AND y.state <> 'skipped') + ORDER BY coalesce(n.published, n.first_seen) DESC, n.guid DESC + LIMIT $2) ORDER BY coalesce(e.published, e.first_seen) DESC, x.id DESC LIMIT $2", // Unlimited is usize::MAX, which `as i64` makes -1: SQLite reads LIMIT -1 as no @@ -1936,6 +1946,29 @@ mod tests { assert_eq!((list["f"].unread, list["g"].unread), (1, 1)); // a read, c hidden; sam's read is sam's } + #[tokio::test] + async fn a_limit_takes_the_newest_episodes_not_the_back_catalogue() { + let db = Db::memory().await.unwrap(); + db.exec_for_test( + "INSERT INTO entries (feed_id, guid, first_seen, published) VALUES + ('f','e1',0,100),('f','e2',0,200),('f','e3',0,300),('f','e4',0,400),('f','e5',0,500); + INSERT INTO enclosures (id, feed_id, guid, url, state, path) VALUES + (1,'f','e1','u1','pending',null),(2,'f','e2','u2','pending',null), + (3,'f','e3','u3','done','/tmp/3'),(4,'f','e4','u4','done','/tmp/4'), + (5,'f','e5','u5','skipped',null);", + ).await + .unwrap(); + let ids = |p: Vec| p.into_iter().map(|p| p.id).collect::>(); + // The newest three with a file worth having are e4, e3 and e2 (e5's was skipped): only + // e2 is waiting. e1 is back catalogue. + assert_eq!(ids(db.pending("f", 3).await.unwrap()), vec![2]); + // Once e2 is down, nothing: before #97 the next scan took e1. + db.exec_for_test("UPDATE enclosures SET state = 'done', path = '/tmp/2' WHERE id = 2").await.unwrap(); + assert!(db.pending("f", 3).await.unwrap().is_empty()); + // Unlimited is the archive: everything still waiting. + assert_eq!(ids(db.pending("f", usize::MAX).await.unwrap()), vec![1]); + } + #[tokio::test] async fn an_unlimited_queue_takes_everything_waiting() { let db = Db::memory().await.unwrap(); diff --git a/web/src/admin.ts b/web/src/admin.ts index eaf4b8b..61f351d 100644 --- a/web/src/admin.ts +++ b/web/src/admin.ts @@ -34,11 +34,11 @@ async function drawServer(){ Applies to every feed that does not set its own. A feed's suggested interval (its ttl) is still honoured when it asks to be polled less often. -
+
Applies to any feed that does not set its own — including every feed - inside an OPML subscription. 0 means unlimited, which will pull a whole back - catalogue the first time a feed is scanned.
+ inside an OPML subscription. Older episodes stay listed to download by hand. 0 means + every episode, the whole back catalogue.
Anything else is still listed and can be downloaded by hand — blog feeds diff --git a/web/src/dialogs.ts b/web/src/dialogs.ts index 46694e8..f98c58a 100644 --- a/web/src/dialogs.ts +++ b/web/src/dialogs.ts @@ -275,10 +275,10 @@ function settingsModal(f, newUrl?: string){ Comma separated words or phrases. An item with one in its title or text is hidden from you and not downloaded for you, as well as those your Settings hide everywhere.
`} -
+
- Blank follows the global default (${globalMax}). The rest wait for - the next scan.
+ Blank follows the global default (${globalMax}). Older episodes stay + listed to download by hand. 0 means every episode, the whole back catalogue.