diff --git a/CHANGELOG.md b/CHANGELOG.md index 48d2c44..cbcb313 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,26 @@ The long form, with what was wrong before and how it was found, is in ## [Unreleased] +### Added + +- A Patreon token pasted into Add feed, or a creator's RSS link without `&show=`, becomes a folder + of that creator's shows, kept in step on every scan like a subscribed OPML. A creator with only + one show stays a plain feed. One already added as a single long feed is split into its shows on + its next scan, keeping its files and what you had read. +- Add feed has an "Allow items marked explicit" box, so a new feed's first scan no longer skips + every explicit item. + +### Fixed + +- Turning on Allow explicit, or changing keywords or auto-download, brings back what those settings + had skipped on the feed's next scan. Before, an item was judged once, when first seen, and a + skipped one stayed skipped whatever you changed. +- Feeds inside an OPML or a Patreon creator follow your settings on the folder unless you set their + own, as the folder's settings dialog said they did. Before, the folder's settings reached nothing + inside it. +- A new feed no longer takes the name of one you removed earlier and shows that feed's old items. + Re-adding the same feed still gets its old name, and its history, back. + ### Changed - The README is a short overview of what ipx does and how to run it, and points into `docs/` for diff --git a/docs/architecture.md b/docs/architecture.md index 4be32ed..f27245d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -28,12 +28,15 @@ The page is compiled in, so **editing `web/index.html` needs a rebuild**. 1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this). 2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there. 3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are - re-derived into the database each scan, never written to config.toml. + re-derived into the database each scan, never written to config.toml. A Patreon creator link + (a token, no `show=`) with more than one show is treated the same way, before any fetch: its + shows come from Patreon's web API and each becomes a derived feed. 4. Record entries. A changed title or description flips the item back to unread. 5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice. 6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with - a reason. + a reason. What a filter skipped is judged again every scan, so a change of settings brings it + back. A feed in a group takes your settings on the group for anything you have not set on it. 7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to the torrent path whatever its advertised type; an HTML body is a failed download — a login wall or an error page — and is deleted. diff --git a/docs/history.md b/docs/history.md index 40e1497..e653488 100644 --- a/docs/history.md +++ b/docs/history.md @@ -6,6 +6,68 @@ reasoning lives. New write-ups go at the top. See [README.md](../README.md) for what the thing is. +## 2026-09-11 — A Patreon creator is a list of shows + +Ray asked whether ipx could sync with Patreon. Not in full. The documented API (v2, the +`identity.memberships` scope) lists the creators you back and whether each has a feed (`has_rss`), +but no resource carries the `auth` token that makes a feed URL work. That token only comes from the +creator's page. It is also one per membership, not one per account: techpod's differs from Glass +Cannon's, so no single token finds everything you back. + +What does work is one creator at a time, which is what Ray wanted for Glass Cannon and its 33 shows: + +- `patreon.com/rss?auth=`, with no creator named, returns that token's creator. Its self link, + about 660 bytes in, gives the campaign by number (`/rss/369921`). Patreon ignores `Range` here, so + ipx reads the stream until the number appears and hangs up, instead of taking all 2.8 MB. +- A show's `show=` number is a Patreon collection. Asked anonymously, the collection listing + (`/api/collection?filter[campaign_id]=`) and a post's `collections` both hide patron-only ones: you + get "FAQ". `/api/campaigns/?include=shows` lists every show, anonymously, in one response. +- Every spelling works: `rss/glasscannon?auth=…&show=N`, `rss/369921?…` and `rss?auth=…&show=N` all + serve the same 131 items. Enclosure URLs are the same in the creator feed and the show feed, and + stable between fetches. + +That last point shaped the design. `enclosures.url` is unique, so whichever feed is scanned first owns +the file. The first cut only asked a creator for its shows while it had no entries of its own, so that a +creator already read as a plain feed, holding every show's episodes, would never be split into shows +that came up empty. Within the hour that was the wrong call: Glass Cannon had gone into production on +the build before this one, been read as one feed of 2,385 items, and the rule kept it that way. Finding +anything in that heap was the problem Ray wanted solved. + +So a creator with more than one show is always a group, run through the same sync as an OPML +(`sync_group`, split out of `sync_opml`). When it becomes one, its items are cleared and each show +takes over the enclosures the creator holds as the show lists them (`Db::adopt`), downloaded files and +everyone's read state included. One show leaves it a plain feed, which is what techpod already was. If +the shows cannot be listed, a creator already split fails the scan rather than being read as one heap; +one that never was is read as one feed until they can be. An answer without a `shows` list is an error, +not "no shows". + +**Filter verdicts follow the settings.** Ray also reported that turning on Allow explicit and +rescanning brought nothing back. An item was judged once, when first seen, and `skipped` was final. The +2026-09-10 entry below saw it coming ("worth a `ipx retry ` command if this bites"). It bit: +2,166 Glass Cannon items and all 88 of Shadowdark's sat at `skipped: explicit` with the setting on. +Every scan now runs the filters again over what they skipped (not over `torrents disabled`, which is not +a filter's call) and requeues what they now let through. Only that direction: a queued item is never +pulled back, because Download latest and a manual download both work by queueing. + +**Two gaps beside it.** Add feed had no explicit box, so every new feed's first scan skipped all its +explicit items; it has one now, stored on your subscription like the feed dialog's. And a feed inside a +group ignored your settings on the group, though the group's dialog said they were inherited: settings +live on each person's subscription, and nothing read the group's. `Db::subscribers` now fills what you +have not set on the feed from your subscription to the group, and the feed list shows the same. + +**A name that was already used.** Replaying the split on a copy of the production database left one +show with a Supercast episode in it. "Glass Cannon Live! Ascension | Pathfinder 2E" slugs to +`glass-cannon-live-ascension-pathfinder-2`, the id of a Supercast feed of the same show that had been +removed. Removing a feed keeps its rows on purpose, so that re-adding it does not fetch the back +catalogue again, but choosing a new id only checked config.toml and derived feeds. The Patreon show took +the old id and everything still filed under it. An id is now also taken when the database has a feed by +that id at a different URL; the same URL may still have it back, which is the re-add case. + +Shows already added by hand are matched by token and show number, not by exact URL (`same_feed`), so a +bare token does not add Get in the Trunk and Shadowdark a second time under another spelling. + +The show listing is Patreon's own undocumented web API. If it changes, only finding new shows stops. + ## 2026-09-11 — One meaning per icon, sortable columns, and one player Ray asked for a pass over the whole UI: consistent icons, and buttons placed next to what they act diff --git a/src/db.rs b/src/db.rs index 34526e1..79f7473 100644 --- a/src/db.rs +++ b/src/db.rs @@ -920,14 +920,20 @@ impl Db { /// Everyone's settings for one feed. The scanner merges these into what it fetches /// and downloads, since one file serves the lot. - pub fn subscribers(&self, feed_id: &str) -> Result> { + /// In a group, whatever someone has not set on the feed itself comes from their + /// subscription to the group, as the group's settings dialog has always said it does. + pub fn subscribers(&self, feed_id: &str, group: Option<&str>) -> Result> { let conn = self.conn.lock().unwrap(); let mut stmt = conn.prepare( - "SELECT keywords, auto_download, allow_explicit, max_new_per_check - FROM subscriptions WHERE feed_id = ?1", + "SELECT coalesce(c.keywords, p.keywords), coalesce(c.auto_download, p.auto_download), + coalesce(c.allow_explicit, p.allow_explicit), + coalesce(c.max_new_per_check, p.max_new_per_check) + FROM subscriptions c + LEFT JOIN subscriptions p ON p.user_id = c.user_id AND p.feed_id = ?2 + WHERE c.feed_id = ?1", )?; let out = stmt - .query_map([feed_id], |r| { + .query_map(params![feed_id, group], |r| { Ok(Sub { feed_id: feed_id.to_string(), keywords: r @@ -1325,6 +1331,71 @@ impl Db { Ok(()) } + /// Empties a feed of its items, leaving its files alone. + pub fn clear_entries(&self, feed_id: &str) -> Result<()> { + let conn = self.conn.lock().unwrap(); + conn.execute("DELETE FROM entries WHERE feed_id = ?1", [feed_id])?; + Ok(()) + } + + /// Every feed the database holds rows for, as (id, url), removed ones included. + pub fn feed_urls(&self) -> Result> { + let conn = self.conn.lock().unwrap(); + let mut stmt = conn.prepare("SELECT id, coalesce(url, '') FROM feeds")?; + let out = stmt + .query_map([], |r| Ok((r.get(0)?, r.get(1)?)))? + .collect::>()?; + Ok(out) + } + + /// Hands a feed in a group the enclosures its parent holds, as (guid, url), with everyone's + /// read state for them. A Patreon creator read as one feed before it was split into shows + /// owns every show's files, and `enclosures.url` is unique, so without this each show would + /// list its items with nothing to play. + pub fn adopt(&self, parent: &str, child: &str, listed: &[(&str, &str)]) -> Result<()> { + let mut conn = self.conn.lock().unwrap(); + let holds: bool = conn.query_row( + "SELECT EXISTS (SELECT 1 FROM enclosures WHERE feed_id = ?1)", + [parent], + |r| r.get(0), + )?; + if !holds { + return Ok(()); // An OPML, or a creator already shared out. + } + let tx = conn.transaction()?; + for &(guid, url) in listed { + let moved = tx.execute( + "UPDATE enclosures SET feed_id = ?3, guid = ?4 WHERE url = ?1 AND feed_id = ?2", + params![url, parent, child, guid], + )?; + if moved == 1 { + // Patreon gives a post the same guid in every feed it appears in. + tx.execute( + "UPDATE OR IGNORE entry_state SET feed_id = ?2 WHERE feed_id = ?1 AND guid = ?3", + params![parent, child, guid], + )?; + } + } + tx.commit()?; + Ok(()) + } + + /// A feed's enclosures skipped by one of its filters, by URL, with the reason: the verdicts a + /// change of settings can overturn. A torrent held back while torrents are off is not a + /// filter's call. + pub fn skipped_by_filter(&self, feed_id: &str) -> Result> { + let conn = self.conn.lock().unwrap(); + let mut stmt = conn.prepare( + "SELECT url, last_error FROM enclosures + WHERE feed_id = ?1 AND state = 'skipped' AND last_error IS NOT NULL + AND last_error != 'torrents disabled'", + )?; + let out = stmt + .query_map([feed_id], |r| Ok((r.get(0)?, r.get(1)?)))? + .collect::>()?; + Ok(out) + } + pub fn set_orphaned(&self, feed_id: &str, on: bool) -> Result<()> { let conn = self.conn.lock().unwrap(); conn.execute( @@ -1561,6 +1632,61 @@ mod tests { assert_eq!(state(4), "done"); } + #[test] + fn a_show_takes_over_what_its_creator_held() { + let db = Db::memory().unwrap(); + db.exec_for_test( + "INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0); + INSERT INTO enclosures (id, feed_id, guid, url, state, path, last_error) VALUES + (1,'creator','a','u1','done','/x/a.mp3',NULL), + (2,'creator','b','u2','skipped',NULL,'explicit'), + (3,'creator','c','u3','skipped',NULL,'explicit'), + (4,'other','d','u4','skipped',NULL,'torrents disabled'); + INSERT INTO entry_state (user_id, feed_id, guid, read) VALUES (1,'creator','a',1);", + ) + .unwrap(); + db.adopt("creator", "show", &[("a", "u1"), ("b", "u2"), ("d", "u4")]).unwrap(); + { + let conn = db.conn.lock().unwrap(); + let owner = |id: i64| -> String { + conn.query_row("SELECT feed_id FROM enclosures WHERE id = ?1", [id], |r| r.get(0)).unwrap() + }; + assert_eq!(owner(1), "show", "a downloaded file moves with its item"); + assert_eq!(owner(2), "show"); + assert_eq!(owner(3), "creator", "this show does not list it"); + assert_eq!(owner(4), "other", "only the parent's are taken"); + let read: String = conn + .query_row("SELECT feed_id FROM entry_state WHERE user_id = 1 AND guid = 'a'", [], |r| r.get(0)) + .unwrap(); + assert_eq!(read, "show", "what you had read stays read"); + } + + // Only a filter's verdict can be overturned by a change of settings. + let skipped = db.skipped_by_filter("show").unwrap(); + assert_eq!(skipped.get("u2").map(String::as_str), Some("explicit")); + assert_eq!(skipped.len(), 1); + assert!(db.skipped_by_filter("other").unwrap().is_empty(), "torrents disabled is not a filter"); + } + + #[test] + fn a_feed_in_a_group_follows_your_settings_on_the_group() { + let db = Db::memory().unwrap(); + db.exec_for_test( + "INSERT INTO users (id, name, is_admin, created) VALUES (1,'ray',1,0),(2,'sam',0,0); + INSERT INTO subscriptions (user_id, feed_id, allow_explicit, created) VALUES + (1,'group',1,0),(1,'show',NULL,0),(2,'group',1,0),(2,'show',0,0);", + ) + .unwrap(); + let explicit = |group| -> Vec> { + let mut v: Vec<_> = + db.subscribers("show", group).unwrap().into_iter().map(|s| s.allow_explicit).collect(); + v.sort(); + v + }; + assert_eq!(explicit(Some("group")), [Some(false), Some(true)], "ray inherits; sam's own choice on the show wins"); + assert_eq!(explicit(None), [None, Some(false)], "outside a group nothing is inherited"); + } + #[test] fn enclosure_url_is_the_dedupe_key() { let db = Db::memory().unwrap(); diff --git a/src/feed.rs b/src/feed.rs index 03db071..8419601 100644 --- a/src/feed.rs +++ b/src/feed.rs @@ -117,6 +117,103 @@ pub fn opml_title(bytes: &[u8]) -> Option { .filter(|t| !t.is_empty()) } +/// The token and show of a Patreon feed link, or None for any other URL. +/// +/// Patreon gives each patron one token per creator. With no show it stands for the creator, +/// whose feed carries every show at once. +fn patreon_parts(url: &str) -> Option<(String, Option)> { + let u = url::Url::parse(url).ok()?; + if !matches!(u.host_str()?, "patreon.com" | "www.patreon.com") || !u.path().starts_with("/rss") { + return None; + } + let param = |name: &str| u.query_pairs().find(|(k, _)| k == name).map(|(_, v)| v.into_owned()); + Some((param("auth")?, param("show"))) +} + +/// A Patreon link naming a creator but no show. +pub fn is_patreon_creator(url: &str) -> bool { + matches!(patreon_parts(url), Some((_, None))) +} + +/// What was typed into Add feed, as a URL. A bare Patreon token is taken as its creator's +/// feed, since the token alone says whose it is. +pub fn expand_input(input: &str) -> String { + let s = input.trim(); + let token = s.len() >= 20 && s.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'); + if token { format!("https://www.patreon.com/rss?auth={s}") } else { s.to_owned() } +} + +/// Whether two URLs are the same feed. One Patreon show has several spellings -- by the +/// creator's name, by number, or with no creator at all -- and the token and show are what +/// identify it. +pub fn same_feed(a: &str, b: &str) -> bool { + a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b)) +} + +/// A Patreon creator's name and shows, each show as (title, feed URL). +/// +/// ponytail: Patreon's own web API, undocumented, asked without signing in. If it changes, +/// finding shows stops and the show feeds already found keep working. The documented API +/// needs an OAuth client per install and does not list shows. +pub async fn patreon_shows( + client: &reqwest::Client, + url: &str, +) -> Result<(Option, Vec<(String, String)>)> { + // The creator feed names its campaign by number in its self link, a few hundred bytes in. + // The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up. + let mut resp = client.get(url).send().await.context("connecting")?; + if !resp.status().is_success() { + return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status())); + } + let mut head = Vec::new(); + while patreon_campaign(&head).is_none() && head.len() < 64 * 1024 { + let Some(chunk) = resp.chunk().await.context("reading the feed")? else { break }; + head.extend_from_slice(&chunk); + } + let campaign = patreon_campaign(&head) + .ok_or_else(|| anyhow!("the Patreon feed does not say whose it is"))?; + + let api = format!( + "https://www.patreon.com/api/campaigns/{campaign}\ + ?include=shows&fields%5Bcampaign%5D=name&fields%5Bcollection%5D=title" + ); + let resp = client.get(api).send().await.context("asking Patreon for the shows")?; + if !resp.status().is_success() { + return Err(anyhow!("Patreon would not list the shows: HTTP {}", resp.status())); + } + let (name, shows) = parse_patreon_shows(&resp.bytes().await.context("reading the shows")?)?; + Ok((name, shows.into_iter().map(|(id, title)| (title, format!("{url}&show={id}"))).collect())) +} + +/// The campaign number in the start of a Patreon feed. +fn patreon_campaign(head: &[u8]) -> Option { + let text = String::from_utf8_lossy(head); + text.match_indices("patreon.com/rss/").find_map(|(i, m)| { + let id: String = text[i + m.len()..].chars().take_while(char::is_ascii_digit).collect(); + (!id.is_empty()).then_some(id) + }) +} + +/// A campaign's name and its shows as (id, title), from Patreon's JSON:API answer. +fn parse_patreon_shows(json: &[u8]) -> Result<(Option, Vec<(String, String)>)> { + let v: serde_json::Value = serde_json::from_slice(json).context("Patreon's answer is not JSON")?; + // Missing is not the same as none. Read as no shows, the creator feed would be scanned as + // a plain feed, claim every show's files, and leave the shows empty once the list returned. + let ids = v["data"]["relationships"]["shows"]["data"] + .as_array() + .ok_or_else(|| anyhow!("Patreon's answer does not list the shows"))?; + let title = |id: &str| -> Option { + let show = v["included"].as_array()?.iter().find(|x| x["type"] == "collection" && x["id"] == id)?; + show["attributes"]["title"].as_str().map(|t| t.trim().to_owned()) + }; + let shows = ids + .iter() + .filter_map(|s| s["id"].as_str()) + .map(|id| (id.to_owned(), title(id).unwrap_or_else(|| format!("Show {id}")))) + .collect(); + Ok((v["data"]["attributes"]["name"].as_str().map(str::to_owned), shows)) +} + /// RSS first, then Atom -- the same split the original made on `parsedFeed.version`. pub fn parse(bytes: &[u8]) -> Result { match rss::Channel::read_from(bytes) { @@ -550,6 +647,45 @@ mod tests { assert!(!is_opml(include_bytes!("../tests/data/atom.xml"))); } + #[test] + fn a_patreon_creator_is_a_list_of_its_shows() { + let tok = "AbCdEfGhIjKlMnOpQrStUvWxYz012_-9"; + assert_eq!(expand_input(&format!(" {tok} ")), format!("https://www.patreon.com/rss?auth={tok}")); + assert_eq!(expand_input("https://example.com/rss"), "https://example.com/rss"); + + assert!(is_patreon_creator(&format!("https://www.patreon.com/rss/glasscannon?auth={tok}"))); + assert!(is_patreon_creator(&format!("https://www.patreon.com/rss?auth={tok}"))); + assert!(!is_patreon_creator(&format!("https://www.patreon.com/rss/x?auth={tok}&show=1")), "one show is a feed"); + assert!(!is_patreon_creator(&format!("https://example.com/rss?auth={tok}"))); + + // The show you already have by name is the one a bare token would add by number. + assert!(same_feed( + &format!("https://www.patreon.com/rss/glasscannon?auth={tok}&show=2073588"), + &format!("https://www.patreon.com/rss?auth={tok}&show=2073588"), + )); + assert!(!same_feed( + &format!("https://www.patreon.com/rss?auth={tok}&show=1"), + &format!("https://www.patreon.com/rss?auth={tok}&show=2"), + )); + + // The self link carries the campaign by number, whichever spelling was asked for. + let head = br#"https://www.patreon.com/glasscannon + "#; + assert_eq!(patreon_campaign(head).as_deref(), Some("369921")); + assert_eq!(patreon_campaign(b"T"), None); + + let json = br#"{"data":{"id":"369921","type":"campaign","attributes":{"name":"The Glass Cannon Network"}, + "relationships":{"shows":{"data":[{"id":"2073588","type":"collection"},{"id":"2073636","type":"collection"}]}}}, + "included":[{"id":"2073588","type":"collection","attributes":{"title":"Get in the Trunk "}}, + {"id":"2073636","type":"collection","attributes":{"title":"Shadowdark"}}]}"#; + let (name, shows) = parse_patreon_shows(json).unwrap(); + assert_eq!(name.as_deref(), Some("The Glass Cannon Network")); + assert_eq!(shows, [("2073588".into(), "Get in the Trunk".into()), ("2073636".into(), "Shadowdark".into())]); + + // An answer that stops naming the shows is an error, never "this creator has none". + assert!(parse_patreon_shows(br#"{"data":{"attributes":{"name":"X"}}}"#).is_err()); + } + #[test] fn an_item_may_carry_several_enclosures() { // The rss crate keeps only one per item -- the last -- so these come from the XML. diff --git a/src/main.rs b/src/main.rs index 3d4a7e3..e284cd2 100644 --- a/src/main.rs +++ b/src/main.rs @@ -517,8 +517,9 @@ async fn add( keywords: Vec<String>, ) -> Result<()> { let mut cfg = (*ctx.cfg()).clone(); + let url = &feed::expand_input(url); // Includes feeds derived from an OPML, or the same show could be added twice. - if let Some(existing) = subscriptions(ctx)?.iter().find(|s| s.cfg.url == url) { + if let Some(existing) = subscriptions(ctx)?.iter().find(|s| feed::same_feed(&s.cfg.url, url)) { anyhow::bail!("already subscribed as {:?}", existing.id); } let id = add_one(ctx, &mut cfg, url, folder, keywords).await?; @@ -569,10 +570,17 @@ pub async fn add_one( // Slugs must be unique across derived feeds too, or a new feed can collide with one // an OPML already introduced. - let taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)? + let mut taken: std::collections::BTreeMap<String, config::Feed> = subscriptions(ctx)? .into_iter() .map(|s| (s.id, s.cfg)) .collect(); + // A removed feed keeps its rows, so its id is only free again for the same feed: re-adding + // it gets its history back, and a different feed does not inherit someone else's. + for (id, other) in ctx.db.feed_urls()? { + if !feed::same_feed(&other, url) { + taken.entry(id).or_insert_with(|| probe.clone()); + } + } let id = config::unique_slug(&title, &taken); cfg.feeds.insert(id.clone(), probe); Ok(id) @@ -807,7 +815,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool) -> Result<()> { ctx.out.emit(Event::FeedSkip { feed: id.clone(), reason: format!( - "OPML: {total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing", + "{total} feed(s) listed, {} added, {removed} unsubscribed, {kept} kept without a listing", added.len() ), }); @@ -963,7 +971,7 @@ struct Scan { enum Outcome { NotModified, Feed(Scan), - /// The URL served an OPML document, so it is a subscription list rather than a feed. + /// The URL is a list of feeds rather than a feed: an OPML, or a Patreon creator's shows. Opml { added: Vec<String>, removed: usize, kept: usize, total: usize }, } @@ -973,6 +981,31 @@ async fn scan_one( feed_cfg: &config::Feed, state: &db::HttpState, ) -> Result<Outcome> { + // A Patreon creator with more than one show is a list of feeds, like an OPML. + if feed::is_patreon_creator(&feed_cfg.url) { + match feed::patreon_shows(&ctx.client, &feed_cfg.url).await { + Ok((name, shows)) if shows.len() > 1 => { + ctx.db.touch_feed(id, &feed_cfg.url)?; + if let Some(name) = name { + ctx.db.set_title(id, &name)?; + } + // Read as one feed before it was split, it listed every show's items in one + // heap. The items go; its files and read state move to each show as the show + // lists them (`Db::adopt`), so no show comes up empty for want of a URL. + ctx.db.clear_entries(id)?; + return sync_group(ctx, id, feed_cfg, &shows).await; + } + Ok(_) => {} // One show: the creator's feed is that show. + // Already split: keep the shows it has rather than read the creator as one heap. + Err(e) if ctx.db.managed_feeds()?.iter().any(|m| m.group_id == id) => return Err(e), + Err(e) => tracing::warn!( + feed = id, + error = %format!("{e:#}"), + "could not list the Patreon shows; reading it as one feed" + ), + } + } + let mut fetched = feed::fetch( &ctx.client, feed_cfg, @@ -1018,19 +1051,38 @@ async fn scan_one( )?; let policy = policy_for(ctx, id, feed_cfg)?; + if let Some(parent) = &feed_cfg.group { + let listed: Vec<(&str, &str)> = parsed + .entries + .iter() + .flat_map(|e| e.enclosures.iter().map(move |x| (e.guid.as_str(), x.url.as_str()))) + .collect(); + ctx.db.adopt(parent, id, &listed)?; + } + // Verdicts are recorded in `state`, so the download queue below is just "everything still + // pending". A filter's verdict is looked at again on every scan, though: made once, at + // discovery, it outlived the setting behind it, and allowing explicit items afterwards + // changed nothing however often the feed was scanned. + let skipped = ctx.db.skipped_by_filter(id)?; let mut scan = Scan::default(); for entry in &parsed.entries { if ctx.db.record_entry(id, entry)? { scan.new_entries += 1; } for enc in &entry.enclosures { - if !ctx.db.record_enclosure(id, &entry.guid, enc)? { - continue; // Seen before: downloaded, skipped or deliberately reaped. - } - // Filters run once, at discovery, and are recorded in `state`. The download - // queue below is then just "everything still pending". - if let Some(reason) = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc) { - ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?; + let was = if ctx.db.record_enclosure(id, &entry.guid, enc)? { + None + } else if let Some(reason) = skipped.get(&enc.url) { + Some(reason.as_str()) + } else { + continue; // Settled: queued, downloaded, reaped, or another feed's file. + }; + let now = reject(&ctx.cfg(), feed_cfg, &policy, entry, enc); + if now != was { + match now { + Some(reason) => ctx.db.mark_enclosure(&enc.url, "skipped", Some(reason))?, + None => ctx.db.mark_enclosure(&enc.url, "pending", None)?, + } } } } @@ -1114,12 +1166,6 @@ async fn scan_one( Ok(Outcome::Feed(scan)) } -/// Brings the feed list in step with a subscribed OPML. -/// -/// New entries are added under the OPML's group and folder. An entry that has gone from -/// the OPML is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it -/// is kept and flagged, because dropping it would orphan files on disk with nothing in -/// the UI to explain them. async fn sync_opml( ctx: &Arc<Ctx>, parent_id: &str, @@ -1130,25 +1176,44 @@ async fn sync_opml( if let Some(title) = feed::opml_title(bytes) { ctx.db.set_title(parent_id, &title)?; } + sync_group(ctx, parent_id, parent, &listed).await +} +/// Brings the feed list in step with a list of feeds: a subscribed OPML, or a Patreon +/// creator's shows. +/// +/// New entries are added under the list's group and folder. An entry that has gone from +/// the list is unsubscribed *only if nothing was ever downloaded for it* -- otherwise it +/// is kept and flagged, because dropping it would orphan files on disk with nothing in +/// the UI to explain them. +async fn sync_group( + ctx: &Arc<Ctx>, + parent_id: &str, + parent: &config::Feed, + listed: &[(String, String)], +) -> Result<Outcome> { let cfg = ctx.cfg(); let existing = ctx.db.managed_feeds()?; let mut added = vec![]; - for (title, url) in &listed { + for (title, url) in listed { // Already known, whether derived or promoted into the config. if let Some(m) = existing.iter().find(|m| &m.url == url) { ctx.db.upsert_managed(&m.id, url, title, parent_id)?; continue; } - if cfg.feeds.values().any(|f| &f.url == url) { + // A Patreon show you added by hand may be spelled differently from the one listed. + if cfg.feeds.values().any(|f| feed::same_feed(&f.url, url)) { continue; } + // A removed feed keeps its rows, so its id is only free again for the same feed. + let known = ctx.db.feed_urls()?; let taken: std::collections::BTreeMap<String, config::Feed> = cfg .feeds .keys() .chain(existing.iter().map(|m| &m.id)) .chain(added.iter()) + .chain(known.iter().filter(|(_, u)| !feed::same_feed(u, url)).map(|(id, _)| id)) .map(|id| (id.clone(), parent.clone())) .collect(); let id = config::unique_slug(title, &taken); @@ -1255,7 +1320,7 @@ pub struct Policy { fn policy_for(ctx: &Ctx, id: &str, feed_cfg: &config::Feed) -> Result<Policy> { let global = ctx.cfg().general.max_new_per_check; - Ok(merge_policy(&ctx.db.subscribers(id)?, feed_cfg, global)) + Ok(merge_policy(&ctx.db.subscribers(id, feed_cfg.group.as_deref())?, feed_cfg, global)) } fn merge_policy(subs: &[db::Sub], feed_cfg: &config::Feed, global: usize) -> Policy { diff --git a/src/web.rs b/src/web.rs index 81b8bc3..d56dfa2 100644 --- a/src/web.rs +++ b/src/web.rs @@ -495,6 +495,9 @@ async fn feeds( let mut out = Vec::with_capacity(mine.len()); for sub in &subs { let (id, feed) = (&sub.id, &sub.cfg); + // In a group, what you have not set on the feed comes from your settings on the group, + // the same fallback the scanner uses (`Db::subscribers`). + let up = feed.group.as_deref().and_then(|g| mine.get(g)); let Some(mine) = mine.get(id) else { continue }; let s = state.ctx.db.feed_summary(id)?; let st = state.ctx.db.http_state(id)?; @@ -504,11 +507,22 @@ async fn feeds( title: s.title, image: s.image, folder: feed.folder.clone(), - keywords: mine.keywords.clone().unwrap_or_else(|| feed.keywords.clone()), - allow_explicit: mine.allow_explicit.unwrap_or(feed.allow_explicit), - auto_download: mine.auto_download.unwrap_or(feed.auto_download), + keywords: mine + .keywords + .clone() + .or_else(|| up.and_then(|u| u.keywords.clone())) + .unwrap_or_else(|| feed.keywords.clone()), + allow_explicit: mine + .allow_explicit + .or(up.and_then(|u| u.allow_explicit)) + .unwrap_or(feed.allow_explicit), + auto_download: mine + .auto_download + .or(up.and_then(|u| u.auto_download)) + .unwrap_or(feed.auto_download), max_new_per_check: mine .max_new_per_check + .or(up.and_then(|u| u.max_new_per_check)) .map(|n| n as usize) .or(feed.max_new_per_check), group: feed.group.clone(), @@ -893,6 +907,19 @@ struct NewFeed { folder: Option<String>, #[serde(default)] keywords: Vec<String>, + #[serde(default)] + allow_explicit: bool, +} + +/// The Add feed dialog's explicit box. Like everything on a feed's own dialog it is yours, so it +/// goes on your subscription, and before the first scan, which would otherwise skip every +/// explicit item. +fn explicit_on_add(state: &WebState, user_id: i64, feed_id: &str, allow: bool) -> Result<(), ApiError> { + if allow { + let sub = crate::db::Sub { feed_id: feed_id.to_owned(), allow_explicit: Some(true), ..Default::default() }; + state.ctx.db.set_subscription(user_id, &sub)?; + } + Ok(()) } async fn add_feed( @@ -901,23 +928,28 @@ async fn add_feed( Json(body): Json<NewFeed>, ) -> Result<Json<serde_json::Value>, ApiError> { let mut cfg = (*state.ctx.cfg()).clone(); + let url = crate::feed::expand_input(&body.url); // Someone else may already have it. Then adding costs nothing: no second fetch, no // second copy on disk, just another name against the same feed. if let Some(existing) = crate::subscriptions(&state.ctx)? .into_iter() - .find(|s| s.cfg.url == body.url) + .find(|s| crate::feed::same_feed(&s.cfg.url, &url)) { let already = state.ctx.db.subscription(user.id, &existing.id)?.is_some(); state.ctx.db.subscribe(user.id, &existing.id)?; + if !already { + explicit_on_add(&state, user.id, &existing.id, body.allow_explicit)?; + } scan_soon(&state, Some(existing.id.clone())).await; return Ok(Json( serde_json::json!({ "id": existing.id, "existing": already }), )); } - let id = crate::add_one(&state.ctx, &mut cfg, &body.url, body.folder, body.keywords).await?; + let id = crate::add_one(&state.ctx, &mut cfg, &url, body.folder, body.keywords).await?; cfg.save(&state.config_path)?; state.ctx.reload_cfg(&state.config_path)?; state.ctx.db.subscribe(user.id, &id)?; + explicit_on_add(&state, user.id, &id, body.allow_explicit)?; scan_soon(&state, Some(id.clone())).await; Ok(Json(serde_json::json!({ "id": id, "existing": false }))) } diff --git a/web/index.html b/web/index.html index 81182bb..9a6692f 100644 --- a/web/index.html +++ b/web/index.html @@ -793,7 +793,7 @@ function renderFeeds(){ (mine.length?`${mine.length} feed${mine.length===1?'':'s'} · ${saved} saved` :`${eps} items · ${saved} saved`)+ `</small></div>`+ - (f.orphaned?'<span class="tag" title="No longer listed in the OPML, kept because it has downloads">gone</span>':'')+ + (f.orphaned?'<span class="tag" title="No longer listed, kept because it has downloads">gone</span>':'')+ `<span class="badge${unread?'':' zero'}" title="${unread} unread">${unread>999?'999+':unread}</span>`; el.onclick=()=>{ selectFeed(f.id); nav(false); }; if(kids) $('.chev',el).onclick=ev=>{ ev.stopPropagation(); toggleGroup(f.id); }; @@ -896,6 +896,9 @@ function sortBy(col){ /// An OPML subscription's page lists the feeds inside it rather than items, but keeps /// every action a normal feed has -- it is still an ordinary feed entry underneath. +// A Patreon creator split into its shows is drawn like an OPML, and named for what it is. +const isPatreon=f=>/patreon\.com\//.test(f&&f.url||''); + function renderGroup(f,kids){ const unread=kids.reduce((n,c)=>n+c.unread,0); const saved=kids.reduce((n,c)=>n+c.downloaded,0); @@ -907,12 +910,12 @@ function renderGroup(f,kids){ ${artHTML(f.image,f.title||f.id)} <div class="meta"> <h2>${esc(f.title||f.id)}</h2> - <div class="sub stat">OPML subscription · ${kids.length} feed${kids.length===1?'':'s'} + <div class="sub stat">${isPatreon(f)?'Patreon creator':'OPML subscription'} · ${kids.length} feed${kids.length===1?'':'s'} · ${unread} unread · ${saved} downloaded · checked ${ago(f.last_checked)} · every ${everyText(f.every_mins)}</div> ${f.last_error?`<div class="sub" style="color:var(--bad)">${esc(f.last_error)}</div>`:''} ${gone?`<div class="sub" style="color:var(--warn)">${gone} feed${gone===1?' is':'s are'} no longer - listed in this OPML but kept because ${gone===1?'it has':'they have'} downloads.</div>`:''} + listed but kept because ${gone===1?'it has':'they have'} downloads.</div>`:''} </div> <div class="acts"> <button class="btn ico primary" data-a="scan" title="Re-read the OPML now" aria-label="Re-read the OPML now">${ICON.scan}</button> @@ -1477,10 +1480,12 @@ $('#modal').onclick=e=>{ if(e.target.id==='modal') closeModal(); }; $('#addFeed').onclick=()=>{ openModal(`<h3>Add a feed</h3> - <div class="field"><label>Feed URL</label><input type="text" id="nurl" placeholder="https://example.com/rss"></div> + <div class="field"><label>Feed URL</label><input type="text" id="nurl" placeholder="https://example.com/rss"> + <span class="hint">A Patreon token on its own adds every show from that creator.</span></div> <div class="field"><label>Folder (optional)</label><input type="text" id="nfolder" placeholder="Defaults to the feed title"></div> <div class="field"><label>Keywords (optional, comma separated)</label> <input type="text" id="nkw"><span class="hint">Only items matching a keyword are downloaded.</span></div> + <label class="check"><input type="checkbox" id="nexp"> Allow items marked explicit</label> <div class="field"><label>Popular on this server</label> <div class="childlist" id="popular"><p class="hint">Loading…</p></div></div> <div class="cardacts"><button class="btn ico" onclick="closeModal()" title="Cancel" aria-label="Cancel">${ICON.close}</button> @@ -1492,7 +1497,7 @@ $('#addFeed').onclick=()=>{ $('#nsave').disabled=true; $('#nsave').title='Adding…'; try{ const r=await api('/api/feeds',{method:'POST',body:JSON.stringify({ - url, folder:$('#nfolder').value.trim()||null, + url, folder:$('#nfolder').value.trim()||null, allow_explicit:$('#nexp').checked, keywords:$('#nkw').value.split(',').map(s=>s.trim()).filter(Boolean)})}); closeModal(); toast(r.existing?`Already subscribed as ${r.id}`:`Added ${r.id}`); await loadFeeds(true); selectFeed(r.id); @@ -1704,10 +1709,10 @@ async function usersModal(){ function settingsModal(f){ const isGroup = S.feeds.some(c=>c.group===f.id); openModal(`<h3>${esc(f.title||f.id)}</h3> - ${isGroup?`<p class="hint" style="margin:-6px 0 12px">This is an OPML subscription. These + ${isGroup?`<p class="hint" style="margin:-6px 0 12px">This is ${isPatreon(f)?'a Patreon creator':'an OPML subscription'}. These settings apply to it and are inherited by every feed inside it.</p>`:''} - ${f.managed?`<p class="hint" style="margin:-6px 0 12px">This feed comes from an OPML - subscription and follows its settings. Saving anything here gives it its own entry in + ${f.managed?`<p class="hint" style="margin:-6px 0 12px">This feed comes from + ${isPatreon(S.feeds.find(p=>p.id===f.group))?'a Patreon creator':'an OPML subscription'} and follows its settings. Saving anything here gives it its own entry in config.toml, and it stops following the subscription's settings.</p>`:''} <p class="hint" style="margin:-4px 0 10px">These are <b>your</b> settings for this feed. Everyone else keeps their own.</p>