Bound the scan work one feed's author can ask for (#339 #337)

Both run on the daemon's single worker, so every other feed's check
and download waited behind them.

#339: prefer_https asked each new artwork host whether it served the
image on https, up to 20 s each, one after another, and the per-scan
cache is keyed by the authority, which ports and user names multiply
for one server. It now asks at most eight hosts a read; past that an
http image stays http, for /api/art to fetch, as one that fails does.

#337: a list (OPML or Patreon) may bring in 2,000 feeds; past that
sync_group refuses it whole, so the feed shows an error and nothing
changes. davewiner's, the largest seen, had 922. sync_group also read
the whole feeds table and rebuilt an id map with a Feed cloned per
entry for every new feed; it now reads the table once and checks ids
against sets, and unique_slug takes a predicate rather than a map.
same_feed skips parsing two URLs unless both are Patreon's, since only
those can differ and match.

sync_group and sync_opml lose the parent argument, which nothing used
once promote_kept stopped taking it (#332).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-23 14:18:32 +00:00
parent 3bfd8f848b
commit d35b8054b4
4 changed files with 71 additions and 29 deletions

View File

@@ -452,7 +452,11 @@ pub fn expand_input(input: &str) -> String {
/// creator's name, by number, or with no creator at all -- and the token and show are what
/// identify it.
pub fn same_feed(a: &str, b: &str) -> bool {
a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b))
// Parsing both is the slow part, and only Patreon's addresses can differ and still match.
a == b
|| (a.contains("patreon.com")
&& b.contains("patreon.com")
&& patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b)))
}
/// A Patreon creator's name and shows, each show as (title, feed URL).
@@ -666,6 +670,9 @@ pub async fn site_icon(client: &reqwest::Client, site: &str) -> Option<String> {
is_image(client, ico.as_str()).await.then(|| ico.into())
}
/// How many hosts one scan of a feed asks whether its artwork is on https.
const MAX_ART_HOSTS: usize = 8;
/// Artwork's address on https when its host serves it there, else as it was. The page is https
/// and must not load http; the host is asked once per `known` (one feed's read), and an http
/// address it does not serve on https stays, for /api/art to fetch (#90). Four of the five
@@ -681,6 +688,10 @@ pub async fn prefer_https(
let secure = format!("https://{rest}");
let ok = match known.get(&host) {
Some(ok) => *ok,
// Each new host is a probe of up to 20 s, one after another on the scan, and the feed's
// author decides how many hosts there are: ports and user names make as many as they like
// of one server (#339). Real feeds put their artwork on a few.
None if known.len() >= MAX_ART_HOSTS => false,
None => {
let ok = is_image(client, &secure).await;
known.insert(host, ok);
@@ -1838,6 +1849,16 @@ Some **bold** and [a link](https://x.example/).</description></item>
assert_eq!((p1.season, p1.episode), (Some(8), Some(1)));
}
#[tokio::test]
async fn artwork_on_more_hosts_than_the_limit_is_not_asked_about() {
let mut known: std::collections::HashMap<String, bool> =
(0..MAX_ART_HOSTS).map(|n| (format!("h{n}:80"), false)).collect();
// No server answers here: were it asked, this would wait on a connection, not return.
let url = "http://192.0.2.1:9/a.png";
assert_eq!(prefer_https(&reqwest::Client::new(), url, &mut known).await, url);
assert_eq!(known.len(), MAX_ART_HOSTS);
}
#[test]
fn deeply_nested_opml_is_refused_before_it_is_parsed() {
let nest = |n: usize| {