Both run on the daemon's single worker, so every other feed's check and download waited behind them. #339: prefer_https asked each new artwork host whether it served the image on https, up to 20 s each, one after another, and the per-scan cache is keyed by the authority, which ports and user names multiply for one server. It now asks at most eight hosts a read; past that an http image stays http, for /api/art to fetch, as one that fails does. #337: a list (OPML or Patreon) may bring in 2,000 feeds; past that sync_group refuses it whole, so the feed shows an error and nothing changes. davewiner's, the largest seen, had 922. sync_group also read the whole feeds table and rebuilt an id map with a Feed cloned per entry for every new feed; it now reads the table once and checks ids against sets, and unique_slug takes a predicate rather than a map. same_feed skips parsing two URLs unless both are Patreon's, since only those can differ and match. sync_group and sync_opml lose the parent argument, which nothing used once promote_kept stopped taking it (#332). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -452,7 +452,11 @@ pub fn expand_input(input: &str) -> String {
|
||||
/// creator's name, by number, or with no creator at all -- and the token and show are what
|
||||
/// identify it.
|
||||
pub fn same_feed(a: &str, b: &str) -> bool {
|
||||
a == b || patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b))
|
||||
// Parsing both is the slow part, and only Patreon's addresses can differ and still match.
|
||||
a == b
|
||||
|| (a.contains("patreon.com")
|
||||
&& b.contains("patreon.com")
|
||||
&& patreon_parts(a).is_some_and(|p| Some(p) == patreon_parts(b)))
|
||||
}
|
||||
|
||||
/// A Patreon creator's name and shows, each show as (title, feed URL).
|
||||
@@ -666,6 +670,9 @@ pub async fn site_icon(client: &reqwest::Client, site: &str) -> Option<String> {
|
||||
is_image(client, ico.as_str()).await.then(|| ico.into())
|
||||
}
|
||||
|
||||
/// How many hosts one scan of a feed asks whether its artwork is on https.
|
||||
const MAX_ART_HOSTS: usize = 8;
|
||||
|
||||
/// Artwork's address on https when its host serves it there, else as it was. The page is https
|
||||
/// and must not load http; the host is asked once per `known` (one feed's read), and an http
|
||||
/// address it does not serve on https stays, for /api/art to fetch (#90). Four of the five
|
||||
@@ -681,6 +688,10 @@ pub async fn prefer_https(
|
||||
let secure = format!("https://{rest}");
|
||||
let ok = match known.get(&host) {
|
||||
Some(ok) => *ok,
|
||||
// Each new host is a probe of up to 20 s, one after another on the scan, and the feed's
|
||||
// author decides how many hosts there are: ports and user names make as many as they like
|
||||
// of one server (#339). Real feeds put their artwork on a few.
|
||||
None if known.len() >= MAX_ART_HOSTS => false,
|
||||
None => {
|
||||
let ok = is_image(client, &secure).await;
|
||||
known.insert(host, ok);
|
||||
@@ -1838,6 +1849,16 @@ Some **bold** and [a link](https://x.example/).</description></item>
|
||||
assert_eq!((p1.season, p1.episode), (Some(8), Some(1)));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn artwork_on_more_hosts_than_the_limit_is_not_asked_about() {
|
||||
let mut known: std::collections::HashMap<String, bool> =
|
||||
(0..MAX_ART_HOSTS).map(|n| (format!("h{n}:80"), false)).collect();
|
||||
// No server answers here: were it asked, this would wait on a connection, not return.
|
||||
let url = "http://192.0.2.1:9/a.png";
|
||||
assert_eq!(prefer_https(&reqwest::Client::new(), url, &mut known).await, url);
|
||||
assert_eq!(known.len(), MAX_ART_HOSTS);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deeply_nested_opml_is_refused_before_it_is_parsed() {
|
||||
let nest = |n: usize| {
|
||||
|
||||
Reference in New Issue
Block a user