Time out a hung feed, serve the precomposed touch icon, and store http artwork on https (#108, #109, #110)

#108: the HTTP client had no timeout, and scans handle feeds in order, so a hung server held
every scan. Dreamwidth answered 504 after 60-67 s for a day and each scan took 70-80 s instead of
15. A feed fetch, and a Patreon creator's show list, now gets 30 s from connecting to the last
byte (feed::FEED_TIMEOUT); the client gets a 10 s connect timeout, which bounds a download's
start but not a long download.

#109: iOS asks for /apple-touch-icon-precomposed.png first when the site is added to a home
screen; it was a 404 and the only non-feed warning in the log. It serves the same icon.

#110: the page is https and loads no http. Artwork on http came through /api/art (#90) even
when its host serves https too. A scan now tries each http artwork host on https once per feed
(feed::prefer_https) and stores the https address where the host answers with an image,
rewriting that feed's stored items from the same host (Db::secure_images). 4 of the 5 hosts in
production do; cdn.thesecretcabal.com presents another name's certificate and stays on
/api/art.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-02 13:31:50 +00:00
parent e2593eaa50
commit 54d827f655
6 changed files with 96 additions and 3 deletions

View File

@@ -53,6 +53,12 @@ pub enum Fetched {
},
}
/// The longest a feed may take, connecting to the last byte: a scan handles feeds in order, and
/// with no limit one hung server held every scan for as long as it did. Dreamwidth answered 504
/// after 60-67 s for a day, and each scan took 70-80 s instead of 15 (#108). A feed is small;
/// downloads, which are not, have no such limit.
pub const FEED_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
/// Conditional GET. reqwest handles gzip and redirects; the original's hand-rolled
/// CONNECT/socket.ssl proxy path is gone -- `system-proxy` reads http_proxy/https_proxy.
#[tracing::instrument(skip_all, fields(url = %cfg.url))]
@@ -62,7 +68,7 @@ pub async fn fetch(
etag: Option<&str>,
last_modified: Option<&str>,
) -> Result<Fetched> {
let mut req = client.get(&cfg.url);
let mut req = client.get(&cfg.url).timeout(FEED_TIMEOUT);
if let Some(tag) = etag {
req = req.header(IF_NONE_MATCH, tag);
}
@@ -253,7 +259,7 @@ pub async fn patreon_shows(
) -> Result<(Option<String>, Vec<(String, String)>)> {
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
let mut resp = client.get(url).send().await.context("connecting")?;
let mut resp = client.get(url).timeout(FEED_TIMEOUT).send().await.context("connecting")?;
if !resp.status().is_success() {
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
}
@@ -451,6 +457,30 @@ pub async fn site_icon(client: &reqwest::Client, site: &str) -> Option<String> {
is_image(client, ico.as_str()).await.then(|| ico.into())
}
/// Artwork's address on https when its host serves it there, else as it was. The page is https
/// and must not load http; the host is asked once per `known` (one feed's read), and an http
/// address it does not serve on https stays, for /api/art to fetch (#90). Four of the five
/// hosts the catalogue had on http served the same image on https; The Secret Cabal's CDN
/// presents another name's certificate (#110).
pub async fn prefer_https(
client: &reqwest::Client,
url: &str,
known: &mut std::collections::HashMap<String, bool>,
) -> String {
let Some(rest) = url.strip_prefix("http://") else { return url.to_owned() };
let host = rest.split('/').next().unwrap_or("").to_owned();
let secure = format!("https://{rest}");
let ok = match known.get(&host) {
Some(ok) => *ok,
None => {
let ok = is_image(client, &secure).await;
known.insert(host, ok);
ok
}
};
if ok { secure } else { url.to_owned() }
}
/// Whether `url` answers with an image. A site with no favicon often answers 200 with its home
/// page, which is not an icon.
pub async fn is_image(client: &reqwest::Client, url: &str) -> bool {