Time out a hung feed, serve the precomposed touch icon, and store http artwork on https (#108, #109, #110)
#108: the HTTP client had no timeout, and scans handle feeds in order, so a hung server held every scan. Dreamwidth answered 504 after 60-67 s for a day and each scan took 70-80 s instead of 15. A feed fetch, and a Patreon creator's show list, now gets 30 s from connecting to the last byte (feed::FEED_TIMEOUT); the client gets a 10 s connect timeout, which bounds a download's start but not a long download. #109: iOS asks for /apple-touch-icon-precomposed.png first when the site is added to a home screen; it was a 404 and the only non-feed warning in the log. It serves the same icon. #110: the page is https and loads no http. Artwork on http came through /api/art (#90) even when its host serves https too. A scan now tries each http artwork host on https once per feed (feed::prefer_https) and stores the https address where the host answers with an image, rewriting that feed's stored items from the same host (Db::secure_images). 4 of the 5 hosts in production do; cdn.thesecretcabal.com presents another name's certificate and stays on /api/art. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
34
src/feed.rs
34
src/feed.rs
@@ -53,6 +53,12 @@ pub enum Fetched {
|
||||
},
|
||||
}
|
||||
|
||||
/// The longest a feed may take, connecting to the last byte: a scan handles feeds in order, and
|
||||
/// with no limit one hung server held every scan for as long as it did. Dreamwidth answered 504
|
||||
/// after 60-67 s for a day, and each scan took 70-80 s instead of 15 (#108). A feed is small;
|
||||
/// downloads, which are not, have no such limit.
|
||||
pub const FEED_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||
|
||||
/// Conditional GET. reqwest handles gzip and redirects; the original's hand-rolled
|
||||
/// CONNECT/socket.ssl proxy path is gone -- `system-proxy` reads http_proxy/https_proxy.
|
||||
#[tracing::instrument(skip_all, fields(url = %cfg.url))]
|
||||
@@ -62,7 +68,7 @@ pub async fn fetch(
|
||||
etag: Option<&str>,
|
||||
last_modified: Option<&str>,
|
||||
) -> Result<Fetched> {
|
||||
let mut req = client.get(&cfg.url);
|
||||
let mut req = client.get(&cfg.url).timeout(FEED_TIMEOUT);
|
||||
if let Some(tag) = etag {
|
||||
req = req.header(IF_NONE_MATCH, tag);
|
||||
}
|
||||
@@ -253,7 +259,7 @@ pub async fn patreon_shows(
|
||||
) -> Result<(Option<String>, Vec<(String, String)>)> {
|
||||
// The creator feed names its campaign by number in its self link, a few hundred bytes in.
|
||||
// The whole feed runs to megabytes and Patreon ignores Range, so read until it turns up.
|
||||
let mut resp = client.get(url).send().await.context("connecting")?;
|
||||
let mut resp = client.get(url).timeout(FEED_TIMEOUT).send().await.context("connecting")?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(anyhow!("Patreon refused the feed: HTTP {}", resp.status()));
|
||||
}
|
||||
@@ -451,6 +457,30 @@ pub async fn site_icon(client: &reqwest::Client, site: &str) -> Option<String> {
|
||||
is_image(client, ico.as_str()).await.then(|| ico.into())
|
||||
}
|
||||
|
||||
/// Artwork's address on https when its host serves it there, else as it was. The page is https
|
||||
/// and must not load http; the host is asked once per `known` (one feed's read), and an http
|
||||
/// address it does not serve on https stays, for /api/art to fetch (#90). Four of the five
|
||||
/// hosts the catalogue had on http served the same image on https; The Secret Cabal's CDN
|
||||
/// presents another name's certificate (#110).
|
||||
pub async fn prefer_https(
|
||||
client: &reqwest::Client,
|
||||
url: &str,
|
||||
known: &mut std::collections::HashMap<String, bool>,
|
||||
) -> String {
|
||||
let Some(rest) = url.strip_prefix("http://") else { return url.to_owned() };
|
||||
let host = rest.split('/').next().unwrap_or("").to_owned();
|
||||
let secure = format!("https://{rest}");
|
||||
let ok = match known.get(&host) {
|
||||
Some(ok) => *ok,
|
||||
None => {
|
||||
let ok = is_image(client, &secure).await;
|
||||
known.insert(host, ok);
|
||||
ok
|
||||
}
|
||||
};
|
||||
if ok { secure } else { url.to_owned() }
|
||||
}
|
||||
|
||||
/// Whether `url` answers with an image. A site with no favicon often answers 200 with its home
|
||||
/// page, which is not an icon.
|
||||
pub async fn is_image(client: &reqwest::Client, url: &str) -> bool {
|
||||
|
||||
Reference in New Issue
Block a user