A failing YouTube feed is asked again hourly for a day (#386)

#382's retries run (each failing fetch takes the two seconds) and still
end 404: YouTube fails about 70% of feed reads from Tower, in streaks
longer than seconds, for ?user= and ?channel_id= addresses alike (142
and 68 failures in 7 days across 26 and 13 feeds). The 39 YouTube
channels are catalogue feeds nobody subscribes to, read once a day, so
each missed about 5 days in 7.

due_at now waits at most an hour for a YouTube feed in the first day of
a run of failures. A channel gone for good costs 24 requests, then backs
off as any feed does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 23:44:51 +00:00
parent d3b8c4cd01
commit bbcf69062e
4 changed files with 31 additions and 6 deletions

View File

@@ -25,7 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- A YouTube channel no longer misses most days' videos: YouTube answers some feed requests with a 404 or 500 it does not mean, and ipx now asks again, twice, a second apart, before calling the feed failing (#382).
- A YouTube channel no longer misses most days' videos: YouTube answers some feed requests with a 404 or 500 it does not mean, and ipx now asks again, twice, a second apart, before calling the feed failing (#382), and then every hour for a day where it would have waited a day, since YouTube's failures run for hours (#386).
- Syncing an OPML or Patreon list reads who subscribes to it once, not once for every feed it lists: a list of 922 was thousands of the same queries a sync (#377).
- A feed you already have, added again by the same address over http rather than https or the other way round, is the one you have, not a second subscription with every item twice. Changing a feed's address to another feed's over the other scheme is refused the same way (#362).
- A podcast that was once in an OPML and is now a feed of its own is no longer hidden from All Subscriptions when that OPML is, so its unread items are listed there as the feed list counts them (#358).

View File

@@ -171,7 +171,13 @@ pub(crate) async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope
pub fn due_at(cfg: &config::Config, feed: &config::Feed, state: &db::HttpState, subscribed: bool) -> Option<i64> {
let last = state.last_checked?;
let floor = if subscribed { 0 } else { 86_400 };
Some(last + due_after(cfg, feed, state.ttl_mins, state.error_since, state.same_since, last).max(floor) as i64)
let wait = due_after(cfg, feed, state.ttl_mins, state.error_since, state.same_since, last).max(floor);
// YouTube fails about 70% of reads from Tower, in streaks longer than fetch's retries; read
// once a day, its 39 catalogue channels missed 5 days in 7 (#386). Hourly for the first day of
// a run of failures finds a moment it answers, and a channel gone for good costs 24 requests.
let youtube_again =
state.error_since.is_some_and(|since| last - since < 86_400) && feed::is_youtube_feed(&feed.url);
Some(last + if youtube_again { wait.min(3600) } else { wait } as i64)
}
/// A per-feed schedule is an explicit instruction and wins outright. Without one, the
@@ -907,6 +913,20 @@ mod tests {
assert_eq!(at(Some(1000), false), Some(1000 + 86_400)); // listed, nobody subscribed
}
#[test]
fn a_failing_youtube_feed_is_asked_again_hourly_for_a_day() {
let cfg = config::Config::default();
let yt = config::Feed { url: "https://www.youtube.com/feeds/videos.xml?channel_id=UCx".into(), ..feed() };
let at = |f: &config::Feed, since: Option<i64>, last: i64| {
let state = db::HttpState { last_checked: Some(last), error_since: since, ..Default::default() };
due_at(&cfg, f, &state, false).map(|t| t - last)
};
assert_eq!(at(&yt, None, 1000), Some(86_400), "well, it waits its day like any listed feed");
assert_eq!(at(&yt, Some(1000), 1000 + 7200), Some(3600), "failing two hours: in an hour");
assert_eq!(at(&yt, Some(1000), 1000 + 90_000), Some(86_400), "failing over a day: back off");
assert_eq!(at(&feed(), Some(1000), 1000 + 7200), Some(86_400), "another host's failure is not hurried");
}
#[test]
fn a_feed_long_the_same_is_checked_less_and_a_schedule_set_wins() {
let (hour, day) = (3600u64, 86_400u64);