rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

Try a feed again in minutes when its host could not be looked up for the moment (#143)

Browse Source 
205 of the Directory's 208 feeds without a category had never been read: on 2026-10-05 at 03:24
UTC every one failed with "dns error: ... Temporary failure in name resolution" within a minute,
during an outage on Tower that also stopped iPX's log reaching Loki until Tower rebooted at 11:56.
They are working podcasts, NPR's, Megaphone's, Acast's, which resolve now. A feed nobody
subscribes to is read once a day and a failing one backs off up to a day (#99), so each sat
failing, unread and so uncategorised (#117) until the next day, and explain_failure called every
DNS error "This address no longer resolves; the site is gone.", on its page and since #142 on its
Directory page. A forced check of the 208 read 205; each has its items, its description and its
own category now.

A temporary lookup failure, EAI_AGAIN, is not recorded on the feed at all: no last_error, no
error_since to back off from or call it gone by. The daemon notes it in Ctx::retry and the feed
is due again in ten minutes, which the scan's due check and the sleep until the next scan both
honour; a restart forgets it, which only makes the retry sooner. A name that does not exist is
still the feed's error, explained as "may be gone", and the Directory page, like the feed's own,
names a cause only once a feed has been failing a day; before that its last check failed.

The page also drops "Nobody here subscribes yet", which said nothing worth a line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-06 00:18:42 +00:00
parent c1b73b3dbe
commit 4f4f2e8d95
6 changed files with 104 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
src/main.rs
View File

@@ -141,9 +141,38 @@ pub struct Ctx {
/// stops every feed scan, every HTTP download and every status command behind it.
/// A one-shot CLI run keeps them inline, or the process would exit mid-download.
pub detach_torrents: bool,
/// Feeds whose host could not be looked up for the moment, and when to try them again: a
/// failure that says nothing about the feed, kept off it and out of the database (#143).
/// Forgotten on a restart, which only makes the retry sooner.
pub retry: std::sync::Mutex<std::collections::HashMap<String, i64>>,
}
/// How soon a feed whose host could not be looked up is tried again (#143).
const RETRY_LOOKUP_SECS: i64 = 600;
impl Ctx {
/// When a feed is next due, taking a lookup retry into account: not before it.
fn due(&self, cfg: &config::Config, feed: &config::Feed, id: &str, state: &db::HttpState, subscribed: bool) -> Option<i64> {
let at = due_at(cfg, feed, state, subscribed);
match self.retry.lock().unwrap().get(id) {
Some(&retry) => Some(at.unwrap_or(0).max(retry)),
None => at,
}
}
/// A scan's failure, recorded on the feed, unless it was only a moment's failure to look its
/// host up: on 2026-10-05 a DNS outage on Tower failed 204 feeds in a minute with
/// "Temporary failure in name resolution", and each, read once a day and backed off as a
/// failing feed, sat a day failing, unread and labelled gone (#143). Tried again in minutes
/// instead, with nothing on the feed to say it failed.
async fn feed_failed(&self, id: &str, url: &str, msg: &str) -> Result<()> {
if feed::is_temporary_lookup_failure(msg) {
self.retry.lock().unwrap().insert(id.to_owned(), db::now() + RETRY_LOOKUP_SECS);
return Ok(());
}
self.db.set_feed_error(id, url, msg).await
}
pub fn cfg(&self) -> std::sync::Arc<config::Config> {
self.cfg.read().unwrap().clone()
}
@@ -275,6 +304,7 @@ async fn main() -> Result<()> {
torrent_slots: Arc::new(tokio::sync::Semaphore::new(2)),
config_path: config_path.clone(),
detach_torrents: is_daemon,
retry: Default::default(),
});
let result = match cli.command {
@@ -1093,7 +1123,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
state.last_modified = None;
}
if !force && let Some(at) = due_at(&cfg, &sub.cfg, &state, subscribed.contains_key(id)) {
if !force && let Some(at) = ctx.due(&cfg, &sub.cfg, id, &state, subscribed.contains_key(id)) {
if at > db::now() {
ctx.out.emit(Event::FeedSkip {
feed: id.clone(),
@@ -1165,7 +1195,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
// One bad feed must not end the scan.
let msg = format!("{e:#}");
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
ctx.feed_failed(id, &feed_cfg.url, &msg).await?;
}
}
}
@@ -1191,7 +1221,7 @@ async fn fetch(ctx: &Arc<Ctx>, only: Option<&str>, force: bool, scope: &[String]
Err(e) => {
let msg = format!("{e:#}");
ctx.out.emit(Event::FeedError { feed: id.clone(), msg: msg.clone() });
ctx.db.set_feed_error(id, &feed_cfg.url, &msg).await?;
ctx.feed_failed(id, &feed_cfg.url, &msg).await?;
}
}
}
@@ -1329,7 +1359,9 @@ async fn until_next_scan(ctx: &Ctx) -> std::time::Duration {
.iter()
// A feed with no row yet has never been checked: due now.
.map(|s| {
states.get(&s.id).and_then(|st| due_at(&cfg, &s.cfg, st, subscribed.contains_key(&s.id))).unwrap_or(0)
let never = db::HttpState::default();
let state = states.get(&s.id).unwrap_or(&never);
ctx.due(&cfg, &s.cfg, &s.id, state, subscribed.contains_key(&s.id)).unwrap_or(0)
})
.min(),
)
@@ -2283,9 +2315,27 @@ mod tests {
torrent_slots: Arc::new(tokio::sync::Semaphore::new(2)),
config_path: PathBuf::new(),
detach_torrents: false,
retry: Default::default(),
}
}
#[tokio::test]
async fn a_moments_lookup_failure_is_tried_again_soon_and_not_held_against_the_feed() {
let ctx = test_ctx(config::Config::default()).await;
let blip = "connecting: dns error: failed to lookup address information: Temporary failure in name resolution";
ctx.feed_failed("npr", "https://feeds.npr.org/x", blip).await.unwrap();
// Nothing on the feed: no error, no run of failures to back off from or call it gone by.
let s = ctx.db.feed_summary("npr").await.unwrap();
assert!(s.last_error.is_none() && s.error_since.is_none(), "{:?}", s.last_error);
// Due again in minutes, though a feed nobody subscribes to is otherwise read once a day.
let state = db::HttpState { last_checked: Some(db::now() - 2 * 86_400), ..Default::default() };
let due = ctx.due(&config::Config::default(), &feed(), "npr", &state, false).unwrap();
assert!((due - db::now() - RETRY_LOOKUP_SECS).abs() <= 2, "{due}");
// A real failure is the feed's.
ctx.feed_failed("gone", "https://x/feed", "HTTP 404 Not Found").await.unwrap();
assert_eq!(ctx.db.feed_summary("gone").await.unwrap().last_error.as_deref(), Some("HTTP 404 Not Found"));
}
#[tokio::test]
async fn a_reload_takes_the_catalogue_another_ipx_wrote() {
let with = |id: &str| {