A moment's DNS failure keeps a listed feed failing for a day: 205 feeds have never been read #143

New Issue

rays · 2026-10-05T16:58:22-07:00

rays commented

2026-10-05 16:58:22 -07:00

Found 2026-10-05 from the question of why 208 Directory feeds have no category. All 208 are in the catalogue, have never stored an item and are failing; 205 of them with "dns error: Temporary failure in name resolution", every one starting 2026-10-05 03:23-03:24 UTC, last checked then and not since. They are working podcasts (feeds.npr.org, feeds.megaphone.fm, feeds.acast.com, feeds.captivate.fm, rss.libsyn.com), which resolve now from Tower and from inside iPX. Loki has no feed_error line for any of them, perhaps because the same outage kept Alloy from shipping. A category is picked from a feed's items (#117), so a feed never read has none.

A feed nobody subscribes to is checked once a day, and a failing one backs off up to a day (#99), so one failed lookup in a burst leaves it failing, unread and uncategorised until the next day, and the same can happen again then. A temporary lookup failure (EAI_AGAIN) says nothing about the feed: it is worth trying again in minutes, not a day, and perhaps not worth recording as the feed's error at all.

Found 2026-10-05 from the question of why 208 Directory feeds have no category. All 208 are in the catalogue, have never stored an item and are failing; 205 of them with "dns error: Temporary failure in name resolution", every one starting 2026-10-05 03:23-03:24 UTC, last checked then and not since. They are working podcasts (feeds.npr.org, feeds.megaphone.fm, feeds.acast.com, feeds.captivate.fm, rss.libsyn.com), which resolve now from Tower and from inside iPX. Loki has no feed_error line for any of them, perhaps because the same outage kept Alloy from shipping. A category is picked from a feed's items (#117), so a feed never read has none. A feed nobody subscribes to is checked once a day, and a failing one backs off up to a day (#99), so one failed lookup in a burst leaves it failing, unread and uncategorised until the next day, and the same can happen again then. A temporary lookup failure (EAI_AGAIN) says nothing about the feed: it is worth trying again in minutes, not a day, and perhaps not worth recording as the feed's error at all.

rays added the bug label 2026-10-05 16:58:22 -07:00

rays commented

2026-10-05 17:04:55 -07:00

More, 2026-10-06:

The outage was Tower's, not the feeds'. iPX's log in Loki goes quiet at about 03:35 on 10-05 (110 lines in the 03:00 hour against about 900 an hour before, none from 03:40), while other containers kept shipping, and comes back at about 11:30. Tower rebooted at 11:56 UTC (host uptime), and 74 containers, the DNS server blocky among them, started at 11:57-11:59.
The 1,372 feeds nobody subscribes to are read in batches: their last checks fall 455 in one hour, 260, 249, 207 and 182 in others. Each is read once a day from its last read, so a batch stays a batch, and a blip that lands on one fails hundreds at once, as at 03:23-03:24.
explain_failure takes every 'dns error' for 'This address no longer resolves; the site is gone.', so these NPR, Megaphone and Acast feeds are now labelled gone on their pages, and since #142 on their Directory pages too. A temporary failure (EAI_AGAIN, 'Temporary failure in name resolution') is not a missing name (EAI_NONAME, 'Name or service not known').

More, 2026-10-06: - The outage was Tower's, not the feeds'. iPX's log in Loki goes quiet at about 03:35 on 10-05 (110 lines in the 03:00 hour against about 900 an hour before, none from 03:40), while other containers kept shipping, and comes back at about 11:30. Tower rebooted at 11:56 UTC (host uptime), and 74 containers, the DNS server blocky among them, started at 11:57-11:59. - The 1,372 feeds nobody subscribes to are read in batches: their last checks fall 455 in one hour, 260, 249, 207 and 182 in others. Each is read once a day from its last read, so a batch stays a batch, and a blip that lands on one fails hundreds at once, as at 03:23-03:24. - explain_failure takes every 'dns error' for 'This address no longer resolves; the site is gone.', so these NPR, Megaphone and Acast feeds are now labelled gone on their pages, and since #142 on their Directory pages too. A temporary failure (EAI_AGAIN, 'Temporary failure in name resolution') is not a missing name (EAI_NONAME, 'Name or service not known').

rays referenced this issue from a commit

2026-10-05 17:18:56 -07:00

Try a feed again in minutes when its host could not be looked up for the moment (#143)

rays commented

2026-10-05 17:21:34 -07:00

Fixed in 4f4f2e8: a temporary lookup failure (EAI_AGAIN) is not recorded on the feed; the daemon tries it again in ten minutes (Ctx::retry, honoured by the due check and the sleep until the next scan). A name that does not exist is still the feed's error, worded 'may be gone', and the Directory page names a cause only after a day of failing, as the feed's own page does. A forced check of the 208 read 205 of them, which now have items, descriptions and their own categories; 3 are really broken. Deployed 2026-10-06 (0.10.3-dev).

Fixed in 4f4f2e8: a temporary lookup failure (EAI_AGAIN) is not recorded on the feed; the daemon tries it again in ten minutes (Ctx::retry, honoured by the due check and the sleep until the next scan). A name that does not exist is still the feed's error, worded 'may be gone', and the Directory page names a cause only after a day of failing, as the feed's own page does. A forced check of the 208 read 205 of them, which now have items, descriptions and their own categories; 3 are really broken. Deployed 2026-10-06 (0.10.3-dev).

rays closed this issue

2026-10-05 17:21:35 -07:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: rays/ipx#143