YouTube channels still miss most days after the retry in #382 #386

New Issue

rays · 2026-11-06T15:30:27-08:00

rays commented

2026-11-06 15:30:27 -08:00

Since the 2026-11-04 20:47 deploy of 245fa79 (#382), YouTube feeds fail as often as before: 404 and 500 feed_error lines run 16 to 22 an hour-cluster, at the daily 23:xx and 00:xx UTC scans, the same on 2026-10-31 as on 2026-11-05 and 11-06 (about 38 a day, 77 in the 51 hours since the deploy).

The retry is running: in scan trace 6d66e6aacab92f0296444f92670ee26f (2026-11-06 23:21) each failing fetch of youtube.com/feeds/videos.xml takes 2.1 to 2.4 s, the two one-second waits, and still ends 404, for android-authority-2, android-developers, apple and architectural-digest-2 one after another. A retry a second later on the same burst does not get a different answer; whatever YouTube keys the 404 on (likely the burst of channel requests from one address) lasts longer than that.

Seen with:

$Q logs '{container="iPX"} |= "\"ev\":\"feed_error\"" | json | error_type=~"404|500" | line_format "{{.timestamp}} {{.feed}} {{.error_type}}"' 7d | awk '{print substr($1,1,13)}' | sort | uniq -c

Feeds affected in 7 days include love-nature, nature-on-pbs, spacex, yahoo-finance, linus-tech-tips, marie-forleo, the-verge-2, sri-lanka-cricket (6 to 7 failures each).

Since the 2026-11-04 20:47 deploy of 245fa79 (#382), YouTube feeds fail as often as before: 404 and 500 feed_error lines run 16 to 22 an hour-cluster, at the daily 23:xx and 00:xx UTC scans, the same on 2026-10-31 as on 2026-11-05 and 11-06 (about 38 a day, 77 in the 51 hours since the deploy). The retry is running: in scan trace `6d66e6aacab92f0296444f92670ee26f` (2026-11-06 23:21) each failing `fetch` of youtube.com/feeds/videos.xml takes 2.1 to 2.4 s, the two one-second waits, and still ends 404, for android-authority-2, android-developers, apple and architectural-digest-2 one after another. A retry a second later on the same burst does not get a different answer; whatever YouTube keys the 404 on (likely the burst of channel requests from one address) lasts longer than that. Seen with: ``` $Q logs '{container="iPX"} |= "\"ev\":\"feed_error\"" | json | error_type=~"404|500" | line_format "{{.timestamp}} {{.feed}} {{.error_type}}"' 7d | awk '{print substr($1,1,13)}' | sort | uniq -c ``` Feeds affected in 7 days include love-nature, nature-on-pbs, spacex, yahoo-finance, linus-tech-tips, marie-forleo, the-verge-2, sri-lanka-cricket (6 to 7 failures each).

rays added the bug label 2026-11-06 15:30:27 -08:00

rays referenced this issue from a commit

2026-11-06 15:44:57 -08:00

A failing YouTube feed is asked again hourly for a day (#386)

rays commented

2026-11-06 15:51:20 -08:00

The cause was not the retry's length alone: YouTube fails about 70% of feed reads from Tower, in streaks of hours, for ?user= and ?channel_id= addresses alike (142 and 68 failures in 7 days, across 26 and 13 feeds). All 39 are catalogue feeds nobody subscribes to, read once a day, so each missed about 5 days in 7. Fixed in bbcf690: in the first day of a run of failures, a YouTube feed waits at most an hour, not a day; a channel gone for good costs 24 requests, then backs off as before. Deployed in ipx 1.5.2-dev. Watch: $Q metric 'sum by (feed) (count_over_time({container="iPX"} |= "\"ev\":\"feed_error\"" |= "youtube" [$range]))' against these feeds' last_checked over the next days.

The cause was not the retry's length alone: YouTube fails about 70% of feed reads from Tower, in streaks of hours, for `?user=` and `?channel_id=` addresses alike (142 and 68 failures in 7 days, across 26 and 13 feeds). All 39 are catalogue feeds nobody subscribes to, read once a day, so each missed about 5 days in 7. Fixed in bbcf690: in the first day of a run of failures, a YouTube feed waits at most an hour, not a day; a channel gone for good costs 24 requests, then backs off as before. Deployed in ipx 1.5.2-dev. Watch: `$Q metric 'sum by (feed) (count_over_time({container="iPX"} |= "\"ev\":\"feed_error\"" |= "youtube" [$range]))'` against these feeds' last_checked over the next days.

rays closed this issue

2026-11-06 15:51:21 -08:00

rays reopened this issue

2026-11-06 15:54:32 -08:00

rays referenced this issue from a commit

2026-11-06 15:54:36 -08:00

YouTube feeds are read one at a time, two seconds apart (#386)

rays commented

2026-11-06 15:59:23 -08:00

Also e5388b4: YouTube feed requests take a turn, one at a time per process and two seconds apart, since a scan sent six at once and neighbouring channels failed together; the retries take their turn in place of the one-second sleep. Each wait is a youtube_turn span under fetch. In ipx and ipx-directory, both deployed 2026-11-06 23:56.

Also e5388b4: YouTube feed requests take a turn, one at a time per process and two seconds apart, since a scan sent six at once and neighbouring channels failed together; the retries take their turn in place of the one-second sleep. Each wait is a `youtube_turn` span under `fetch`. In ipx and ipx-directory, both deployed 2026-11-06 23:56.

rays closed this issue

2026-11-06 15:59:24 -08:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: rays/ipx#386