Faster scans and directory rounds, and fetches named by feed (#227, #228, #229, #230)

From the slow traces of 2026-10-13: the directory checked a round's 25 feeds one after
another, so two libsyn feeds timing out at 30 s made a round 70 s; a forced scan looked up
62 site icons in turn, 22 s of 62; mjg59.dreamwidth.org takes the connection and never
answers, so it ran the full 30 s in every scan it was in; and since #225 strips queries,
every YouTube fetch span read the same address, outside any span naming its feed.

The directory now checks five at a time (SQLite stays on one connection, so its writes
queue). A forced scan looks up the icon in the fetch-ahead task, which already runs six
deep, and hands it to scan_one with the body. Both feed clients give up after 15 s of
silence (no successful fetch took over 8 s in a day of traces); FEED_TIMEOUT stays 30 s
for a feed still sending. The directory's client had no connect timeout; it has ipx's 10 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-14 13:27:19 +00:00
parent cece114d98
commit 8d7a30d1f9
7 changed files with 72 additions and 10 deletions

View File

@@ -61,6 +61,12 @@ pub enum Fetched {
/// downloads, which are not, have no such limit.
pub const FEED_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
/// The longest a feed's server may send nothing, for the clients that fetch feeds to set: one
/// that took the connection and never answered ran the whole 30 s, every scan it was in (#230).
/// No feed that answered took over 8 s in a day of traces; one that keeps sending, however
/// slowly, still has until FEED_TIMEOUT.
pub const FEED_SILENCE: std::time::Duration = std::time::Duration::from_secs(15);
/// Feeds fetched and the bytes of their bodies, since the process started, for /metrics to set
/// beside memory (#224): a body is read whole, so a night of large feeds is a night of large
/// allocations.