Faster scans and directory rounds, and fetches named by feed (#227, #228, #229, #230)

From the slow traces of 2026-10-13: the directory checked a round's 25 feeds one after
another, so two libsyn feeds timing out at 30 s made a round 70 s; a forced scan looked up
62 site icons in turn, 22 s of 62; mjg59.dreamwidth.org takes the connection and never
answers, so it ran the full 30 s in every scan it was in; and since #225 strips queries,
every YouTube fetch span read the same address, outside any span naming its feed.

The directory now checks five at a time (SQLite stays on one connection, so its writes
queue). A forced scan looks up the icon in the fetch-ahead task, which already runs six
deep, and hands it to scan_one with the body. Both feed clients give up after 15 s of
silence (no successful fetch took over 8 s in a day of traces); FEED_TIMEOUT stays 30 s
for a feed still sending. The directory's client had no connect timeout; it has ipx's 10 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/directory/CHANGELOG.md

@@ -10,6 +10,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- The directory checks five feeds at a time, so a publisher that does not answer no longer holds up the rest of its round (#227).
+- A feed's server that takes 10 s to connect, or sends nothing for 15 s, is given up on (#230).
+
 ## [1.1.0] - 2026-10-13
 
 ### Added

crates/directory/Cargo.toml

@@ -12,6 +12,7 @@ ammonia.workspace = true
 anyhow.workspace = true
 axum.workspace = true
 clap.workspace = true
+futures-util.workspace = true
 ipx-feed.workspace = true
 opentelemetry.workspace = true
 opentelemetry-otlp.workspace = true

crates/directory/src/lib.rs

@@ -134,6 +134,8 @@ impl Directory {
     pub fn new(store: Store, tokens: Vec<Token>) -> Result<Shared, reqwest::Error> {
         // No redirects: ipx_feed::fetch follows them itself, so it sees each hop.
         let client = reqwest::Client::builder()
+            .connect_timeout(std::time::Duration::from_secs(10))
+            .read_timeout(ipx_feed::FEED_SILENCE)
             .redirect(reqwest::redirect::Policy::none())
             .user_agent(concat!("ipx-directory/", env!("CARGO_PKG_VERSION")))
             .build()?;
@@ -193,12 +195,17 @@ pub async fn refresher(dir: Shared) {
                     let round = tracing::info_span!("refresh", feeds = due.len());
                     tracing::Instrument::instrument(
                         async {
-                            for d in &due {
+                            // A few at a time: in turn, two publishers timing out held a round
+                            // for a minute and more (#227).
+                            futures_util::StreamExt::for_each_concurrent(futures_util::stream::iter(&due), 5, |d| {
                                 let check = tracing::info_span!("feed", feed = %d.id);
-                                if let Err(e) = tracing::Instrument::instrument(dir.refresh(d), check).await {
-                                    tracing::warn!(feed = %d.id, error = %format!("{e:#}"), "could not store a feed check");
+                                async {
+                                    if let Err(e) = tracing::Instrument::instrument(dir.refresh(d), check).await {
+                                        tracing::warn!(feed = %d.id, error = %format!("{e:#}"), "could not store a feed check");
+                                    }
                                 }
-                            }
+                            })
+                            .await;
                             // One line a round, for the dashboard's refresher panel (#210); failures have their own.
                             tracing::info!(checked = due.len(), "feeds checked");
                         },