mimalloc for ipx and the directory, and metrics and traces for memory (#224)
ipx's resident memory climbed from about 40 MB to 500-850 MB each night while scans ran five times as often, and stayed there until a restart; the directory sat at 1.25 GiB idle. glibc's allocator keeps what each thread's arena freed. mimalloc gives freed pages back, so resident memory now follows what is in use. jemalloc was the first choice for its in-use figures, but its upstream was archived in 2025. Declared once in the directory crate, which ipx builds in, so both binaries get it. What was missing to see it: the directory served no /metrics (now on metrics_bind, a port apart from the tunnel's), nothing counted the feed bytes fetched, and the fetch span had no body size. The feed crate's spans never reached Tempo either: both trace exporters filtered to their own crate, so ipx_feed's fetch and parse spans were dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -61,13 +61,19 @@ pub enum Fetched {
|
||||
/// downloads, which are not, have no such limit.
|
||||
pub const FEED_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||
|
||||
/// Feeds fetched and the bytes of their bodies, since the process started, for /metrics to set
|
||||
/// beside memory (#224): a body is read whole, so a night of large feeds is a night of large
|
||||
/// allocations.
|
||||
pub static FETCHES: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
|
||||
pub static FETCHED_BYTES: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
|
||||
|
||||
/// Conditional GET, following redirects itself: `client` must follow none (`feed_client`), so
|
||||
/// each hop is seen. The second value is where the feed now is when every hop said so for good
|
||||
/// (301 or 308): a publisher that moved its feed, which the catalogue should follow rather
|
||||
/// than be redirected on every read. A temporary redirect (302, 307) moves nothing.
|
||||
/// `system-proxy` reads http_proxy/https_proxy.
|
||||
/// `auth` is a username and password, sent only to the feed's own host.
|
||||
#[tracing::instrument(skip_all, fields(url = %feed_url))]
|
||||
#[tracing::instrument(skip_all, fields(url = %feed_url, http.response.body.size))]
|
||||
pub async fn fetch(
|
||||
client: &reqwest::Client,
|
||||
feed_url: &str,
|
||||
@@ -106,6 +112,7 @@ pub async fn fetch(
|
||||
continue;
|
||||
}
|
||||
let moved = (permanent && url != start).then(|| url.to_string());
|
||||
FETCHES.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
if status == StatusCode::NOT_MODIFIED {
|
||||
return Ok((Fetched::NotModified, moved));
|
||||
}
|
||||
@@ -119,6 +126,8 @@ pub async fn fetch(
|
||||
let etag = header(ETAG);
|
||||
let last_modified = header(LAST_MODIFIED);
|
||||
let bytes = resp.bytes().await.context("reading body")?.to_vec();
|
||||
tracing::Span::current().record("http.response.body.size", bytes.len());
|
||||
FETCHED_BYTES.fetch_add(bytes.len() as u64, std::sync::atomic::Ordering::Relaxed);
|
||||
return Ok((Fetched::Body { bytes, etag, last_modified }, moved));
|
||||
}
|
||||
// Worded as reqwest worded it, which failure_kind reads as a redirect loop.
|
||||
@@ -364,6 +373,7 @@ fn parse_patreon_shows(json: &[u8]) -> Result<(Option<String>, Vec<(String, Stri
|
||||
}
|
||||
|
||||
/// RSS first, then Atom -- the same split the original made on `parsedFeed.version`.
|
||||
#[tracing::instrument(skip_all, fields(bytes = bytes.len()))]
|
||||
pub fn parse(bytes: &[u8]) -> Result<ParsedFeed> {
|
||||
match rss::Channel::read_from(bytes) {
|
||||
Ok(ch) => Ok(from_rss(ch, bytes)),
|
||||
|
||||
Reference in New Issue
Block a user