mimalloc for ipx and the directory, and metrics and traces for memory (#224)

ipx's resident memory climbed from about 40 MB to 500-850 MB each night while scans ran five
times as often, and stayed there until a restart; the directory sat at 1.25 GiB idle. glibc's
allocator keeps what each thread's arena freed. mimalloc gives freed pages back, so resident
memory now follows what is in use. jemalloc was the first choice for its in-use figures, but its
upstream was archived in 2025.

Declared once in the directory crate, which ipx builds in, so both binaries get it.

What was missing to see it: the directory served no /metrics (now on metrics_bind, a port apart
from the tunnel's), nothing counted the feed bytes fetched, and the fetch span had no body size.
The feed crate's spans never reached Tempo either: both trace exporters filtered to their own
crate, so ipx_feed's fetch and parse spans were dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-13 17:15:01 +00:00
parent e4b505564c
commit 45c9f9252b
15 changed files with 138 additions and 16 deletions

View File

@@ -295,6 +295,13 @@ table("Failing feeds in the directory", [loki(f'sum by (feed, error) (count_over
table("Snapshots by server", [loki(f'sum by (instance) (count_over_time({DIR} | message="snapshot" [$__range]))', instant=True)],
12, rename={"instance": "Token", "Value": "Snapshots"}, sort=[{"displayName": "Snapshots", "desc": True}])
y += 8
# The directory's own /metrics, on a port apart from the tunnel's (#224).
ts("Directory memory", [prom('directory_process_resident_bytes{job="ipx-directory"}', legend="resident")], 0, unit="bytes",
desc="mimalloc gives back what is freed, so this follows what is in use: a rise that never comes down is a leak.")
ts("Feed bytes fetched", [prom('rate(ipx_feed_bytes_total{job="ipx"}[$__rate_interval])', legend="ipx"),
prom('rate(directory_feed_bytes_total{job="ipx-directory"}[$__rate_interval])', ref="B", legend="directory")],
12, unit="Bps", desc="Feeds are read whole, so this is what scans and checks allocate; set it beside memory.")
y += 8
# ---- Traces: all three send them to Tempo (#213)
row("Traces")