mimalloc for ipx and the directory, and metrics and traces for memory (#224)
ipx's resident memory climbed from about 40 MB to 500-850 MB each night while scans ran five times as often, and stayed there until a restart; the directory sat at 1.25 GiB idle. glibc's allocator keeps what each thread's arena freed. mimalloc gives freed pages back, so resident memory now follows what is in use. jemalloc was the first choice for its in-use figures, but its upstream was archived in 2025. Declared once in the directory crate, which ipx builds in, so both binaries get it. What was missing to see it: the directory served no /metrics (now on metrics_bind, a port apart from the tunnel's), nothing counted the feed bytes fetched, and the fetch span had no body size. The feed crate's spans never reached Tempo either: both trace exporters filtered to their own crate, so ipx_feed's fetch and parse spans were dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -295,6 +295,13 @@ table("Failing feeds in the directory", [loki(f'sum by (feed, error) (count_over
|
||||
table("Snapshots by server", [loki(f'sum by (instance) (count_over_time({DIR} | message="snapshot" [$__range]))', instant=True)],
|
||||
12, rename={"instance": "Token", "Value": "Snapshots"}, sort=[{"displayName": "Snapshots", "desc": True}])
|
||||
y += 8
|
||||
# The directory's own /metrics, on a port apart from the tunnel's (#224).
|
||||
ts("Directory memory", [prom('directory_process_resident_bytes{job="ipx-directory"}', legend="resident")], 0, unit="bytes",
|
||||
desc="mimalloc gives back what is freed, so this follows what is in use: a rise that never comes down is a leak.")
|
||||
ts("Feed bytes fetched", [prom('rate(ipx_feed_bytes_total{job="ipx"}[$__rate_interval])', legend="ipx"),
|
||||
prom('rate(directory_feed_bytes_total{job="ipx-directory"}[$__rate_interval])', ref="B", legend="directory")],
|
||||
12, unit="Bps", desc="Feeds are read whole, so this is what scans and checks allocate; set it beside memory.")
|
||||
y += 8
|
||||
|
||||
# ---- Traces: all three send them to Tempo (#213)
|
||||
row("Traces")
|
||||
|
||||
Reference in New Issue
Block a user