rays/ipx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 18 Wiki Activity

mimalloc for ipx and the directory, and metrics and traces for memory (#224)

Browse Source 
ipx's resident memory climbed from about 40 MB to 500-850 MB each night while scans ran five
times as often, and stayed there until a restart; the directory sat at 1.25 GiB idle. glibc's
allocator keeps what each thread's arena freed. mimalloc gives freed pages back, so resident
memory now follows what is in use. jemalloc was the first choice for its in-use figures, but its
upstream was archived in 2025.

Declared once in the directory crate, which ipx builds in, so both binaries get it.

What was missing to see it: the directory served no /metrics (now on metrics_bind, a port apart
from the tunnel's), nothing counted the feed bytes fetched, and the fetch span had no body size.
The feed crate's spans never reached Tempo either: both trace exporters filtered to their own
crate, so ipx_feed's fetch and parse spans were dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ray Slakinski
2026-10-13 17:15:01 +00:00
parent e4b505564c
commit 45c9f9252b
15 changed files with 138 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.claude/skills/ipx-prod-check/SKILL.md
View File

@@ -96,7 +96,8 @@ from 2026-10-11 15:15 UTC. All gauges:
| `ipx_torrent_slots_free` | of 2 |
| `ipx_deadair_playing` | episodes being played through DeadAir's playlist |
| `ipx_runtime_workers`, `ipx_runtime_tasks`, `ipx_runtime_queue` | tokio: threads, live tasks, tasks waiting for a thread |
| `ipx_process_resident_bytes`, `ipx_process_open_fds` | memory and open files and sockets |
| `ipx_process_resident_bytes`, `ipx_process_open_fds` | memory and open files and sockets; mimalloc gives back what is freed, so memory follows use and a rise that never comes down is a leak (#224) |
| `ipx_feed_fetches_total`, `ipx_feed_bytes_total` | counters: feeds fetched (304s included) and the bytes of their bodies, read whole, so what scans allocate |
| `ipx_uptime_seconds`, `ipx_build_info{version}` | since the daemon started; what runs |
| `ipx_worker_busy`, `ipx_worker_job_seconds{kind}` | the worker's job now (`scan`, `reap`, `download`, `status`, `reload`) and for how long; the healthcheck passes while it is stuck |
| `ipx_jobs_total{kind}`, `ipx_job_seconds_total{kind}` | counters: jobs finished, and their seconds all told |
@@ -108,6 +109,7 @@ from 2026-10-11 15:15 UTC. All gauges:
| `ipx_feed_retries` | feeds waiting to retry a failed name lookup |
| `ipx_art_cache_bytes`, `ipx_art_cache_limit_bytes` | artwork kept, as of the last sweep, and the most kept: kept sits at the limit by design |
| `ipx_art_evicted_total`, `ipx_art_requests_total{result="hit"\|"miss"}` | counters: images the sweep threw out to stay under the limit, and artwork served from disk or fetched from the publisher; many evicted at every sweep with a falling hit rate means the limit is too small |
| `directory_process_resident_bytes`, `directory_process_open_fds`, `directory_feed_fetches_total`, `directory_feed_bytes_total`, `directory_uptime_seconds`, `directory_build_info` | the directory's, `job="ipx-directory"`, from its `metrics_bind` (production `192.168.1.130:9188`, LAN only) |
| `ipx_feeds`, `ipx_downloads_pending`, `ipx_files_downloaded` | the library, as the healthcheck last counted it (every 30 s) |
### The checks