Memory: no way to tell what holds it, and the directory reports none #224

New Issue

rays · 2026-10-13T10:07:55-07:00

rays commented

2026-10-13 10:07:55 -07:00

ipx's resident memory climbs from about 40 MB to 500-850 MB between 23:30 and 03:00 UTC, while scans run five times as often (about 90 an hour instead of 15-20), and stays near 645 MB until a restart. ipx-directory sits at 1.25 GiB doing almost nothing (docker stats). Nothing says why:

ipx_process_resident_bytes cannot tell memory in use from memory the allocator freed and kept (glibc's arenas), so a leak and fragmentation look the same.
The fetch span has no body size, so a trace cannot say which feeds are large; feeds are read whole (crates/feed fetch) with no limit.
No counter of bytes fetched to set beside memory.
ipx-directory serves no /metrics at all, so its memory has no history.

Found looking into memory use on 2026-10-13.

ipx's resident memory climbs from about 40 MB to 500-850 MB between 23:30 and 03:00 UTC, while scans run five times as often (about 90 an hour instead of 15-20), and stays near 645 MB until a restart. ipx-directory sits at 1.25 GiB doing almost nothing (docker stats). Nothing says why: * ipx_process_resident_bytes cannot tell memory in use from memory the allocator freed and kept (glibc's arenas), so a leak and fragmentation look the same. * The fetch span has no body size, so a trace cannot say which feeds are large; feeds are read whole (crates/feed fetch) with no limit. * No counter of bytes fetched to set beside memory. * ipx-directory serves no /metrics at all, so its memory has no history. Found looking into memory use on 2026-10-13.

rays added the enhancement label 2026-10-13 10:07:55 -07:00

rays referenced this issue from a commit

2026-10-13 10:15:04 -07:00

mimalloc for ipx and the directory, and metrics and traces for memory (#224)

rays commented

2026-10-13 10:21:20 -07:00

Fixed in 45c9f92, deployed (ipx and ipx-directory 1.0.1-dev):

Both use mimalloc instead of glibc's allocator, which gives freed pages back, so resident memory now follows what is in use: a rise that never comes down is a leak. (jemalloc was the first choice for its in-use figures; its upstream was archived in 2025.)
The directory serves /metrics on metrics_bind, 192.168.1.130:9188, scraped as job="ipx-directory": resident memory, open files, feeds fetched and their bytes, uptime, version.
ipx's /metrics adds ipx_feed_fetches_total and ipx_feed_bytes_total.
The fetch span carries http.response.body.size and parse has a span with the body's size; both exporters now pass ipx_feed's spans, which they had been dropping.
Grafana: Directory memory, and Feed bytes fetched for both, in the ipx-directory row.

The overnight scans (23:30-03:00 UTC) are the test: ipx's memory should come back down after them.

Fixed in 45c9f92, deployed (ipx and ipx-directory 1.0.1-dev): * Both use mimalloc instead of glibc's allocator, which gives freed pages back, so resident memory now follows what is in use: a rise that never comes down is a leak. (jemalloc was the first choice for its in-use figures; its upstream was archived in 2025.) * The directory serves /metrics on metrics_bind, 192.168.1.130:9188, scraped as job="ipx-directory": resident memory, open files, feeds fetched and their bytes, uptime, version. * ipx's /metrics adds ipx_feed_fetches_total and ipx_feed_bytes_total. * The fetch span carries http.response.body.size and parse has a span with the body's size; both exporters now pass ipx_feed's spans, which they had been dropping. * Grafana: Directory memory, and Feed bytes fetched for both, in the ipx-directory row. The overnight scans (23:30-03:00 UTC) are the test: ipx's memory should come back down after them.

rays closed this issue

2026-10-13 10:21:21 -07:00

rays commented

2026-10-14 07:10:45 -07:00

Verified overnight 2026-10-13/14 on mimalloc: ipx's resident memory peaked at 143 MB during the daily batch (853 MB the night before on glibc) and came back to about 100 MB by 10:00 UTC.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: rays/ipx#224