Artwork kept on disk is read from FUSE, blocking the async threads #345

Closed
opened 2026-10-23 05:35:26 -07:00 by rays · 1 comment
Owner

The art cache is data_dir()/art, /data/art, which is /mnt/user/ipx/art (shfs, FUSE). A hit (art::read) makes four blocking std::fs calls on a tokio worker: the type, the body, an open and an mtime write. A page asks for ~50 images at once. Over 7 days 90% of /api/art requests were hits (588 of 649), yet p50 is 11 ms, p75 124 ms, p90 1053 ms, p95 1177 ms. art::trim also walks the 1884 files synchronously in the reap job. Fix: read, write and trim in spawn_blocking, and mount the cache from /mnt/fast (/mnt/fast/appdata/ipx/art:/data/art in the compose file).

The art cache is data_dir()/art, /data/art, which is /mnt/user/ipx/art (shfs, FUSE). A hit (art::read) makes four blocking std::fs calls on a tokio worker: the type, the body, an open and an mtime write. A page asks for ~50 images at once. Over 7 days 90% of /api/art requests were hits (588 of 649), yet p50 is 11 ms, p75 124 ms, p90 1053 ms, p95 1177 ms. art::trim also walks the 1884 files synchronously in the reap job. Fix: read, write and trim in spawn_blocking, and mount the cache from /mnt/fast (/mnt/fast/appdata/ipx/art:/data/art in the compose file).
rays added the bug label 2026-10-23 05:35:26 -07:00
Author
Owner

Fixed in cb4ee5a: art::read, art::write and the reap job's art::trim run in spawn_blocking. The cache moved off FUSE: /mnt/user/ipx/art was copied (cp -a, mtimes kept) to /mnt/fast/appdata/ipx/art, mounted at /data/art in the compose file. The old copy is still at /mnt/user/ipx/art for rollback. Deployed; a cached cover answers in 2 ms.

Fixed in cb4ee5a: art::read, art::write and the reap job's art::trim run in spawn_blocking. The cache moved off FUSE: /mnt/user/ipx/art was copied (cp -a, mtimes kept) to /mnt/fast/appdata/ipx/art, mounted at /data/art in the compose file. The old copy is still at /mnt/user/ipx/art for rollback. Deployed; a cached cover answers in 2 ms.
rays closed this issue 2026-10-23 08:35:34 -07:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rays/ipx#345