A subscribed OPML with very many outlines causes quadratic sync work and an unbounded set of child feeds scanned on the shared worker #337
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by the security-audit skill (run-1, 2026-10-20). Source-read only: nothing was run, so this is a lead that still needs its local check, not a confirmed finding. Two independent verifiers did not disprove it.
find_feed accepts any body that is_opml sniffs as OPML (crates/feed/src/lib.rs:533), so any signed-in user, admin or not, can subscribe through add_feed to a remote OPML whose publisher is untrusted. On each read, scan_one hands it to sync_opml, parse_opml returns every outline at any depth with no count limit from a body of up to 50 MB, and sync_group loops over all of them. For every child not already managed it reads every feed row (ctx.db.feed_urls()) and rebuilds a BTreeMap of every catalogue, managed, added and known id before unique_slug, so the first sync of N new children is O(N^2) in rows read and allocation (identical titles add a further linear slug probe per child). It then writes a managed feed row per child and, for every managed child times every user, reads subscription() and calls subscribe(), leaving persistent rows for each subscriber of the OPML. Every newly added child is pushed to fresh and scanned in the same job, one at a time with no prefetch (scan_one with None), each fetch bounded by a 10 s connect timeout and a 30 s request timeout, on the daemon's single worker, so scans of every other user's feeds and their downloads wait behind it. Source corrections to the hunter's wording: on later scheduled scans children are fetched six ahead in parallel (AHEAD = 6, main.rs:1237) rather than strictly one at a time, and children that fail back off (due_after/backoff, up to a day), so the repeat cost is chiefly the first sync, a forced refresh, or children the publisher keeps answering slowly but successfully; the sustained-hours effect depends on N and on the publisher's server behaviour and was not measured.
Claimed root cause. No limit on the number of children a list-type feed (OPML or Patreon) may introduce, a full feeds-table read and id-map rebuild inside sync_group's per-child loop, and inline sequential scanning of every newly listed child on the single worker without a per-scan budget.
Local check. {'local': 'In a sandbox, add a test beside the existing sync_group/OPML tests in crates/ipx/src/main.rs on SQLite in a temp dir. Call sync_group with a parent and synthetic listed vectors of 500, 1,000 and 2,000 (title, http://127.0.0.1:1/) entries and time each call or count queries, expecting roughly 4x growth per doubling. Then run fetch with a local OPML server listing N children on 127.0.0.1:1 (connection refused, fast and safe) and check all N are queued as fresh and attempted in the same job; repeat with a local server that accepts and answers each child slowly (for example 5 s) to measure how long the job holds the worker. After the fix: a cap on children per list with a logged refusal, feed_urls() hoisted out of the loop, and new children left to the normal schedule or the prefetching path instead of scanned inline.', 'deployment': 'Owner check: whether non-admin accounts that can add feeds exist in production (auto_create_users, Cloudflare Access users), and how many users subscribe to OPML-type feeds today, which multiplies the subscription rows.'}
Owner check (no probing). Owner check: whether non-admin accounts that can add feeds exist in production (auto_create_users, Cloudflare Access users), and how many users subscribe to OPML-type feeds today, which multiplies the subscription rows.
Full detail:
/config/security-audit-skill/ipx/run-1/NEEDS-VALIDATION.md(fingerprintipx:main:sync_group-unbounded-opml-children).Fixed in
d35b805: a list may bring in 2,000 feeds (davewiner's had 922); one with more is refused whole, the feed shows the error, and nothing changes. sync_group reads the feeds table once per sync instead of once per new feed, and checks ids against sets. Deployed.