Disconnecting sign-in requests escape the password-check limit and run unbounded Argon2 work on the blocking pool #324
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found by the security-audit skill (run-1, 2026-10-20). Source-read only: nothing was run, so this is a lead that still needs its local check, not a confirmed finding. Two independent verifiers did not disprove it.
check_password limits Argon2 verifications with the CHECKS semaphore (half the cores), so that a flood of sign-ins waits on itself (#137). The permit
_turnis a local of the async fn, but the verification runs in tokio::task::spawn_blocking, which cannot be cancelled. In hyper 1.11.1, while a handler is pending after the request body has been read, the HTTP/1 dispatcher keeps polling the socket (poll_read_keep_alive -> mid_message_detect_eof, with half-close off by default). An EOF there ends the connection future with an error, and axum's per-connection task then drops the handler future, and the permit with it. The Argon2 call keeps running on its blocking thread, and the next request in the queue takes the permit. So the CHECKS bound holds only for clients that wait for their answer. An unauthenticated client that sends POST /api/login and closes each connection soon after its check starts can keep many more than CHECKS verifications running at once. The ceiling is tokio's default of 512 blocking threads, because main is a plain #[tokio::main]. Each verification is Argon2id with argon2 0.6.0's defaults (m=19 MiB, t=2), so the ceiling is all cores busy and about 9.5 GiB of memory. Tower has 4-7 GB to spare and no swap. Work past the 512 waits in tokio's unbounded blocking queue, together with the daemon's other spawn_blocking and tokio::fs work. To build the backlog, the attacker has to start checks faster than the cores finish them, a few hundred small requests a second. In production ipx's port is published nowhere, so this goes through Caddy (ipx-web), and whether the attack works depends on Caddy closing or cancelling the upstream request when the client goes away.Claimed root cause. The concurrency permit (
let Ok(_turn) = CHECKS.acquire().await, auth.rs:48) lives in the cancellable async future, not in the uncancellable spawn_blocking closure that does the Argon2 work (auth.rs:49-55). Dropping the future at the JoinHandle await frees the permit before the work ends.Local check. {'local': 'In a sandbox, add a test in crates/ipx/src/auth.rs on a multi-thread runtime. Wrap verify_password, for tests only, with an AtomicUsize that counts verifications in progress and records the peak. Spawn 4 x CHECKS tasks that each call check_password("x".into(), None). After each task acquires its permit (a 1 ms sleep), abort it. The bug is shown if the peak goes above CHECKS's permit count. Service level, against a scratch daemon on loopback with a SQLite test database: send 50 POST /api/login requests with a JSON body, each closing its socket about 5 ms after sending. Watch Threads and VmRSS in /proc//status rise far beyond CHECKS threads and CHECKS x 19 MiB. Fix: make CHECKS a LazyLock<Arc>, take
let turn = CHECKS.clone().acquire_owned().await, and move it into the closure (move || { let _turn = turn; ... }), so the permit is freed only when the Argon2 work ends. The same test then never sees more than CHECKS running.', 'deployment': "Owner-observed and non-destructive: reproduce ipx-web's Caddy version and Caddyfile in front of a scratch ipx, not production. Send one POST /api/login, close the client socket before the answer comes, and check in Caddy's debug log for a 'context canceled' upstream error, or check with ss on the ipx side that the upstream connection closed. Also note the host's MemAvailable to judge the 512 x 19 MiB ceiling."}Owner check (no probing). Owner-observed and non-destructive: reproduce ipx-web's Caddy version and Caddyfile in front of a scratch ipx, not production. Send one POST /api/login, close the client socket before the answer comes, and check in Caddy's debug log for a 'context canceled' upstream error, or check with ss on the ipx side that the upstream connection closed. Also note the host's MemAvailable to judge the 512 x 19 MiB ceiling.
Full detail:
/config/security-audit-skill/ipx/run-1/NEEDS-VALIDATION.md(fingerprintipx:auth.rs:check_password:permit-released-on-cancel-blocking-argon2-continues).Fixed in
79b496b: the CHECKS permit is owned and moved into the spawn_blocking closure, so it is released when the Argon2 check finishes, not when the request's future is dropped. Deployed.