A flood of wrong passwords stalls every other request #137

New Issue

rays · 2026-10-05T10:21:35-07:00

rays commented

2026-10-05 10:21:35 -07:00

Found by the load tests (#133) on 2026-10-05. Forty clients sending wrong passwords to /api/login, 54 attempts a second, which needs no account, made everyone else's requests take 4s (median 3.96s for /api/feeds, normally about 20ms), and ipx status, the healthcheck, took up to 1.3s.

login checks the password with Argon2id (19 MiB, two passes: tens of milliseconds of CPU) inside the async handler, on one of the runtime's worker threads, so a handful of attempts at once holds every worker the rest of the server answers on. The LAN port is open to anyone on the network; Cloudflare Access fronts only the tunnel.

Found by the load tests (#133) on 2026-10-05. Forty clients sending wrong passwords to /api/login, 54 attempts a second, which needs no account, made everyone else's requests take 4s (median 3.96s for /api/feeds, normally about 20ms), and `ipx status`, the healthcheck, took up to 1.3s. login checks the password with Argon2id (19 MiB, two passes: tens of milliseconds of CPU) inside the async handler, on one of the runtime's worker threads, so a handful of attempts at once holds every worker the rest of the server answers on. The LAN port is open to anyone on the network; Cloudflare Access fronts only the tunnel.

rays referenced this issue from a commit

2026-10-05 10:38:31 -07:00

Check passwords a few at a time off the async workers, and check unknown names too (#137, #138)

rays referenced this issue from a commit

2026-10-05 10:38:31 -07:00

Load tests with k6, for what many people at once do to the server (#133)

rays commented

2026-10-05 10:41:17 -07:00

Fixed in f1d3604: auth::check_password runs Argon2 on the blocking pool, at most half the cores at once. Under the load test's flood (40 clients of wrong passwords) everyone else's p95 went from 4.29s to 57ms and the slowest ipx status from 1.3s to 90ms. Deployed 2026-10-05 (0.10.1-dev).

Fixed in f1d3604: auth::check_password runs Argon2 on the blocking pool, at most half the cores at once. Under the load test's flood (40 clients of wrong passwords) everyone else's p95 went from 4.29s to 57ms and the slowest ipx status from 1.3s to 90ms. Deployed 2026-10-05 (0.10.1-dev).

rays closed this issue

2026-10-05 10:41:17 -07:00

rays referenced this issue

2026-10-05 10:41:19 -07:00

Load tests: what many people at once do to the server, which the browser tests cannot show #133

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: rays/ipx#137