POST /api/login answered every wrong password 401 for as long as someone
kept trying. Now ipx counts them by the client's address: the tenth in a
window holds that address off until the window, which starts at its first
wrong password, ends, answered 429 with how long is left. Counted by
address and not name, since by name anyone could lock anyone out; a right
password does not reset the count, or a guesser with an account of their
own could clear it between guesses.
Behind Caddy every peer is Caddy, so the address is the last one in
X-Forwarded-For, taken only from trusted_proxies. Caddy now sets that to
{client_ip}: Cf-Connecting-IP for the tunnel's requests, believed only
from TUNNEL_PEER (servers trusted_proxies), and the peer for everyone
else, replacing whatever the client sent. Checked with Caddy against an
echo upstream: a forged XFF and Cf-Connecting-IP from elsewhere reach ipx
as the peer, the tunnel's Cf-Connecting-IP as itself.
The sign-in load test sends each attempt from its own address, so it
still measures a flood from many, whose passwords are all checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RqkrVDqdiQEMWAkNGQ2tYA
The page and the service worker were strict TypeScript; the tests, the
build, the home page's generator, the load tests, boot.js and the
directory's site.js were plain JavaScript that nothing checked. All of
it is TypeScript now, in five programs, each with its own globals, all
checked by npm run typecheck: the page (tsconfig.json, with boot.ts and
the directory's site.ts), the worker, Node's (tsconfig.tools.json: the
Node tests, the browser suite, the build, the site generator, the load
harness), and k6's (tsconfig.k6.json, with @types/k6).
Node 24 runs the .ts files by stripping their types, so they hold only
TypeScript that erases (erasableSyntaxOnly) and import each other by
.ts name; package.json says "type": "module" for it, and Dockerfile.web
moves to Node 24 to run build.mts. k6 2.3 strips its scripts' types
itself.
The browser suite's page.evaluate calls reach the page's globals; the
page's tsc run now also writes its declarations to target/page-types,
which the suite's program includes, so those calls are typed by the
page's own code rather than by hand-written copies.
boot.ts is compiled by build.mts with swc, as the worker is, and named
by the hash of what is served. The directory's build has no Node, so
site.js is compiled from site.ts and committed (node web/build.mts
--directory-script), and tests/directory-script.ts fails if they drift.
Checked: the site generator's 139 pages are byte for byte what the
JavaScript made; the page build's app.js is identical and only boot.js
(now minified) and its hash differ; contrast, page-smoke, page-unit,
native-bridge, the browser suite (96 passed, 2 skipped) and the
directory's and ipx's Rust tests pass; k6 parses each load script. The
load tests themselves did not run: they build a release binary outside
the capped builder. .agents/skills/security-audit is an installed
skill's, not ours, and is left as it is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With no network the home-screen app showed nothing: no service worker,
and the lists the page had fetched, full text included, lived only in
its memory.
A service worker (web/sw/sw.ts, TypeScript checked against a worker's
globals by tsconfig.sw.json, built by swc to /sw.js) goes to the network
first for the page and every API read and keeps the answers, so online
nothing is an old copy and #388's version check works as before;
offline the kept copy answers. Scripts named by hash come from what is
kept. Reads, flags and places in an episode that cannot reach the
server wait in an outbox, sent in order when the connection is back
(and new ones queue behind them meanwhile, so none lands out of turn);
a kept list served offline shows the reads waiting there.
Pictures: a worker cannot keep another site's pictures sensibly, since
browsers count each opaque answer as megabytes. ipx now remembers the
picture addresses in item text it sends (bounded, in memory) and
/api/art fetches those as it does artwork, through the guarded client;
the worker keeps them, for unread items, under a 200 MB budget, and
serves them offline in place of the publisher's. Online, pictures still
come from their publisher.
Sign-out, or another account signing in on the device, clears what was
kept. The browser suite blocks the worker except in the offline tests,
since it answers before page.route can.
The fixture JPEGs are placeholders a browser cannot draw; the offline
test uses a real PNG.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The browser tests ran Debian's Caddy 2.6, which has no global metrics
option, so the Caddyfile nested it in servers, which production's 2.11
logs as deprecated at every start. The tests now run Caddy 2.11 from
Caddy's own repository (install.sh), the version the caddy:2 image has.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewing what production's traces and metrics leave out turned up
twelve gaps; each is an issue.
Metrics. A Prometheus histogram for ipx and the directory both
(ipx_directory::metrics::Histograms), which ipx's request histogram
(#364) now uses too.
- ipx: each statement's time once it has a connection, by its first
word, from SeaORM's metric callback; and the wait for one, for
hand-written SQL in db.rs, measured as the call less the statement
(the callback runs in the poll that returns the result, so the time
read after the await is that statement's) (#367). SeaORM's own queries
are timed but their wait is not seen: a shortcut, noted.
- ipx: counters of what the events say, counted in emit, which every
event passes once: feed failures by error.type, new items, downloads
saved and failed, DeadAir's seconds found (#368).
- ipx: the page's list_shown, play_start and stall timings (#369).
- directory: requests by method, route (a route layer puts the matched
route on the response, or each feed's page was a row) and status;
feeds checked and failed; snapshots taken (#370).
- ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from
the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11
calls deprecated, because the browser tests' Caddy is 2.6 (#371).
Traces.
- A list's sync is list_sync with steps, an OPML upload opml_import
(#372). Doing so showed sync_group reading every account and its
subscription to the list once per listed feed; read once now (#377).
- Each command the worker runs, from the socket, the CLI or the page,
is a command span with its job inside (#373).
- The daemon's start is a trace, from opening the database through the
catalogue's repairs to "daemon started" (#374).
- The directory's lines carry trace_id and span_id: WithTrace moves to
ipx_directory::otel and both use it (#375).
- A tap or key is its trace's root span (#376). The page names a root
span id in each request's traceparent, where the parent was random and
pointed at nothing, and reports the tap (ids, start, its last answer)
when the next begins or the page is hidden. opentelemetry 0.33 gives a
span builder no way to take ids, so the shared tracer provider has an
id generator that takes given ids inside with_ids, on one thread.
Alerts. grafana/alerts.py, Grafana alerting provisioning beside the
dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a
job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals
stopped, the directory's refresher stopped, Tower's load and memory, the
monitoring down, Tempo not reading its blocks, iPX's lines not reaching
Loki. No contact point yet: firing shows in Grafana. Prometheus now
scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's
config, not this repository's.
The dashboard has database and page-timing panels; the prod check knows
the new metrics, spans and alerts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx trusts [web] trusted_header from Caddy's address, and Caddy
stripped only the two Cf-Access-* names from requests not from
TUNNEL_PEER. With trusted_header set to anything else (X-Remote-User
for another proxy), a request through ipx-web carried it to ipx as
Caddy's and was believed: anyone could sign in as anyone.
TRUSTED_HEADER, defaulting to Cloudflare's name, is stripped the same
way. Production uses Cloudflare's header and sets nothing, so it
behaves as before. caddy adapt shows the delete list with and without.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy named every span ipx-web, so Tempo's lists and trace view showed the
same name for each request. Caddy 2.11 takes placeholders in the span name:
now the method and path, as ipx names its own. The prod check's span list
shows a span's url.path when it has no route.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured 2026-10-16, uncompressed until now: the directory's answers (the feed list ipx reads,
603 KB of JSON, ~134 ms of a 159 ms Directory request), ipx's snapshot upload (~100 KB each half
hour), both /metrics (4.7 and 0.9 KB every 15 s) and every trace export to Tempo. Already
compressed: ipx-web's answers (zstd, #239), the home page and the directory's public pages
through Cloudflare (brotli, zstd).
- The directory compresses with brotli, zstd or gzip as asked, and unpacks a gzipped request.
- ipx sends its snapshot gzipped; the directory is deployed first.
- Both /metrics answer gzipped, as Prometheus asks.
- opentelemetry-otlp's gzip-http: OTEL_EXPORTER_OTLP_COMPRESSION=gzip compresses the export, set
for all three in production's compose file; Caddy's exporter reads the same variable.
- web/build.mjs writes .br (quality 11) and .gz (level 9) beside each text file, and Caddy serves
them in place (file_server precompressed): app.js 30 KB where zstd on the fly gave 37, app.css
11 where 14. A .br or .gz asked for by name is refused, or /admin.js.br would be the admin
page's script without the check that keeps it to admins; in a handle of its own, since a bare
respond runs after the handles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.
- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
something is missing a read would fill or the check is forced; and a feed without a schedule
of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).
#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy's tracing directive, switched by WEB_TRACING so a server without a collector does not log
a failed export every few seconds. It continues an incoming traceparent (the page starts one per
tap, #190) and passes it to ipx, so a request is one trace from the browser through Caddy into
ipx, and Caddy's access log carries its traceID. Checked against Tempo with a probe container:
service.name and deployment.environment.name come from the OTEL_ variables, over OTLP/HTTP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy logs one JSON line a request, filtered: no ?token= in the address and no request
headers, which hold the session cookie and Access's token. Its default log, errors included,
has the same filter, since an error line carries the request too. The directory logs a line a
refresher round, so the dashboard can show its progress.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy reports an IPv4 peer as plain IPv4 even on its dual-stack socket, so remote_ip
192.168.64.1 matched all along. The stripping seen came from testing on Tower itself, whose
requests reach the published port as 192.168.1.130, rightly not the tunnel. From
cloudflared's network a forged Access token reaches ipx's check and is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy listens dual-stack, so the ipx_default gateway connects as ::ffff:192.168.64.1, and
remote_ip 192.168.64.1 does not match it: every Cf-Access-* header was stripped, the
tunnel's too. The browser suite binds Caddy to 127.0.0.1, IPv4 only, so it never saw it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
web/build.mjs builds a static site, web/dist, with the icons and typeface
beside the pages; Dockerfile.web serves it with Caddy, which passes /api
and /media to ipx on the same origin, so cookies, live updates and
Cloudflare Access work as before. ipx serves no page any more: build.rs
and every include_str! of the web files are gone, and its image is built
without node.
What ipx did to the page as it served it is kept another way. Caddy asks
ipx before serving a page (forward_auth): /api/page-check sends a browser
not signed in to /login and a non-admin away from /admin, and refuses them
/admin.js; /api/login-check sends someone Access vouches for past the
form; a page asked for with ?token= goes to /api/signin, which sets the
cookie. The theme and the admin link, once written into the page per
person, are set by web/boot.js in <head> from the theme cookie and the
last visit, before the first frame, and /api/me brings the version and
the account's theme from before.
Caddy strips the Cf-Access-* headers from every peer but TUNNEL_PEER, the
address cloudflared's requests come from, which is the boundary
trusted_proxies drew; ipx then trusts only Caddy's address.
The browser suite runs Caddy in front of its daemon, so it tests what
production would run. Not deployed: production's cutover is on #203.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
