Reviewing what production's traces and metrics leave out turned up
twelve gaps; each is an issue.
Metrics. A Prometheus histogram for ipx and the directory both
(ipx_directory::metrics::Histograms), which ipx's request histogram
(#364) now uses too.
- ipx: each statement's time once it has a connection, by its first
word, from SeaORM's metric callback; and the wait for one, for
hand-written SQL in db.rs, measured as the call less the statement
(the callback runs in the poll that returns the result, so the time
read after the await is that statement's) (#367). SeaORM's own queries
are timed but their wait is not seen: a shortcut, noted.
- ipx: counters of what the events say, counted in emit, which every
event passes once: feed failures by error.type, new items, downloads
saved and failed, DeadAir's seconds found (#368).
- ipx: the page's list_shown, play_start and stall timings (#369).
- directory: requests by method, route (a route layer puts the matched
route on the response, or each feed's page was a row) and status;
feeds checked and failed; snapshots taken (#370).
- ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from
the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11
calls deprecated, because the browser tests' Caddy is 2.6 (#371).
Traces.
- A list's sync is list_sync with steps, an OPML upload opml_import
(#372). Doing so showed sync_group reading every account and its
subscription to the list once per listed feed; read once now (#377).
- Each command the worker runs, from the socket, the CLI or the page,
is a command span with its job inside (#373).
- The daemon's start is a trace, from opening the database through the
catalogue's repairs to "daemon started" (#374).
- The directory's lines carry trace_id and span_id: WithTrace moves to
ipx_directory::otel and both use it (#375).
- A tap or key is its trace's root span (#376). The page names a root
span id in each request's traceparent, where the parent was random and
pointed at nothing, and reports the tap (ids, start, its last answer)
when the next begins or the page is hidden. opentelemetry 0.33 gives a
span builder no way to take ids, so the shared tracer provider has an
id generator that takes given ids inside with_ids, on one thread.
Alerts. grafana/alerts.py, Grafana alerting provisioning beside the
dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a
job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals
stopped, the directory's refresher stopped, Tower's load and memory, the
monitoring down, Tempo not reading its blocks, iPX's lines not reaching
Loki. No contact point yet: firing shows in Grafana. Prometheus now
scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's
config, not this repository's.
The dashboard has database and page-timing panels; the prod check knows
the new metrics, spans and alerts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ipx trusts [web] trusted_header from Caddy's address, and Caddy
stripped only the two Cf-Access-* names from requests not from
TUNNEL_PEER. With trusted_header set to anything else (X-Remote-User
for another proxy), a request through ipx-web carried it to ipx as
Caddy's and was believed: anyone could sign in as anyone.
TRUSTED_HEADER, defaulting to Cloudflare's name, is stripped the same
way. Production uses Cloudflare's header and sets nothing, so it
behaves as before. caddy adapt shows the delete list with and without.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy named every span ipx-web, so Tempo's lists and trace view showed the
same name for each request. Caddy 2.11 takes placeholders in the span name:
now the method and path, as ipx names its own. The prod check's span list
shows a span's url.path when it has no route.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured 2026-10-16, uncompressed until now: the directory's answers (the feed list ipx reads,
603 KB of JSON, ~134 ms of a 159 ms Directory request), ipx's snapshot upload (~100 KB each half
hour), both /metrics (4.7 and 0.9 KB every 15 s) and every trace export to Tempo. Already
compressed: ipx-web's answers (zstd, #239), the home page and the directory's public pages
through Cloudflare (brotli, zstd).
- The directory compresses with brotli, zstd or gzip as asked, and unpacks a gzipped request.
- ipx sends its snapshot gzipped; the directory is deployed first.
- Both /metrics answer gzipped, as Prometheus asks.
- opentelemetry-otlp's gzip-http: OTEL_EXPORTER_OTLP_COMPRESSION=gzip compresses the export, set
for all three in production's compose file; Caddy's exporter reads the same variable.
- web/build.mjs writes .br (quality 11) and .gz (level 9) beside each text file, and Caddy serves
them in place (file_server precompressed): app.js 30 KB where zstd on the fly gave 37, app.css
11 where 14. A .br or .gz asked for by name is refused, or /admin.js.br would be the admin
page's script without the check that keeps it to admins; in a handle of its own, since a bare
respond runs after the handles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measured on 2026-10-15: the feed list and item lists went out uncompressed at over 100 KB; ipx
read 2.6 GB of feed bodies a day and the directory 3.9 GB.
- ipx-web compresses with zstd or gzip, leaving out /api/events, which a compressor would hold
back, and /media (#239).
- A body identical to the last one (SHA-256 in feeds.body_hash) is taken as a 304, unless
something is missing a read would fill or the check is forced; and a feed without a schedule
of its own whose body has been the same waits a twelfth of that, up to four hours, beside the
failure backoff (#242). 35 Patreon show feeds sent ~300 KB each hourly, about 250 MB a day.
- The directory checks each feed every twelve hours, not six (#243).
- reqwest accepts brotli and zstd too (#244).
#240 (cache the feed list's counts) is left open: at production's size on the test Postgres
that count takes 12 ms and All Subscriptions 32 ms, against 366 and 300 ms in production, so
the cause is production's database, not the query (#241).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy's tracing directive, switched by WEB_TRACING so a server without a collector does not log
a failed export every few seconds. It continues an incoming traceparent (the page starts one per
tap, #190) and passes it to ipx, so a request is one trace from the browser through Caddy into
ipx, and Caddy's access log carries its traceID. Checked against Tempo with a probe container:
service.name and deployment.environment.name come from the OTEL_ variables, over OTLP/HTTP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy logs one JSON line a request, filtered: no ?token= in the address and no request
headers, which hold the session cookie and Access's token. Its default log, errors included,
has the same filter, since an error line carries the request too. The directory logs a line a
refresher round, so the dashboard can show its progress.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy reports an IPv4 peer as plain IPv4 even on its dual-stack socket, so remote_ip
192.168.64.1 matched all along. The stripping seen came from testing on Tower itself, whose
requests reach the published port as 192.168.1.130, rightly not the tunnel. From
cloudflared's network a forged Access token reaches ipx's check and is refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Caddy listens dual-stack, so the ipx_default gateway connects as ::ffff:192.168.64.1, and
remote_ip 192.168.64.1 does not match it: every Cf-Access-* header was stripped, the
tunnel's too. The browser suite binds Caddy to 127.0.0.1, IPv4 only, so it never saw it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
web/build.mjs builds a static site, web/dist, with the icons and typeface
beside the pages; Dockerfile.web serves it with Caddy, which passes /api
and /media to ipx on the same origin, so cookies, live updates and
Cloudflare Access work as before. ipx serves no page any more: build.rs
and every include_str! of the web files are gone, and its image is built
without node.
What ipx did to the page as it served it is kept another way. Caddy asks
ipx before serving a page (forward_auth): /api/page-check sends a browser
not signed in to /login and a non-admin away from /admin, and refuses them
/admin.js; /api/login-check sends someone Access vouches for past the
form; a page asked for with ?token= goes to /api/signin, which sets the
cookie. The theme and the admin link, once written into the page per
person, are set by web/boot.js in <head> from the theme cookie and the
last visit, before the first frame, and /api/me brings the version and
the account's theme from before.
Caddy strips the Cf-Access-* headers from every peer but TUNNEL_PEER, the
address cloudflared's requests come from, which is the boundary
trusted_proxies drew; ipx then trusts only Caddy's address.
The browser suite runs Caddy in front of its daemon, so it tests what
production would run. Not deployed: production's cutover is on #203.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
