Reviewing what production's traces and metrics leave out turned up twelve gaps; each is an issue. Metrics. A Prometheus histogram for ipx and the directory both (ipx_directory::metrics::Histograms), which ipx's request histogram (#364) now uses too. - ipx: each statement's time once it has a connection, by its first word, from SeaORM's metric callback; and the wait for one, for hand-written SQL in db.rs, measured as the call less the statement (the callback runs in the poll that returns the result, so the time read after the await is that statement's) (#367). SeaORM's own queries are timed but their wait is not seen: a shortcut, noted. - ipx: counters of what the events say, counted in emit, which every event passes once: feed failures by error.type, new items, downloads saved and failed, DeadAir's seconds found (#368). - ipx: the page's list_shown, play_start and stall timings (#369). - directory: requests by method, route (a route layer puts the matched route on the response, or each feed's page was a row) and status; feeds checked and failed; snapshots taken (#370). - ipx-web: Caddy's own metrics on METRICS_BIND (:9180), served apart from the port the tunnel reaches. `servers { metrics }`, which Caddy 2.11 calls deprecated, because the browser tests' Caddy is 2.6 (#371). Traces. - A list's sync is list_sync with steps, an OPML upload opml_import (#372). Doing so showed sync_group reading every account and its subscription to the list once per listed feed; read once now (#377). - Each command the worker runs, from the socket, the CLI or the page, is a command span with its job inside (#373). - The daemon's start is a trace, from opening the database through the catalogue's repairs to "daemon started" (#374). - The directory's lines carry trace_id and span_id: WithTrace moves to ipx_directory::otel and both use it (#375). - A tap or key is its trace's root span (#376). The page names a root span id in each request's traceparent, where the parent was random and pointed at nothing, and reports the tap (ids, start, its last answer) when the next begins or the page is hidden. opentelemetry 0.33 gives a span builder no way to take ids, so the shared tracer provider has an id generator that takes given ids inside with_ids, on one thread. Alerts. grafana/alerts.py, Grafana alerting provisioning beside the dashboard (#365): ipx, the directory or ipx-web down, scans stopped, a job over 30 minutes, the pool full, 5xx, 502s, Jev refused, Signals stopped, the directory's refresher stopped, Tower's load and memory, the monitoring down, Tempo not reading its blocks, iPX's lines not reaching Loki. No contact point yet: firing shows in Grafana. Prometheus now scrapes Tempo, Loki, Alloy and itself (#366), in the monitoring project's config, not this repository's. The dashboard has database and page-timing panels; the prod check knows the new metrics, spans and alerts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
158 lines
5.4 KiB
Caddyfile
158 lines
5.4 KiB
Caddyfile
# ipx-web (#203): the frontend as static files, and everything else passed to ipx, on one origin,
|
|
# so the session cookie, live updates and Cloudflare Access work as they did when ipx served it.
|
|
# IPX_UPSTREAM where ipx is, host:port (ipx:8099)
|
|
# WEB_BIND where to listen (:8080)
|
|
# WEB_TRACING on: a span for each request to OpenTelemetry (OTEL_EXPORTER_OTLP_ENDPOINT,
|
|
# OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_SERVICE_NAME), continuing the page's traceparent
|
|
# and passing it to ipx, so one trace runs browser, Caddy, ipx (#213). Off: none,
|
|
# and no export failing every few seconds where there is nothing to send to.
|
|
# METRICS_BIND where to serve Caddy's metrics for Prometheus (:9180); keep it off the tunnel
|
|
# TUNNEL_PEER the one address whose Cf-Access-* headers are passed on: where cloudflared's
|
|
# requests come from. Nobody else's are, so nobody else can claim to be signed in
|
|
# by Cloudflare. Unset passes none.
|
|
{
|
|
admin off
|
|
auto_https off
|
|
# Requests, their times and upstream errors as Prometheus metrics, served below (#371).
|
|
# Nested in servers, which Caddy 2.11 calls deprecated, for the browser tests' Caddy, 2.6,
|
|
# which knows no other way; the global `metrics` option once both are 2.8 or later.
|
|
servers {
|
|
metrics
|
|
}
|
|
# Everything else Caddy logs, its errors among them, which carry the request too: the same
|
|
# filter as the access log's below.
|
|
log default {
|
|
output stderr
|
|
format filter {
|
|
wrap json
|
|
fields {
|
|
request>uri query {
|
|
delete token
|
|
}
|
|
request>headers delete
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
# Caddy's own metrics (#371), on a port of their own: the tunnel reaches only WEB_BIND's.
|
|
http://{$METRICS_BIND::9180} {
|
|
metrics
|
|
}
|
|
|
|
(tracing-on) {
|
|
tracing {
|
|
# Named for the request, as ipx names its own (#286): one name for all left every span in
|
|
# Tempo looking the same.
|
|
span "{http.request.method} {http.request.uri.path}"
|
|
}
|
|
}
|
|
(tracing-off) {
|
|
}
|
|
|
|
http://{$WEB_BIND::8080} {
|
|
import tracing-{$WEB_TRACING:off}
|
|
|
|
root * {$WEB_ROOT:/srv}
|
|
|
|
# One JSON line a request on stdout, for Loki and the dashboard's Page row (#210). Not the
|
|
# shared token a ?token= link carries, nor the request's headers, which hold the session
|
|
# cookie and Cloudflare Access's token.
|
|
log {
|
|
output stdout
|
|
format filter {
|
|
wrap json
|
|
fields {
|
|
request>uri query {
|
|
delete token
|
|
}
|
|
request>headers delete
|
|
}
|
|
}
|
|
}
|
|
|
|
# Compressed, zstd or gzip as the browser takes them: the feed list and a page of items went
|
|
# out at over 100 KB each (#239). Not the live updates, which a compressor would hold back
|
|
# until it had enough to send, nor /media, which is audio and video already compressed.
|
|
@compress not path /api/events /media/*
|
|
encode @compress zstd gzip
|
|
|
|
# The copies web/build.mjs compressed are served in place of their file, never asked for by
|
|
# name: /admin.js.br would be the admin page's script without the check below that keeps it to
|
|
# admins (#253).
|
|
# A handle of its own: a bare respond runs after the handles, and the last one serves any file.
|
|
@packed path *.br *.gz
|
|
handle @packed {
|
|
respond 404
|
|
}
|
|
|
|
# Before anything else, so no page or API call sees them: only ipx sets these.
|
|
@forged not remote_ip {$TUNNEL_PEER:255.255.255.255}
|
|
request_header @forged -Cf-Access-Authenticated-User-Email
|
|
request_header @forged -Cf-Access-Jwt-Assertion
|
|
# Whatever header ipx's [web] trusted_header names, when it is not Cloudflare's: ipx trusts it
|
|
# from this address, and anyone could send it through here (#323).
|
|
request_header @forged -{$TRUSTED_HEADER:Cf-Access-Authenticated-User-Email}
|
|
|
|
@api path /api/* /media/*
|
|
handle @api {
|
|
# Live updates are a stream: flushed as they come, never buffered.
|
|
reverse_proxy {$IPX_UPSTREAM:ipx:8099} {
|
|
flush_interval -1
|
|
}
|
|
}
|
|
|
|
# Signing in with the shared token in the address (?token=): ipx sets the session cookie and
|
|
# sends the browser on to the page without it.
|
|
@tokened {
|
|
path / /admin
|
|
query token=*
|
|
}
|
|
handle @tokened {
|
|
rewrite * /api/signin?{query}
|
|
reverse_proxy {$IPX_UPSTREAM:ipx:8099}
|
|
}
|
|
|
|
# The app and the admin page go to whoever ipx says may have them: a browser not signed in is
|
|
# sent to /login, anyone but an admin asking for /admin to the app.
|
|
@page path / /index.html /admin /admin.html /admin.js
|
|
handle @page {
|
|
forward_auth {$IPX_UPSTREAM:ipx:8099} {
|
|
uri /api/page-check
|
|
}
|
|
rewrite / /index.html
|
|
rewrite /admin /admin.html
|
|
header Cache-Control no-cache
|
|
file_server {
|
|
precompressed br gzip
|
|
}
|
|
}
|
|
|
|
# The sign-in form, except for someone Cloudflare Access vouches for, who is sent to the app.
|
|
@login path /login /login.html
|
|
handle @login {
|
|
forward_auth {$IPX_UPSTREAM:ipx:8099} {
|
|
uri /api/login-check
|
|
}
|
|
rewrite * /login.html
|
|
header Cache-Control no-cache
|
|
file_server {
|
|
precompressed br gzip
|
|
}
|
|
}
|
|
|
|
# Scripts and the stylesheet are named by a hash of their contents (web/build.mjs), so a browser
|
|
# keeps them a year; the icons a day, under names that do not change.
|
|
@hashed path /app.js /admin.js /login.js /app.css /hls.js /boot.js /icons/index.json
|
|
header @hashed Cache-Control "public, max-age=31536000, immutable"
|
|
# The same PNG as /favicon.png, for a browser that asks for /favicon.ico on its own.
|
|
header /favicon.ico Content-Type image/png
|
|
@icons path *.png *.svg *.ico *.woff2
|
|
header @icons Cache-Control "public, max-age=86400"
|
|
handle {
|
|
file_server {
|
|
precompressed br gzip
|
|
}
|
|
}
|
|
}
|