diff --git a/docs/architecture.md b/docs/architecture.md index 1104ef5..9bac985 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,13 +1,27 @@ # How it works -One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a Unix-socket -control server, and the web UI. Everything else is a CLI that either does the work itself or hands -it to a running daemon. +Three services, each a container of its own with its own version and changelog (#207): -The repo is a Cargo workspace (#202): `crates/ipx` is the daemon and CLI, and `crates/feed` -(`ipx-feed`) fetches and parses feeds, shared with `crates/directory` (`ipx-directory`), the -directory of public podcasts that ipx servers share ([directory.md](directory.md)). -The version is the workspace's, in the root `Cargo.toml`. +| Service | What it is | Source | Image | +|---|---|---|---| +| ipx-web | Caddy: serves the page, a static site built from `web/`, and passes `/api` and `/media` to ipx on the same address. It asks ipx before it serves the app or the admin page (`/api/page-check`) | `web/`, `web/Caddyfile` | `Dockerfile.web` | +| ipx | The daemon and CLI, one binary. `ipx daemon` runs a scheduler, a Unix-socket control server and the HTTP API in one process; everything else is a CLI that either does the work itself or hands it to a running daemon | `crates/ipx` | `Dockerfile` | +| ipx-directory | The directory of public podcasts that ipx servers share. ipx asks it on its people's behalf; the page never talks to it ([directory.md](directory.md)) | `crates/directory` | `Dockerfile.directory` | + +The Rust source is a Cargo workspace (#202); `crates/feed` (`ipx-feed`) fetches and parses feeds +for both Rust services. Versions: ipx's in `crates/ipx/Cargo.toml`, the directory's in +`crates/directory/Cargo.toml`, the page's in `package.json`; CLAUDE.md has the release steps. + +## Traces and logs + +A request is one trace (#213). The page starts one for each tap and sends `traceparent` with its +requests (#190); Caddy's span continues it and passes it on to ipx; ipx's request span, its steps +and its calls to the directory follow, and the directory's request span joins it. Scans, +downloads, DeadAir's work, shares with the directory and the directory's refresher rounds are +traces of their own. All three log one JSON object a line, for Loki; ipx's lines carry their trace +id, and Caddy's request lines theirs. ipx also serves Prometheus metrics (`[web] metrics_bind`, +#194). The exporter and the `traceparent` handling both Rust services use are in +`crates/directory/src/otel.rs`. ## Modules @@ -25,6 +39,12 @@ The version is the workspace's, in the root `Cargo.toml`. | `crates/ipx/src/auth.rs` | Argon2id hashing, session tokens, header names | — | | `crates/ipx/src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — | | `crates/ipx/src/logbuf.rs` | Ring buffer behind the UI's Log view | — | +| `crates/ipx/src/directory.rs` | Asking the directory service, and sharing this server's public feeds with it | — | +| `crates/ipx/src/metrics.rs` | `/metrics` for Prometheus | — | +| `crates/directory/src/lib.rs` | The directory's HTTP API, its token check, the refresher | — | +| `crates/directory/src/store.rs` | The directory's database: feeds, snapshots, blocked | — | +| `crates/directory/src/config.rs` | The directory's config file: its tokens and their access | — | +| `crates/directory/src/otel.rs` | Traces for both Rust services: the exporter, `traceparent` in and out | — | | `web/index.html` | The app's markup | — | | `web/admin.html` | The admin page's markup: server settings, accounts, the log. Sent to admins only | — | | `web/app.css` | The stylesheet both pages share | — | diff --git a/docs/cli.md b/docs/cli.md index 3f692ab..7e99215 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -13,12 +13,12 @@ processes must never download the same thing. `--local` forces the work to happe | `ipx list` | Subscriptions and their state | | `ipx status` | Counts: feeds, pending, downloaded | | `ipx fetch [FEED] [--force]` | Scan everything, or one feed. `--force` ignores the TTL | -| `ipx add [--folder X] [--keywords a,b] [--list] [--category C]` | Subscribe; the id comes from the feed title. `--list` puts it in the Directory for anyone to subscribe to instead, and keeps it there when its last subscriber leaves; `--category` files it under one of the Directory's categories. Run for a feed already in the catalogue, these list it or set its category | +| `ipx add [--folder X] [--keywords a,b] [--list] [--category C]` | Subscribe; the id comes from the feed title. `--list` subscribes no one but keeps it in the catalogue, and so in what this server shares with the directory, when its last subscriber leaves; `--category` is the category the directory files it under when the feed names none. Run for a feed already in the catalogue, these list it or set its category | | `ipx rm ` | Unsubscribe; downloads and history are kept | | `ipx import ` / `ipx export ` | Move subscriptions in or out. Import subscribes the first admin, as the shared web token does; in the web UI it subscribes whoever is signed in | | `ipx reap [--dry-run]` | Run retention now | | `ipx user ` | Accounts for the web UI | -| `ipx daemon [--web ADDR]` | Scheduler, control socket and web UI | +| `ipx daemon [--web ADDR]` | Scheduler, control socket and the HTTP API (the page is ipx-web's) | `add`, `rm` and `import` change the catalogue in a process of their own. With a daemon running they then tell it to read the catalogue again; without that it kept its own copy and wrote it back at @@ -71,7 +71,7 @@ stops the next scan fetching it again. ## The daemon ```sh -ipx daemon # scheduler + socket + web UI +ipx daemon # scheduler + socket + HTTP API ipx daemon --web 0.0.0.0:8099 # override the configured bind for one run ``` diff --git a/docs/configuration.md b/docs/configuration.md index b65ef01..7573f1b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -140,7 +140,7 @@ folder = "Accidental Tech Podcast" # default: the feed title schedule = "every 6h" # overrides [general] for this feed media_types = ["audio"] # overrides [general] for this feed category = "Technology" # the Directory's, if the feed names none -listed = true # in the Directory with no subscribers +listed = true # shared with the directory with no subscribers username = "ray" # HTTP basic auth password_env = "IPX_ATP_PASS" # preferred over a literal `password` ``` @@ -150,9 +150,14 @@ With more than one account, **`keywords`, `auto_download`, `allow_explicit` and config.toml are the fallback for a feed nobody has claimed. The keys above describe the feed itself and are the same for everyone. See [users.md](users.md). -**The Directory** lists every feed in the catalogue, whether or not anyone subscribes to it yet. -To put one there for others to find, `ipx add --list --category News `: it subscribes no one, -and a listed feed stays when its last subscriber leaves, where another is dropped. A feed nobody +**The Directory** is the directory service's ([directory.md](directory.md)), which an admin sets +up in Admin, Directory: its address, this server's token and whether to share. They are server +settings, kept in the database (`directory_url`, `directory_token`, `directory_share`), not here. +With sharing on, this server sends it every feed in the catalogue that is not private, with how +many people here subscribe, and the `category` above for a feed that names none. To offer one with +nobody subscribed, `ipx add --list --category News `: it subscribes no one, and a listed feed +stays in the catalogue, and so in what is shared, when its last subscriber leaves, where another +is dropped. A feed nobody subscribes to is checked once a day. One that is dead (failing for 30 days) or quiet (nothing new in a year), with nobody subscribed and no file on disk, is removed from the catalogue. @@ -172,3 +177,17 @@ and they are re-derived on every scan. Editing one in the UI promotes it to a ca | `IPX_UI_LOG` | What the in-process log buffer captures for the UI's Log view | | `OTEL_EXPORTER_OTLP_ENDPOINT` | An OTLP/HTTP collector, such as Tempo at `http://host:4318`: the daemon sends it traces of scans, downloads and web requests. The other `OTEL_EXPORTER_OTLP_*` variables apply too | | `http_proxy` / `https_proxy` | Honoured for feed and enclosure fetches | + +## ipx-web + +The page's container reads only its environment (`web/Caddyfile`): + +| Variable | Effect | +|---|---| +| `IPX_UPSTREAM` | Where ipx is, `host:port`; `ipx:8099` by default | +| `WEB_BIND` | Where to listen; `:8080` by default | +| `WEB_ROOT` | The built page; `/srv` in the image | +| `TUNNEL_PEER` | The one address whose `Cf-Access-*` headers are passed on to ipx: where cloudflared's requests come from. Unset passes none. See [sso.md](sso.md) | +| `WEB_TRACING` | `on` sends a span for each request to the collector the `OTEL_*` variables name (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`, `OTEL_SERVICE_NAME`), continuing the page's trace into ipx. Off by default | + +ipx-directory's settings are in [directory.md](directory.md). diff --git a/docs/directory.md b/docs/directory.md index a81a3be..c5ac808 100644 --- a/docs/directory.md +++ b/docs/directory.md @@ -71,6 +71,13 @@ token_sha256 = "…" `DIRECTORY_LOG_FORMAT=json` logs one JSON object a line, for Loki, as ipx's `IPX_LOG_FORMAT` does. Each request is a line with its `method`, `path`, `status`, `duration_ms` and the `token`'s name. +With `OTEL_EXPORTER_OTLP_ENDPOINT` set (`http://host:4318`, OTLP over HTTP), it sends traces: a +span a request, a child of the caller's when the request carries a `traceparent`, as ipx's do, and +a trace a refresher round with a span a feed (#213). `OTEL_RESOURCE_ATTRIBUTES` tags them, as +production does with `deployment.environment.name=production`. Its version is in +`crates/directory/Cargo.toml`, logged at start and answered by `ipx-directory --version`; its +changelog is `crates/directory/CHANGELOG.md`. + ## Running one ```yaml diff --git a/docs/sso.md b/docs/sso.md index 7922250..5a1fd24 100644 --- a/docs/sso.md +++ b/docs/sso.md @@ -18,7 +18,7 @@ real setup and pointed at the wrong address. ``` browser ─► Cloudflare Access, app "ipodderx" ─── sign in ───► Authentik (OpenID Connect) ─► tunnel "rays-unraid" (the cloudflared container on Tower) - ─► http://192.168.1.130:8099 ─► ipx + ─► http://192.168.1.130:8099, ipx-web (Caddy) ─► ipx ``` Authentik is not in the request path. It is the identity provider Cloudflare Access asks. Access @@ -38,7 +38,7 @@ request it forwards through the tunnel, and ipx signs that person in. enabled = true bind = "0.0.0.0:8099" trusted_header = "Cf-Access-Authenticated-User-Email" -trusted_proxies = ["127.0.0.1", "::1", "192.168.64.1"] +trusted_proxies = ["127.0.0.1", "::1", "192.168.64.10"] # ipx-web; see "Behind ipx-web" access_team = "rays-sdf1.cloudflareaccess.com" access_aud = "8bfe73dfbc8c548d1cb5dc11c6db6887bcaf4f5144840396f83a620a140e1c4f" auto_create_users = true @@ -67,7 +67,7 @@ Cloudflare and Authentik were already right. Three things on the ipx side were n ### The address to trust, and why it is 192.168.64.1 -`cloudflared` runs in its own container and reaches ipx through the host's published port. Docker +`cloudflared` runs in its own container and reaches ipx-web (before #203, ipx) through the host's published port. Docker (iptables firewall backend) masquerades traffic between its bridge networks, so the tunnel's requests arrive from the **gateway of ipx's own network**, `ipx_default`: @@ -81,8 +81,8 @@ lists them in hex.) The `ipx` project's compose file pins that network's subnet to `192.168.64.0/20`, so its gateway stays `192.168.64.1` when the network is recreated. It was `192.168.16.1`, `content_default`'s, while -ipx was part of the `content` project. If the subnet is ever changed, update `trusted_proxies` to -match. +ipx was part of the `content` project. If the subnet is ever changed, update `TUNNEL_PEER` in the +compose file to match (before #203, `trusted_proxies`). ### Behind ipx-web (#203) @@ -185,10 +185,12 @@ itself, with a **Proxy Provider** and an **outpost** that adds `X-authentik-user ## How this is secured -**The header is only believed from `trusted_proxies`.** Every other source is ignored, and the -request falls through to a session cookie or the shared token. That is the whole security boundary. +**The header is only passed on from `TUNNEL_PEER`, and only believed from `trusted_proxies`.** +Caddy strips `Cf-Access-*` from every request that does not come from `TUNNEL_PEER`, and ipx +ignores the header from anyone but Caddy, its one trusted proxy; such a request falls through to a +session cookie or the shared token. That is the whole security boundary. -With the tunnel reaching ipx through the host's port, `192.168.64.1` means **any container on Tower +With the tunnel reaching ipx-web through the host's port, `192.168.64.1` means **any container on Tower that connects to `192.168.1.130:8099`**, not only `cloudflared`. Machines on the LAN, and Tower itself, arrive under their own addresses and cannot set the header; the checks above show both sides. Never list a LAN address or range: anyone there could then send diff --git a/docs/users.md b/docs/users.md index c973eba..c593aa4 100644 --- a/docs/users.md +++ b/docs/users.md @@ -70,8 +70,10 @@ list with their own read state. Unsubscribing removes it from their list alone; subscriber leaves does the feed stop being scanned, and even then its files and history stay, so re-subscribing does not pull the back catalogue again. -**Directory** sits at the top of the feed list, above your own feeds. It opens on the ten feeds -with the most subscribers on this server, you included, ranked, then the categories, each a tile +**Directory** sits at the top of the feed list, above your own feeds. It shows the directory +service an admin has set up ([directory.md](directory.md)), which iPX servers share, and says so +when there is none. It opens on the ten feeds with the most subscribers on every server sharing +with it, you included, ranked, then the categories, each a tile showing a few of its shows' covers. A category has a page of its own: its most subscribed feeds, then all of them as a grid of cover art, with its subcategories along the top (Tech News under News, Video Games under Leisure) and a choice of order. See all lists every feed, and the search @@ -81,7 +83,7 @@ Blogs (the rest). A show's category is the one it gives itself in iTunes, or for none, one an admin or iPX picked for it. Your own feeds are marked Subscribed. It shows a title, artwork and a count, never a URL or who reads it. An OPML subscription is listed as the feeds inside it, one by one, and never the OPML itself, so you can take just the shows you -want. Anything that looks private is left out: a login configured for the feed, credentials in its URL, +want. Anything private is left out: a feed someone marked Private in its settings, a login configured for the feed, credentials in its URL, or a key such as `auth=` or `token=` in the query, or a feed from a paid-feed service such as Patreon or Supercast, which put the key in the path, and any feed inside an OPML that looks private itself. Those are someone's paid subscriptions, and listing them would let anyone here read what @@ -99,7 +101,7 @@ removed there, so there is always someone who can manage the rest. ## Admin The first account is an admin. An admin can change global settings (scanning interval, quota, -retention, media types, download folder), a feed's URL, folder and schedule, and who has an account +retention, media types, download folder, the directory), a feed's URL, folder and schedule, and who has an account and who else is an admin, and read the log, which names everyone's feeds and sign-ins. Everyone else gets the Settings and Log buttons hidden and a `403` if they ask anyway.