ipx/CLAUDE.md

# Working on ipx

Notes for whoever picks this up next. Read [docs/architecture.md](docs/architecture.md) for how the
thing is built; this file is about working on it without repeating mistakes that have already been
made here.

## Where things are

Production is the `iPX` container on Tower (192.168.1.130), the `ipx` service of the
Arcane project `ipx`: `/mnt/fast/arcane/projects/ipx/compose.yaml`. That file is what runs, and
the containers of the split (#207) join it there. `docker-compose.yml` in this repo is a sample for
others, ipx alone on SQLite; editing it changes nothing in production. Until 2026-10-12 ipx was a
service of the `content` project, beside audiobookshelf, immich and Plex (#208).

| | Host | In the container |
|---|---|---|
| Image | `192.168.1.130:5000/ipx:latest` | |
| Config | `/mnt/fast/appdata/ipx/config.toml`: bind address, token, trusted proxies, torrent, paths. The feeds and server settings are in the database | `/config/config.toml` |
| Database | Postgres 18, database `ipodderx`, login `ipodderx`, on the `postgres` container of the Arcane project `databases` (`192.168.1.130:5433`). The URL is in `ipx.env` beside the compose file (`/mnt/fast/arcane/projects/ipx/ipx.env`, mode 600), passed to the container as `IPX_DATABASE_URL`. A relative `env_file`: Arcane runs compose in its own container, where `/mnt/fast/appdata` does not exist | |
| Old database | `/mnt/user/ipx/state.db`, SQLite, used until the move to Postgres on 2026-09-18 and kept for rollback | `/data/state.db` |
| Downloads | `/mnt/user/ipx/downloads` | `/downloads` |
| Directory | `ipx-directory` in the same project, `192.168.1.130:8100`, public at `directory.ipxreader.app` through the tunnel with no Access in front: its pages are anyone's to browse, read only (#215), and its API (`/v1`) takes a token for everything. Until 2026-10-12 it was `ipx-dir.sdf1.net` (#214). Config `/mnt/fast/appdata/ipx-directory/directory.toml` (token hashes; Tower's own token in `tower.token` beside it, and entered in ipx's Admin, Directory, which keeps it in the database's `settings`, with the address ipx reaches it at: `http://ipx-directory:8100`, inside Docker, not the public address, which went out through Cloudflare and the tunnel, 170-340 ms a call where the directory answers in 16 ms), database `ipx_directory` on the same Postgres, URL in `directory.env` beside the compose file. See [docs/directory.md](docs/directory.md) | `/config/directory.toml` |
| Shared episodes | `ipx-play` in the same project, `192.168.1.130:8102`, public at `play.ipxreader.app` through the tunnel with no Access in front: the page a shared episode opens, anyone's to see (#447). `/v1` takes a token. Config `/mnt/fast/appdata/ipx-play/play.toml` (token hashes; Tower's token entered in ipx's Admin, Directory, Sharing episodes, with the address `http://ipx-play:8102`), database `ipx_play` on the same Postgres, URL in `play.env` beside the compose file. Metrics for Prometheus on `192.168.1.130:9191`, LAN only, as `job="ipx-play"` (#458). See [docs/play.md](docs/play.md) | `/config/play.toml` |
| Home page | `ipxreader.app` (and `www`): a static page from `site/`, on the Cloudflare Pages project `ipxreader`, not on Tower. See below | |
| Web UI | `ipx-web` (Caddy) in the same project, at `192.168.1.130:8099`, also `ipodderx.sdf1.net` via a Cloudflare tunnel. It serves the page and passes `/api` and `/media` to ipx, whose own port is published nowhere; its address on `ipx_default` is pinned to `192.168.64.10` (#203) | ipx `0.0.0.0:8099`, Caddy `:8080` |
| Sign-in via the tunnel | Cloudflare Access app `ipodderx`, with Authentik as its identity provider; see [docs/sso.md](docs/sso.md) | Caddy passes `Cf-Access-*` only from `192.168.64.1`, the `ipx_default` gateway cloudflared arrives from (`TUNNEL_PEER`; the subnet is pinned in the compose file); ipx trusts them only from Caddy, `192.168.64.10`, and only with a valid `Cf-Access-Jwt-Assertion` (`access_team`, `access_aud`) |

Work to do lives in the Gitea issues at https://git.sdf1.net/rays/ipx/issues, not in a
`TODO.md`. `/src/tea` is logged in: `/src/tea issues list --login git.sdf1.net --repo rays/ipx`.

**Every problem found gets an issue, before it is fixed.** A bug, a gap or a security hole turned
up along the way (reading logs, a review, a test that fails for another reason) is filed as soon
as it is found, even if it is fixed a minute later, so there is a record of what was wrong and
when. Name the issue in the commit that fixes it (`(#40)` in the subject). Once the fix is on
`main` and pushed, comment on the issue with what changed and the commit, then close it:

```sh
R="--login git.sdf1.net --repo rays/ipx"
t() { timeout 30 /src/tea "$@" < /dev/null; }
t issues create $R -t "Web token printed in the startup log" -L bug -d "What is wrong, where, how it was found."
t comment $R 40 "Fixed in 3625cf4: the startup line says where the token is kept, not what it is. Deployed in 0.8.3."
t issues close $R 40
```

**Give tea a closed stdin and a timeout**, as `t` does. Without a terminal, `tea comment` waits on
stdin and never exits; a script closing seven issues sat hung for a day on the second (#39).

Labels: `bug` for something wrong, `enhancement` for something missing. A problem found and left
for later stays open, and that is how it gets picked up again.

Deploying a change is: build and push the image, then pull it and recreate the container.
**Build only with `scripts/build-image.sh`, never a bare `docker buildx build`.** On 2026-11-04
a plain build of ipx, its dependency layer invalidated by a `Cargo.toml` change, rebuilt
everything in release on all twelve threads with 5 GB free; Tower thrashed for three hours with
every core pegged and had to be rebooted. The script refuses to start under 9 GB available or
above load 12, and builds in the BuildKit container `ipx-capped` (made on first use), held to 6
GB of memory and six threads by its cgroup: a build that outgrows that is killed, and production
keeps running. If one is killed, say so and wait for the user to decide; do not raise the caps or go
around the script. Run the build in the foreground on its own, never chained after a cargo
command, and look at `MemAvailable` and the load again when it ends.

```sh
sh scripts/build-image.sh ipx
docker compose -f /mnt/fast/arcane/projects/ipx/compose.yaml pull ipx
docker compose -f /mnt/fast/arcane/projects/ipx/compose.yaml up -d ipx
docker logs --tail 20 iPX
```

The directory is the same, with its own image:

```sh
sh scripts/build-image.sh directory
docker compose -f /mnt/fast/arcane/projects/ipx/compose.yaml pull ipx-directory
docker compose -f /mnt/fast/arcane/projects/ipx/compose.yaml up -d ipx-directory
```

ipx-play the same:

```sh
sh scripts/build-image.sh play
docker compose -f /mnt/fast/arcane/projects/ipx/compose.yaml pull ipx-play
docker compose -f /mnt/fast/arcane/projects/ipx/compose.yaml up -d ipx-play
```

And the page, `ipx-web`, the same again; a change to `web/` alone needs only this one:

```sh
sh scripts/build-image.sh web
docker compose -f /mnt/fast/arcane/projects/ipx/compose.yaml pull ipx-web
docker compose -f /mnt/fast/arcane/projects/ipx/compose.yaml up -d ipx-web
```

The home page (`site/`, #216) is not on Tower. `sh site/build.sh` puts it in `site/dist` with the
typeface, logo and icon from `web/`, and it goes to the Cloudflare Pages project `ipxreader` with
wrangler, which takes `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` from `/src/.envrc` in
place of a login:

```sh
sh site/build.sh
. /src/.envrc && npx -y wrangler@4.150.0 pages deploy site/dist --project-name ipxreader --branch main --commit-hash "$(git rev-parse HEAD)"
curl -sL https://ipxreader.app/ | head -5
```

`--branch main` makes it the production deployment that ipxreader.app serves; another branch is a
preview at its own `*.ipxreader.pages.dev` address. The token is an account API token with
Cloudflare Pages: Edit on this account and nothing else (dashboard, My Profile, API Tokens). If it
is missing or expired, ask the owner for one rather than working around it. Its screenshots (`site/shots`) come from a scratch iPX on public feeds, never
production's, which would show whose subscriptions are whose.

Testing sign-in from Tower itself proves nothing about the tunnel: Tower's own requests reach the
published port as `192.168.1.130`, and Caddy rightly strips their `Cf-Access-*`. Send them from a
container on another network, as cloudflared's are (#209).

The three containers resolve names through Cloudflare (`dns: [1.1.1.1, 1.0.0.1]` in the compose
file), not the router: the LAN's DNS is NextDNS, which answers for a domain it filters with its own
block page, and ipx then logged a feed as failing with `invalid peer certificate: UnknownIssuer`
(tomrenner.com, 2026-10-17). Docker's own DNS still answers for the services' names first.

**Name the service.** A bare `up -d` recreates every container in the project, which once the
split lands means the frontend and directory too. Run `pull` before `up`, because `up` reuses whatever `latest` the host already has.

**Base images come from Google's mirror** (`mirror.gcr.io`, set in `scripts/buildkitd.toml`),
not Docker Hub, which rate-limits this host with `429 Too Many Requests`. The capped builder
cannot see images the Docker daemon has stored, so pulling and tagging them locally does nothing
for it. If `buildkitd.toml` changes, remove the builder (`docker buildx rm ipx-capped`) so the
next build makes it again with the new config. The builder also keeps the compiled crates
between builds (cache mounts, #438), so a `Cargo.lock` change compiles only what changed; removing
it throws them away, and the next build of each image compiles all of them again.

The containers have no Docker healthcheck (#218): Docker runs one with `exec`, which on Tower
fails whenever root is logged out, because elogind removes `/run/user/0`, where containerd puts
each exec's state. Whether ipx is well is Prometheus's question (`up`, the worker's job and time)
and `ipxd status` against the control socket, which says the daemon answers and can read its
database. The socket answers `status` itself instead of queuing it behind the worker's current
job, so a worker stuck on one job still answers it. The containers restart on their own after a
reboot.

Before the container, ipx ran by hand in code-server, with its files in `/config/.config/ipx/` and
`/config/.local/share/ipx/`. Those are still there and the container does not read them. If you run
a daemon by hand for testing, **stop it by its own PID**: start it with `& echo $! > pid` and
`kill $(cat pid)`. Never `pkill -x ipx`: Tower sees the container's processes, so it kills
production's daemon as well (issue #38). Never `pkill -f ipx` either: `-f` matches the shell
running the command and kills the session (exit 144), which has happened more than once.

## The iOS app

`ios/` is the page in a `WKWebView` with the phone's own player behind `web/src/native.ts` (#418);
see [ios/README.md](ios/README.md). It builds only on a Mac with Xcode, from `ios/project.yml`
through XcodeGen: nothing here compiles or checks the Swift, so say so when handing over a change
to it. A change to the bridge's messages is a change to `native.ts`, `tests/native-bridge.ts` and
`ios/Sources/Bridge.swift` together, and the page deploys first.

## Before you touch the page

There are three pages: the app (`web/index.html`), the admin page (`web/admin.html`, sent to
admins only) and sign-in (`web/login.html`). The app and admin pages share one stylesheet,
`web/app.css`, and their script is TypeScript in `web/src/`; `web/build.mts` lists which files
make up each page's script. The page is its own image, **`ipx-web`** (#203): `web/build.mts` uses
swc to strip the types and minify each script (`app.js`, `admin.js`, `login.js`), minifies the
pages, and puts them with the icons and typeface in `web/dist`, which Caddy serves
(`web/Caddyfile`, `Dockerfile.web`), passing `/api` and `/media` to ipx on the same origin. ipx
serves no page. The page loads its script as `/app.js?v=<hash of its contents>`, and `/app.css`
and `/boot.js` the same way: the page is served `no-cache` and those `immutable`, so a browser
keeps them until a deploy changes them and their names. So **every page change needs ipx-web
rebuilt and deployed**, not ipx, and building needs node and `npm ci` run once.

Caddy asks ipx before it serves a page (`forward_auth`): `/api/page-check` sends a browser that is
not signed in to `/login` and anyone but an admin away from `/admin` (and refuses them
`/admin.js`); `/api/login-check` sends someone Cloudflare Access vouches for past the form; a page
asked for with `?token=` goes to `/api/signin`, which sets the cookie. The page is the same file
for everyone, so what ipx used to write into it per person is the page's own job now:
`web/boot.ts`, in `<head>`, puts the theme from its cookie and an admin's link from the last visit
on the page before the first frame, and `/api/me` brings the rest (version, the account's old
theme).

The files in `web/src` are not modules. They are one script split up, concatenated in the order
`web/build.mts` lists them, sharing one top-level scope as the single inline script did; a new
file goes into that list. Top-level names are kept as they are, because markup calls some by
name (`onclick="closeModal()"`) and the browser tests reach others through `page.evaluate`.

After any edit to it:

```sh
npm run -s typecheck
node tests/page-smoke.ts
node --test tests/page-unit.ts
```

**All the code here is TypeScript** (#392): the page, the service worker, the tests, the build, the
home page's generator and the load tests. Node runs a `.ts` file as it is, by stripping the types,
so it takes only TypeScript that erases (`erasableSyntaxOnly`: no enums, namespaces or constructor
parameter properties) and imports name their `.ts` files; k6 strips its own. What a browser loads
on its own is compiled by `web/build.mts` with swc (`boot.ts`, the worker). The directory's
`site.js` is compiled from `crates/directory/src/site.ts` and committed, since its build has no
Node: after editing `site.ts`, `node web/build.mts --directory-script`, which
`node tests/directory-script.ts` checks. `npm run -s typecheck` checks five programs, each with its
own globals: the page (`tsconfig.json`, which also writes the page's declarations to
`target/page-types` for the browser tests' `page.evaluate` calls), the worker, Node's
(`tsconfig.tools.json`), and k6's (`tsconfig.k6.json`).

The service worker that keeps iPX readable offline (#389) is TypeScript too, `web/sw/sw.ts`,
a script of its own built to `/sw.js`, and checked by `tsconfig.sw.json` against a worker's
globals, which cannot share a program with the page's. It goes to the network first for
everything, so online nothing is an old copy, but for an episode someone kept on the device
(#412): the page asks for it as `/media/...?offline`, and the worker answers from its
`ipx-episodes` cache, byte ranges and all. The browser tests block it
(`serviceWorkers: 'block'`), since it answers before `page.route` can; the offline tests allow it.

The first type-checks `web/src` in TypeScript's `strict` mode: every parameter and variable has a
type, what the API sends is typed in `web/src/types.ts`, which mirrors the Rust structs and changes
with them, and a value that may be null has to be checked before use. `$<T>` returns the element
type it is given (`$<HTMLInputElement>('#nurl')`) and assumes the element is there; one that may
not be is written `$(...)?.`, and the page check below proves the ones wired at load exist. The second
builds the page as shipped and runs its script against a stub DOM, checking every selector it
wires at load actually exists. That check exists because a patch once anchored on a deleted
function, `String.replace` silently matched nothing, and the whole UI died with a
`ReferenceError` while every server-side test passed. The third loads the same built script and
calls its helpers directly (#234): a pure function added to the page gets its check there.

Patching that file by guessing an anchor string has failed repeatedly. Read the exact block first
(`sed -n 'START,ENDp'`), match it verbatim, and assert the replacement happened rather than hoping.

## Tests

```sh
cargo fmt --all --check   # rustfmt, as rustfmt.toml says; the pre-commit hook refuses a commit without it
cargo clippy --workspace --all-targets -- -D warnings  # no warnings, ever
cargo deny check          # known vulnerabilities, licences, sources (deny.toml); reads Cargo.lock, builds nothing
cargo machete             # dependencies a crate lists and does not use; now and then, it can be wrong
cargo test                # parsing, filters, retention, schedules, SQL, per-user state
npm run -s typecheck      # the page, the service worker, the tests', build's and load tests' TypeScript
node tests/page-smoke.ts
node tests/directory-script.ts  # the directory's site.js is its site.ts as it stands
node --test tests/page-unit.ts  # the page's own functions, called directly, in a second
node tests/native-bridge.ts  # the page hands playback to a native shell
node tests/contrast.ts    # every theme's palette against WCAG AA
npx playwright test       # the browser tests, against a real daemon on fixture feeds
node tests/load/run.ts    # k6: many people at once against a scratch daemon with 1,500 feeds
sh scripts/coverage.sh    # which Rust lines the cargo tests reach, capped like a build (#390)
sh scripts/coverage.sh api  # which Rust lines the browser suite reaches through the API (#394)
npm run coverage:page     # which lines of web/src the browser tests reach (#390)
```

**Builds share Tower with everything else, production included.** On 2026-10-16 a build of the
directory followed in the same command by its tests' compile took Tower's load past 200 and the
session was stopped. So:

* One cargo command at a time: a build, then the tests, as separate commands, never chained or
  run in the background beside another, and never two crates' builds at once. Do not tune `-j`.
* Look at the load before and after each (`cat /proc/loadavg`). Above about 24, Tower's twelve
  threads twice over, wait for it to come down before starting another.
* Test only what the change is in. ipx uses ipx-play and ipx-directory, ipx-play uses
  ipx-directory, and all use ipx-feed, so:

  | changed | run |
  |---|---|
  | `web/` alone | the three page checks; `npx playwright test -g '<what changed>'`; no cargo |
  | `crates/ipx/` | `cargo test -p ipx`; the browser tests too if the API changed |
  | `crates/directory/` | `cargo test -p ipx-directory`, then `-p ipx-play`, then `-p ipx` |
  | `crates/play/` | `cargo test -p ipx-play`, then `-p ipx` |
  | `crates/feed/` | `cargo test -p ipx-feed`, then `-p ipx-directory`, `-p ipx-play`, then `-p ipx` |
  | `crates/tui/` | `cargo test -p ipx-tui`; it uses only the API, none of the other crates |
  | `Cargo.toml`, `Cargo.lock` | all five, one at a time |

  The pre-commit hook's clippy follows the same table. The whole browser suite once, at the end
  of a change to the page or the API; never `cargo test` for the workspace in one command.
* Builds and test binaries live under `target/`, a link to `/mnt/fast`; linking through
  `/mnt/user` (FUSE) once took the load to 248 on its own.
* **Memory is the tighter limit.** Tower has 31 GB, no swap, and the containers leave about 4-7
  GB. A build past that stalls everything on page faults: load 160 with the CPU idle and the
  disks quiet, many processes blocked (`node_procs_blocked`, `node_vmstat_pgmajfault`). On
  2026-10-18 linking a 431 MB test binary did it for 54 minutes. So `[profile.dev]` keeps only
  line tables and dependencies no debug info; keep it that way, and look at `MemAvailable` beside
  the load.

The Rust is formatted by rustfmt (`rustfmt.toml`) and kept free of clippy's warnings. Claude Code
formats each `.rs` file it edits (a hook in `.claude/settings.json`), and the pre-commit hook refuses
a commit touching Rust that is unformatted or has a clippy warning. A clone turns on the hook once: `git config core.hooksPath .githooks`; and `git config blame.ignoreRevsFile
.git-blame-ignore-revs` keeps blame past the commit that first formatted everything. A clippy
suggestion is not always the same code: on adopting it, `fold(false, |any, id| remove(id) || any)`
was offered as `any(...)`, which stops at the first and would have removed one feed of many. A lint
that is wrong for a line gets `#[allow(clippy::...)]` there with a comment saying why.

Things about the browser suite that have cost time:

* It starts **its own daemon and database** under `/tmp/ipx-ui-test`, wiped once per run. Playwright
  re-imports the config in every worker, so `prepare()` guards on `TEST_WORKER_INDEX` — without
  that guard a worker deleted the database out from under the running daemon, which then kept
  serving from the unlinked inode while everything else saw an empty file.
* Tests **share that daemon and run in order**. A test that opens an item marks it read and changes
  what later tests see. Write assertions that do not depend on what ran before, or normalise the
  state first.
* Fixture feeds must not share an enclosure URL, because `enclosures.url` is globally unique and
  whichever feed is scanned first claims it.
* `webServer` starts **before** `globalSetup`, which is why the fixture config is written at
  config-load time instead.

The load tests (`tests/load`, issue #133) are for what one browser cannot show: twenty-five people
browsing at once, a hundred players saving positions while a scan writes, fifty listeners
seeking through files, a flood of wrong passwords. Things about them:

* They need **k6**, which `/src/install.sh` installs from k6's own signed apt repository, and they
  build and run a **release** binary: Argon2 in a debug build takes about a second a sign-in,
  which would measure the build.
* `run.js` serves the feeds itself, generated, and starts **its own daemon** under
  `/tmp/ipx-load`, wiped each run, on ports 8793 and 8794, so it can run beside the browser suite.
  It stops that daemon by its PID. People sign in by the `X-Load-User` header, trusted from
  127.0.0.1, as production trusts Cloudflare Access's; only the sign-in test uses a password.
* Each threshold is a budget well above what the request takes now: it is there to catch a query
  that has started asking once per feed, or writers queueing on a lock, not a slow minute.
* `ipxd status` runs every second throughout, and a run fails if one takes 5s.
* One at a time: `node tests/load/run.ts signin`. All four take about six minutes.
* The daemon is on **SQLite** unless `--postgres`, which puts it in `IPX_TEST_DATABASE_URL`, in
  a schema of its own (`ipx_load`) dropped and made again each run. That URL, in `/src/.envrc`, is
  `ipodderx_test` on the `databases` project's server, production's, as `ipodderx`, taken from
  `ipx.env`; the harness refuses a database whose name does not end in `_test`. Postgres's
  numbers are what production would see, and a run loads the server production's database is on,
  so not while someone is using iPX in earnest. Each budget has a value for each database.
* The Rust tests run on Postgres with the same URL: `. /src/.envrc && cargo test`. Each test
  makes a schema `ipxt_<pid>_<n>`, and the next run drops those an earlier one left.

Coverage (#390, #394) says what the tests leave untested; read it before deciding a change is
covered. `sh scripts/coverage.sh` runs the cargo tests instrumented in the `ipx-coverage` image
(`sh scripts/build-image.sh coverage` makes it), in a container held to the builder's 6 GB and six
threads, one crate at a time, into `target/coverage/cargo` (`summary.txt`, `lcov.info`, `html/`).
`sh scripts/coverage.sh api` builds an instrumented ipx and ipx-directory there and runs the
browser suite here against them (`IPX_API_COVERAGE`), into `target/coverage/api`: what the API
and the daemon really run, which the cargo tests alone put at 17% for `web.rs`. The image is
Ubuntu 24.04, as code-server is, or the binaries it builds would not start here (glibc); the
suite stops its servers with SIGTERM in that mode, since an instrumented binary writes its profile
as it exits. The same run measures the page; `npm run coverage:page` does that alone, on a build of
the page with a source map to each `web/src` file (`IPX_COVERAGE=1`, `web/concat-map.mts`), into
`target/coverage/page`. The specs take `test` and `expect` from `tests/ui/coverage.ts`, not from
`@playwright/test`, so every test is counted, every page it opens (a second person's context
too), and each document before it unloads: the fixture stops it in `beforeunload` at a breakpoint
of its own to take what it ran, since a reload or sign-in otherwise throws that away (#407). The
unit tests and the native-bridge check are measured in Node first (`scripts/coverage-node.sh`);
the service worker is not.

Each run ends with `node scripts/coverage-metrics.ts`, which writes every suite's last numbers to
`/mnt/fast/appdata/node-exporter/textfile/ipx_coverage.prom`; node-exporter's textfile collector
(the monitoring project's compose file) serves them as `ipx_coverage_lines{suite,part,state}`, job
`node`, and the iPX dashboard's Coverage row draws them. A coverage run started by hand updates
Grafana; nothing runs one on its own.

Non-trivial logic leaves one runnable check behind. Pure functions (`merge_policy`, `pick`,
`matches_keywords`, `parse_interval`) are the easiest place to put it.

## Production's database

The owner allows `EXPLAIN (ANALYZE, BUFFERS)` of any query against production's Postgres, to see
where a query's time goes; nothing else without asking. Run it read only, so nothing can change:

```sh
U=$(grep -E '^IPX_DATABASE_URL=' /mnt/fast/arcane/projects/ipx/ipx.env | cut -d= -f2- | tr -d '"')
psql "$U" -X -v ON_ERROR_STOP=1 <<'SQL'
BEGIN READ ONLY;
EXPLAIN (ANALYZE, BUFFERS) SELECT ...;
ROLLBACK;
SQL
```

Never print the URL: it carries the password. Write the query as ipx sends it (`$1` parameters
become `PREPARE`/`EXECUTE`, or values), for a real account. A query fast here and slow in ipx's
trace is time outside the query: on 2026-10-15 the feed list's count took 5 ms in Postgres and 366
ms in ipx, the difference being new connections, 80-330 ms each to open (#241).

## Things that are true and easy to get wrong

* **`enclosures.url` is globally UNIQUE.** It is the dedupe key and the reason one file serves every
  subscriber. Two feeds publishing the same URL means only the first one scanned shows it.
* **Read state lives in `entry_state`, per user, and nowhere else.** `entries` had `read`, `flagged`
  and `position` columns from before accounts; two bugs came from queries still reading them
  (retention, and the entry pruner), and they were dropped in 0.5.
* **The catalogue and the server settings are in the database, not config.toml** (issue #18):
  tables `catalogue` (each feed's `config::Feed` as JSON) and `settings` (`general`:
  `config::Stored`). ipx still runs from one in-memory `Config`, config.toml for where things are
  and who gets in, the database for the rest (`assemble_config`); a change goes through
  `Ctx::store_cfg`, never a write to the file. The first start on a database without them imports
  config.toml's and trims the file, keeping `config.toml.pre-database`. A feed exists once;
  `subscriptions(user_id, feed_id)` says who wants it and with what settings. OPML children are
  derived and never in the catalogue.
* **Postgres connections ask for no notices** (`client_min_messages=warning`, `db::url_for`).
  Postgres sends one for every `CREATE ... IF NOT EXISTS` on something existing, sqlx logs each,
  and tracing-subscriber's per-layer filters then dropped the next line ipx logged.
* **One fetch serves everyone**, so scan policy is a union of subscribers' wants (`merge_policy`).
  Anyone wanting an item is enough to fetch it.
* **The UI hiding a control is not enforcement.** Admin-only actions check `user.is_admin` in the
  handler and return `403`.
* **A `tokio::select!` only races its branches at the point of selection.** A long download has to
  watch the shutdown channel itself; the daemon ignored SIGTERM for exactly this reason.
* Only one daemon per socket. Removing the socket file defeats the guard and you get two daemons
  fighting over the database, with the stale one still holding the port.
* **The Grafana dashboard reads the log's fields and the metrics' names** (`grafana/dashboard.py`).
  Production logs JSON (`IPX_LOG_FORMAT=json`); the events' `ev`, `feed`, `new`, `bytes`, `msg`
  (`log_event` in ipc.rs) and the metrics in metrics.rs (requests are
  `ipx_http_request_duration_seconds`, #364) are what the panels query. Rename one and its panels
  go blank without an error; regenerate the dashboard to match. Routine lines (a request that went
  well, a scan's end) are debug: the admin page's Log view has them, Loki does not.
* `/api/settings` answering `200` does **not** mean the daemon is well — the web server is a
  different task. `ipxd status` checks the control socket and the database; to see the worker
  getting through its jobs, watch for `scan complete` in the log.
* **The database goes through SeaORM, and the entities in `crates/ipx/src/entity.rs` are the schema.**
  `Db::open` creates any missing table or index from them (`create_missing`), on every `ipx`
  command, the healthcheck's `ipxd status` included, so it must never write when nothing is
  missing: SeaORM's experimental schema sync dropped and remade an index on every open, the
  write lock that took made `ipxd status` time out behind a busy daemon, and it was removed for
  it. A new column on an existing table needs its own `ALTER`; nothing adds one for you. Look for
  the column first and alter only when it is missing, as `feeds.description` does.
* **SQL written by hand in `db.rs` has to run on SQLite and Postgres both** (issue #18): `$1`
  parameters, bound only if used; `ON CONFLICT`, not `INSERT OR IGNORE`; yes/no columns tested
  as themselves (`NOT coalesce(s.read, false)`) and written as `true`/`false`, never compared to
  1; no `rowid`, `GLOB` or `UPDATE OR IGNORE`. `Args` in `db.rs` builds the parameters.

## House style

Comments explain **why**, not what. If a line looks odd, the comment says what went wrong without
it. No emoji, no exclamation marks, no "obviously". Prose in the UI and docs is plain English and
addressed to the person using it.

Every change gets one line under `## [Unreleased]` in the changelog of the service it changes, in
its [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) group: Added, Changed, Deprecated,
Removed, Fixed or Security. One changelog a service: [crates/ipx/CHANGELOG.md](crates/ipx/CHANGELOG.md)
(the daemon and its API; until 0.12.0 everything), [web/CHANGELOG.md](web/CHANGELOG.md) (the page
and ipx-web) [crates/directory/CHANGELOG.md](crates/directory/CHANGELOG.md) (ipx-directory),
[crates/play/CHANGELOG.md](crates/play/CHANGELOG.md) (ipx-play) and
[ios/CHANGELOG.md](ios/CHANGELOG.md) (the iOS app). A
change to more than one gets a line in each, each saying what that service does differently. The
monitoring (the Grafana dashboard, the prod-check skill) and the repository's own tooling are not
a service anyone uses and get no line. Say it the way someone using ipx would notice it. When there is more to
say, such as what was wrong before or what it cost to find out, it goes in the commit message's
body, where `git log` and `git blame` find it beside the change. (There was a long-form
`docs/history.md` until 0.7.0; it grew too large to be useful and was removed. It is in git.)

Each service has its own version and is released on its own, when its changelog has something to
release:

| service | version in | changelog | tag |
|---|---|---|---|
| ipx | `crates/ipx/Cargo.toml` | `crates/ipx/CHANGELOG.md` | `vX.Y.Z`, as every release before the split |
| ipx-directory | `crates/directory/Cargo.toml` | `crates/directory/CHANGELOG.md` | `directory-vX.Y.Z` |
| ipx-play | `crates/play/Cargo.toml` | `crates/play/CHANGELOG.md` | `play-vX.Y.Z` |
| ipx-web, the page | `package.json` | `web/CHANGELOG.md` | `web-vX.Y.Z` |

`crates/feed` is a library the two Rust services build in, not released (`0.0.0`); a change to it
is a line in the changelog of each service it changes. The page shows ipx's version and its own in
the logo's tooltip; the directory logs its own at start; each binary answers `--version`.

Cutting a release of one: first read its `[Unreleased]` against its last release, not line by
line, and rewrite it as what changed between the two. Lines go in as the work happens, so they
record the way there: a feature added and then reworked in the same release is one line saying
what it is now, under Added; something added and taken out again before the release, or a fix for
a bug no release ever had, is not there at all; and "instead of" and "no longer" compare with the
last release, not with last week. Issue numbers stay, all of them, on the line that survives. Then
rename `[Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD` and open a new empty `[Unreleased]` above it,
set the service's version to `X.Y.Z` (dropping `-dev`), tag the commit with the service's tag, and
update the compare links at the bottom of its changelog. Push the commit and the tag, build and
deploy the service's image, then `sh scripts/release.sh ipx|directory|web` makes the Gitea
release: that version's changelog section as its notes, and a tarball of what the image runs (the
binary, or the page's files with the Caddyfile) attached, checked to be that version. Then, in the
next commit, set its version
to the next patch with `-dev` (after 0.9.0, `0.9.1-dev`), so a build between releases says so
instead of claiming to be the last release. The release that follows can still be a minor or major
one; `-dev` only says the work comes after `X.Y.Z`. A service never released is at `0.1.0-dev`,
working towards its first release, `0.1.0`.

**Traces first, logs second.** Anything that does work (a service, a handler, a job, a call to
another service) gets spans, and passes `traceparent` on to whatever it calls, so one trace in
Tempo follows a request or a job across ipx-web, ipx and the directory. A trace shows where the
time went and what called what; a log line only that something happened. Log what a trace cannot
carry: events worth counting on the dashboard, warnings and errors. A new service or route without
traces is not finished.

Deliberate simplifications get a `ponytail:` comment naming the ceiling and the upgrade path, e.g.
`// ponytail: global connection mutex, move to a pool if feed count makes it contend`.

## Known gaps

* They are the open issues in Gitea, not a list here: a limitation known and left in place is an
  issue left open.

## Styleguide

* Best effort to adhere to the rust style guide found at /src/styleguide/rustguide.md

<!-- rtk-instructions v2 -->
# Command output

Command output here is condensed to save tokens, keeping every signal and
dropping costly noise. Treat it as the complete result: run commands
normally, and batch related commands into one call to avoid extra turns.
Truncated results state their recovery path in their own output. Re-run a
command as `rtk proxy <cmd>` only when its result is unusable: empty when
output was clearly expected, contradicting its exit code, or garbled.
<!-- /rtk-instructions -->