ipx/docs/architecture.md

# How it works

One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a Unix-socket
control server, and the web UI. Everything else is a CLI that either does the work itself or hands
it to a running daemon.

The repo is a Cargo workspace (#202): `crates/ipx` is the daemon and CLI, and `crates/feed`
(`ipx-feed`) fetches and parses feeds, shared with `crates/directory` (`ipx-directory`), the
directory of public podcasts that ipx servers share ([directory.md](directory.md)).
The version is the workspace's, in the root `Cargo.toml`.

## Modules

| File | Responsibility | What it replaced in the Python |
|---|---|---|
| `crates/ipx/src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
| `crates/ipx/src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
| `crates/ipx/src/db.rs` | Every query, through SeaORM; creates missing tables | `.ipxd` plists, `history.dat`, `qmcache.dat` |
| `crates/ipx/src/entity.rs` | The tables, as SeaORM entities: the schema | — |
| `crates/feed/src/lib.rs` | Conditional GET, RSS/Atom/OPML parsing, `url_looks_private` | `FeedData.__getFeed/__getEntries` |
| `crates/ipx/src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
| `crates/ipx/src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
| `crates/ipx/src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` |
| `crates/ipx/src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout |
| `crates/ipx/src/auth.rs` | Argon2id hashing, session tokens, header names | — |
| `crates/ipx/src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — |
| `crates/ipx/src/logbuf.rs` | Ring buffer behind the UI's Log view | — |
| `web/index.html` | The app's markup | — |
| `web/admin.html` | The admin page's markup: server settings, accounts, the log. Sent to admins only | — |
| `web/app.css` | The stylesheet both pages share | — |
| `web/src/*.ts` | The page's script, one scope split across files, type-checked by `npx tsc` | — |
| `web/build.mjs` | swc: strips the types into `app.js`/`admin.js`/`login.js`, named in the page by a hash of their contents, minifies, and writes the static site to `web/dist` | — |
| `web/boot.js` | In `<head>`: the theme from its cookie, and an admin's link, before the first frame | — |
| `web/Caddyfile` | `ipx-web`: serves `web/dist`, asks ipx before serving a page, passes `/api` and `/media` to ipx | — |

The page is a separate image, `ipx-web` (`Dockerfile.web`, #203), so **editing `web/` needs
ipx-web rebuilt**, not ipx, and building it needs node and `npm ci` run once. ipx serves the API
and the files, and no page.

## A scan

1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
   re-derived into the database each scan, never written to config.toml. A Patreon creator link
   (a token, no `show=`) with more than one show is treated the same way, before any fetch: its
   shows come from Patreon's web API and each becomes a derived feed.
4. Record entries: the new ones, and those the feed now gives a different title, text, artwork,
   length or number, compared with what is stored for the items it lists (#141). A corrected item
   keeps its read state.
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
   original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
   a reason. What a filter skipped is judged again every scan, so a change of settings brings it
   back. A feed in a group takes your settings on the group for anything you have not set on it.
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
   the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
   or an error page — and is deleted.

## Data model

```
feeds       id, url, title, image, category, description, etag, last_modified,
            last_checked, ttl_mins, last_error, orphaned, group_id, managed
entries     feed_id, guid, title, link, published, description, first_seen,
            image, duration, episode, season            PK (feed_id, guid)
enclosures  id, feed_id, guid, url UNIQUE, mime, length, path, state,
            bytes_done, downloaded_at, last_error
users       id, name, pass_hash, is_admin, created, last_login
sessions    token, user_id, seen
subscriptions  user_id, feed_id, keywords, auto_download, allow_explicit,
               max_new_per_check                         PK (user_id, feed_id)
entry_state    user_id, feed_id, guid, read, flagged, position
                                                         PK (user_id, feed_id, guid)
```

Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
before accounts; two bugs came from queries still reading them, and they were dropped in 0.5.

Schema changes: the tables are the entities in `crates/ipx/src/entity.rs`, and `Db::open` creates whatever
table or index a database is missing from them (`db::create_missing`), with `IF NOT EXISTS`. It
never alters a table that exists, so a new column on one needs its own `ALTER` in
`create_missing`, looked for first and run only when it is missing, as `feeds.description`'s is,
or `sea-orm-migration` once there are several. `Db::memory()` builds its
database the same way, so the tests run on the schema production gets. A database from before
0.7 takes its last columns from the old `migrate()`, so it upgrades through a 0.7 release first.

## Control socket

Newline-delimited JSON, both directions, over `$XDG_RUNTIME_DIR/ipx.sock`.

```sh
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
{"ev":"feed_start","feed":"atp"}
{"ev":"progress","feed":"atp","enclosure":42,"file":"ep1.mp3","done":8192,"total":3000000}
{"ev":"download_done","feed":"atp","enclosure":42,"path":"…","bytes":3000000}
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
{"ev":"scan_done","feeds":1}
```

**Commands** — `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `download`
(`enclosure`), `status`, `reload` (read the catalogue and settings again from the database, which
`ipx add`, `rm` and `import` send after changing them in a process of their own; answered with
`status`).

**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
there. Commands run one at a time, in the order they arrive, except `status` and `reload`: the
socket answers them straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and
answers only the client that asked, since `status` would end any other client's session.

Progress carries the enclosure id, without which a UI cannot tell one download from another and
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
so a client attached to a busy daemon also sees that daemon's other work.

Inside the process the same events go over a `tokio::broadcast`; commands arrive on an `mpsc` and
are handled by a single worker, so nothing races over the same download. Shutdown is a `watch`
channel raced *inside* each job — `tokio::select!` only races branches at the point of selection,
so a long download had to be able to notice the signal itself.

## HTTP API

Everything below `/api` needs a signed-in user; the browser gets a redirect to `/login`, anything
else a `401`. A feed's items and files (its entries, `download-latest`, `/api/enclosures/{id}` and
`/media/{id}`) answer only someone who subscribes to the feed; anyone else gets a `404`.

| Route | |
|---|---|
| `GET /api/page-check`, `GET /api/login-check`, `GET /api/signin` | for Caddy (#203): whether the visitor may have the page in `X-Forwarded-Uri` (`204`), or where to send them; signing in with `?token=` |
| `POST /api/login`, `POST /api/logout`, `GET /api/me` | sign-in; `/api/me` also gives the version and the account's theme from before |
| `GET /api/feeds`, `POST /api/feeds` | your subscriptions; subscribe |
| `PATCH /api/feeds/{id}`, `DELETE /api/feeds/{id}` | your settings or (admin) the feed's; unsubscribe |
| `GET /api/feeds/{id}/entries` | paged, filtered, searchable, sortable (`sort` = kept, title, feed, type, size or published; `dir` = asc or desc) |
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, kept, position |
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
| `POST /api/fetch` | |
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
| `GET /api/directory/{id}` | a listed feed's description, why its last check failed when it did (in words, never its error, which can name its address), and latest twenty items, for its page before you subscribe: title, link, date, length and text, never a file or its address, and only for a feed the directory lists |
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with an OPML's feeds in place of the OPML and everyone counted (id, title, art, count, whether it is yours, the feed's iTunes category, whether it carries audio or video; never a URL, never a private feed); subscribe by id |
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |
| `GET /api/events` | SSE, the same broadcast the socket carries |
| `GET /api/logs` | admin-only; the ring buffer, with a sequence cursor |
| `GET /media/{id}` | the file, with Range support so seeking works |

Show notes are feed-supplied HTML from an untrusted source, sanitized with `ammonia` server-side
before they reach the page.

## Testing

```sh
cargo test                # parsing, filters, retention, schedules, SQL, per-user isolation
node tests/page-smoke.js  # the page script loads and every selector it wires at load exists
node tests/native-bridge.js  # the page hands playback to a native shell
npx playwright test       # a real browser against a real daemon on fixture feeds
node tests/load/run.js    # k6: many people at once against a daemon with 1,500 generated feeds
```

The Rust tests cannot see a wrong selector, a handler that runs and does nothing, or a page that
renders empty — which is what has actually reached users. Each Playwright case maps to a bug that
did.

The suite starts its own daemon and database under `/tmp/ipx-ui-test`, wiped once per run. Tests
share that daemon and run in order, so a test that marks something read changes what later tests
see — make assertions that do not depend on earlier ones.