ipodderx-rs/docs/architecture.md

# How it works

One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a Unix-socket
control server, and the web UI. Everything else is a CLI that either does the work itself or hands
it to a running daemon.

## Modules

| File | Responsibility | What it replaced in the Python |
|---|---|---|
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
| `src/db.rs` | SQLite schema, migrations, every query | `.ipxd` plists, `history.dat`, `qmcache.dat` |
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
| `src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` |
| `src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout |
| `src/auth.rs` | Argon2id hashing, session tokens, header names | — |
| `src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — |
| `src/logbuf.rs` | Ring buffer behind the UI's Log view | — |
| `web/index.html` | The app's markup | — |
| `web/admin.html` | The admin page's markup: server settings, accounts, the log. Sent to admins only | — |
| `web/app.css` | The stylesheet both pages share | — |
| `web/src/*.ts` | The page's script, one scope split across files, type-checked by `npx tsc` | — |
| `web/build.mjs` | swc: strips the types into `app.js`/`login.js`, named in the page by a hash of their contents, and minifies | — |
| `build.rs` | Runs `web/build.mjs` into `OUT_DIR`, where `web.rs` `include_str!`s the result | — |

The page is compiled in, so **editing `web/index.html` or `web/src` needs a rebuild**, and a
build needs node and `npm ci` run once.

## A scan

1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
   re-derived into the database each scan, never written to config.toml. A Patreon creator link
   (a token, no `show=`) with more than one show is treated the same way, before any fetch: its
   shows come from Patreon's web API and each becomes a derived feed.
4. Record entries. A changed title or description flips the item back to unread.
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
   original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
   a reason. What a filter skipped is judged again every scan, so a change of settings brings it
   back. A feed in a group takes your settings on the group for anything you have not set on it.
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
   the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
   or an error page — and is deleted.

## Data model

```
feeds       id, url, title, image, etag, last_modified, last_checked, ttl_mins,
            last_error, orphaned, group_id, managed
entries     feed_id, guid, title, link, published, description, first_seen,
            image, duration, episode, season            PK (feed_id, guid)
enclosures  id, feed_id, guid, url UNIQUE, mime, length, path, state,
            bytes_done, downloaded_at, last_error
users       id, name, pass_hash, is_admin, created, last_login
sessions    token, user_id, seen
subscriptions  user_id, feed_id, keywords, auto_download, allow_explicit,
               max_new_per_check                         PK (user_id, feed_id)
entry_state    user_id, feed_id, guid, read, flagged, position
                                                         PK (user_id, feed_id, guid)
```

Read state is `entry_state` alone. `entries` had `read`, `flagged` and `position` columns from
before accounts; two bugs came from queries still reading them, and `migrate()` drops them from an
older database.

Schema changes: add the table or column to `SCHEMA`. `CREATE TABLE IF NOT EXISTS` leaves a table
that already exists alone, so a new column on one also goes in `migrate()`'s `wanted` list, and a
retired one in its `retired` list; both are checked with `PRAGMA table_info`. Columns from before
0.3.0, the oldest version an upgrade may start from, need no entry. `Db::memory()` runs the same
path as `Db::open`, so a migration cannot pass the tests while missing in production.

## Control socket

Newline-delimited JSON, both directions, over `$XDG_RUNTIME_DIR/ipx.sock`.

```sh
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
{"ev":"feed_start","feed":"atp"}
{"ev":"progress","feed":"atp","enclosure":42,"file":"ep1.mp3","done":8192,"total":3000000}
{"ev":"download_done","feed":"atp","enclosure":42,"path":"…","bytes":3000000}
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
{"ev":"scan_done","feeds":1}
```

**Commands** — `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `download`
(`enclosure`), `status`.

**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
there. Commands run one at a time, in the order they arrive, except `status`: the socket answers it
straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and
answers only the client that asked, since `status` would end any other client's session.

Progress carries the enclosure id, without which a UI cannot tell one download from another and
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
so a client attached to a busy daemon also sees that daemon's other work.

Inside the process the same events go over a `tokio::broadcast`; commands arrive on an `mpsc` and
are handled by a single worker, so nothing races over the same download. Shutdown is a `watch`
channel raced *inside* each job — `tokio::select!` only races branches at the point of selection,
so a long download had to be able to notice the signal itself.

## HTTP API

Everything below `/api` needs a signed-in user; the browser gets a redirect to `/login`, anything
else a `401`.

| Route | |
|---|---|
| `GET /` | the app |
| `GET /login`, `POST /api/login`, `POST /api/logout`, `GET /api/me` | sign-in |
| `GET /api/feeds`, `POST /api/feeds` | your subscriptions; subscribe |
| `PATCH /api/feeds/{id}`, `DELETE /api/feeds/{id}` | your settings or (admin) the feed's; unsubscribe |
| `GET /api/feeds/{id}/entries` | paged, filtered, searchable, sortable (`sort` = kept, title, feed, type, size or published; `dir` = asc or desc) |
| `GET /api/entries` | the same, across every feed you subscribe to (All Subscriptions) |
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
| `POST /api/read-all` | everything read in every feed you subscribe to (All Subscriptions) |
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, kept, position |
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
| `POST /api/fetch` | |
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
| `GET /api/popular`, `GET /api/directory`, `POST /api/popular/{id}` | the ten most subscribed feeds, and every listable feed A to Z, with an OPML's feeds in place of the OPML and everyone counted (id, title, art, count, whether it is yours, the feed's iTunes category, whether it carries audio or video; never a URL, never a private feed); subscribe by id |
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |
| `GET /api/events` | SSE, the same broadcast the socket carries |
| `GET /api/logs` | admin-only; the ring buffer, with a sequence cursor |
| `GET /media/{id}` | the file, with Range support so seeking works |

Show notes are feed-supplied HTML from an untrusted source, sanitized with `ammonia` server-side
before they reach the page.

## Testing

```sh
cargo test                # parsing, filters, retention, schedules, SQL, per-user isolation
node tests/page-smoke.js  # the page script loads and every selector it wires at load exists
npx playwright test       # a real browser against a real daemon on fixture feeds
```

The Rust tests cannot see a wrong selector, a handler that runs and does nothing, or a page that
renders empty — which is what has actually reached users. Each Playwright case maps to a bug that
did.

The suite starts its own daemon and database under `/tmp/ipx-ui-test`, wiped once per run. Tests
share that daemon and run in order, so a test that marks something read changes what later tests
see — make assertions that do not depend on earlier ones.