ipx/docs/architecture.md

How it works

Three services, each a container of its own with its own version and changelog (#207):

Service	What it is	Source	Image
ipx-web	Caddy: serves the page, a static site built from web/, and passes /api and /media to ipx on the same address. It asks ipx before it serves the app or the admin page (/api/page-check)	web/, web/Caddyfile	Dockerfile.web
ipx	The daemon and CLI, one binary. ipx daemon runs a scheduler, a Unix-socket control server and the HTTP API in one process; everything else is a CLI that either does the work itself or hands it to a running daemon	crates/ipx	Dockerfile
ipx-directory	The directory of public podcasts that ipx servers share. ipx asks it on its people's behalf; the page never talks to it (directory.md)	crates/directory	Dockerfile.directory

The Rust source is a Cargo workspace (#202); crates/feed (ipx-feed) fetches and parses feeds for both Rust services. Versions: ipx's in crates/ipx/Cargo.toml, the directory's in crates/directory/Cargo.toml, the page's in package.json; CLAUDE.md has the release steps.

Traces and logs

A request is one trace (#213). The page starts one for each tap and sends traceparent with its requests (#190); Caddy's span continues it and passes it on to ipx; ipx's request span, its steps and its calls to the directory follow, and the directory's request span joins it. Scans, downloads, DeadAir's work, shares with the directory and the directory's refresher rounds are traces of their own. All three log one JSON object a line, for Loki; ipx's lines carry their trace id, and Caddy's request lines theirs. ipx also serves Prometheus metrics ([web] metrics_bind, #194). The exporter and the traceparent handling both Rust services use are in crates/directory/src/otel.rs.

Modules

File	Responsibility	What it replaced in the Python
crates/ipx/src/main.rs	CLI, dispatch, scan loop, download policy	iPXAgent.py
crates/ipx/src/config.rs	TOML load/save, General/Feed/Web, intervals, slugs	iPXSettings.py, feeds.plist
crates/ipx/src/db.rs	Every query, through SeaORM; creates missing tables	.ipxd plists, history.dat, qmcache.dat
crates/ipx/src/entity.rs	The tables, as SeaORM entities: the schema	—
crates/feed/src/lib.rs	Conditional GET, RSS/Atom/OPML parsing, url_looks_private	FeedData.__getFeed/__getEntries
crates/ipx/src/download.rs	Streaming download, naming, type sniffing, placement	iPXDownloader.getFile
crates/ipx/src/torrent.rs	librqbit session, seeding limits, stall abort	vendored BitTorrent 4.2.1
crates/ipx/src/retention.rs	Quota and age sweeps	iPXQuotaManager.py
crates/ipx/src/ipc.rs	Event and command types, the socket server	printMSG on stdout
crates/ipx/src/auth.rs	Argon2id hashing, session tokens, header names	—
crates/ipx/src/web.rs	axum: HTTP API, auth, SSE, media streaming	—
crates/ipx/src/logbuf.rs	Ring buffer behind the UI's Log view	—
crates/ipx/src/directory.rs	Asking the directory service, and sharing this server's public feeds with it	—
crates/ipx/src/metrics.rs	/metrics for Prometheus	—
crates/directory/src/lib.rs	The directory's HTTP API, its token check, the refresher	—
crates/directory/src/store.rs	The directory's database: feeds, snapshots, blocked	—
crates/directory/src/config.rs	The directory's config file: its tokens and their access	—
crates/directory/src/otel.rs	Traces for both Rust services: the exporter, traceparent in and out	—
web/index.html	The app's markup	—
web/admin.html	The admin page's markup: server settings, accounts, the log. Sent to admins only	—
web/app.css	The stylesheet both pages share	—
web/src/*.ts	The page's script, one scope split across files, type-checked by npx tsc	—
web/build.mjs	swc: strips the types into app.js/admin.js/login.js, named in the page by a hash of their contents, minifies, and writes the static site to web/dist	—
web/boot.js	In <head>: the theme from its cookie, and an admin's link, before the first frame	—
web/Caddyfile	ipx-web: serves web/dist, asks ipx before serving a page, passes /api and /media to ipx	—

The page is a separate image, ipx-web (Dockerfile.web, #203), so editing web/ needs ipx-web rebuilt, not ipx, and building it needs node and npm ci run once. ipx serves the API and the files, and no page.

A scan

1. Skip the feed unless last_checked + max(schedule, ttl) has passed (--force ignores this). Without a schedule of its own, a feed whose body has been the same for a while waits longer: a twelfth of how long it has been the same, up to four hours (#242).
2. Conditional GET with the stored ETag / Last-Modified. 304 ends it there, and so does a body identical to the last one (body_hash), from a server that ignores the validators.
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are re-derived into the database each scan, never written to config.toml. A Patreon creator link (a token, no show=) with more than one show is treated the same way, before any fetch: its shows come from Patreon's web API and each becomes a derived feed.
4. Record entries: the new ones, and those the feed now gives a different title, text, artwork, length or number, compared with what is stored for the items it lists (#141). A corrected item keeps its read state.
5. Record enclosures. enclosures.url is UNIQUE, which is the dedupe key and subsumes the original's history.dat pickle: a reaped file keeps its row so it is never fetched twice.
6. Apply the merged policy (see users.md) and mark anything rejected as skipped with a reason. What a filter skipped is judged again every scan, so a change of settings brings it back. A feed in a group takes your settings on the group for anything you have not set on it.
7. Download what is still pending, newest first, up to the per-scan cap. A .torrent body goes to the torrent path whatever its advertised type; an HTML body is a failed download — a login wall or an error page — and is deleted.

Data model

feeds       id, url, title, image, category, description, etag, last_modified,
            last_checked, ttl_mins, last_error, orphaned, group_id, managed,
            body_hash, same_since
entries     feed_id, guid, title, link, published, description, first_seen,
            image, duration, episode, season            PK (feed_id, guid)
enclosures  id, feed_id, guid, url UNIQUE, mime, length, path, state,
            bytes_done, downloaded_at, last_error
users       id, name, pass_hash, is_admin, created, last_login
sessions    token, user_id, seen
subscriptions  user_id, feed_id, keywords, auto_download, allow_explicit,
               max_new_per_check                         PK (user_id, feed_id)
entry_state    user_id, feed_id, guid, read, flagged, position
                                                         PK (user_id, feed_id, guid)

Read state is entry_state alone. entries had read, flagged and position columns from before accounts; two bugs came from queries still reading them, and they were dropped in 0.5.

Schema changes: the tables are the entities in crates/ipx/src/entity.rs, and Db::open creates whatever table or index a database is missing from them (db::create_missing), with IF NOT EXISTS. It never alters a table that exists, so a new column on one needs its own ALTER in create_missing, looked for first and run only when it is missing, as feeds.description's is, or sea-orm-migration once there are several. Db::memory() builds its database the same way, so the tests run on the schema production gets. A database from before 0.7 takes its last columns from the old migrate(), so it upgrades through a 0.7 release first.

Control socket

Newline-delimited JSON, both directions, over $XDG_RUNTIME_DIR/ipx.sock.

printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
{"ev":"feed_start","feed":"atp"}
{"ev":"progress","feed":"atp","enclosure":42,"file":"ep1.mp3","done":8192,"total":3000000}
{"ev":"download_done","feed":"atp","enclosure":42,"path":"…","bytes":3000000}
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
{"ev":"scan_done","feeds":1}

Commands — fetch (optional feed, force), reap (optional dry_run), download (enclosure), status, reload (read the catalogue and settings again from the database, which ipx add, rm and import send after changing them in a process of their own; answered with status).

Events — feed_start, feed_skip, feed_done, feed_error, progress, download_done, download_error, torrent_deferred, reaped, reap_done, scan_done, status, error. scan_done, reap_done and status are terminal: a client that asked for work stops reading there. Commands run one at a time, in the order they arrive, except status and reload: the socket answers them straight away, so the Docker healthcheck is never left waiting behind a scan or a download, and answers only the client that asked, since status would end any other client's session.

Progress carries the enclosure id, without which a UI cannot tell one download from another and ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast, so a client attached to a busy daemon also sees that daemon's other work.

Inside the process the same events go over a tokio::broadcast; commands arrive on an mpsc and are handled by a single worker, so nothing races over the same download. Shutdown is a watch channel raced inside each job — tokio::select! only races branches at the point of selection, so a long download had to be able to notice the signal itself.

HTTP API

Everything below /api needs a signed-in user; the browser gets a redirect to /login, anything else a 401. A feed's items and files (its entries, download-latest, /api/enclosures/{id} and /media/{id}) answer only someone who subscribes to the feed; anyone else gets a 404.

Route
GET /api/page-check, GET /api/login-check, GET /api/signin	for Caddy (#203): whether the visitor may have the page in X-Forwarded-Uri (204), or where to send them; signing in with ?token=
POST /api/login, POST /api/logout, GET /api/me	sign-in; /api/me also gives the version and the account's theme from before
GET /api/feeds, POST /api/feeds	your subscriptions; subscribe
PATCH /api/feeds/{id}, DELETE /api/feeds/{id}	your settings or (admin) the feed's; unsubscribe
GET /api/feeds/{id}/entries	paged, filtered, searchable, sortable (sort = kept, title, feed, type, size or published; dir = asc or desc)
GET /api/entries	the same, across every feed you subscribe to (All Subscriptions)
POST /api/feeds/{id}/read-all, POST /api/feeds/{id}/download-latest
POST /api/read-all	everything read in every feed you subscribe to (All Subscriptions)
POST /api/entries/{feed}/{guid}/flags, …/position	your read, kept, position
POST /api/enclosures/{id}/download, DELETE /api/enclosures/{id}	?force=true overrides the shared-file warning
POST /api/fetch
GET /api/opml, POST /api/opml	export your subscriptions; subscribe to every feed in an OPML
GET /api/directory/{id}	a feed's page from the directory service (#205), by the directory's id: its description and latest twenty items, sanitized here, and why its last check failed, in words; never a file or its address
GET /api/popular, GET /api/directory, POST /api/popular/{id}	the directory service's ten most subscribed and every feed it lists, A to Z, asked by ipx with its token (directory id, title, art, subscribers across every sharing server, whether it is yours and its id here, category, whether it carries audio or video; never a URL); subscribe by the directory's id, which takes the feed's address from the directory. 503 with no directory set up
DELETE /api/directory/{id}, POST /api/directory/{id}/refresh	admin-only, and the directory refuses unless this server's token is rwa (#206): remove and block a feed; check it now. GET /api/directory/{id} says can_admin when both hold
GET /api/directory-admin, DELETE /api/directory-blocked/{id}, DELETE /api/directory-instances/{name}	admin-only, rwa (#206): the servers sharing (name, feeds, subscribers) and the blocked feeds (id, title, when; never the address, often why it was blocked); unblock; drop a server's snapshot. Unblocking or dropping sends this server's snapshot again at once
GET /api/directory-status	admin-only: the name and access the directory gives this server's token, or why it refused, and how the last snapshot went
GET /api/settings, PATCH /api/settings	admin-only to write. The directory's token is written, never read back: directory_token_set says whether there is one
GET /api/users, POST /api/users, PATCH /api/users/{id}, DELETE /api/users/{id}	admin-only; the only admin cannot be demoted or removed
GET /api/events	SSE, the same broadcast the socket carries
GET /api/logs	admin-only; the ring buffer, with a sequence cursor
GET /media/{id}	the file, with Range support so seeking works

Show notes are feed-supplied HTML from an untrusted source, sanitized with ammonia server-side before they reach the page.

Testing

cargo test                # parsing, filters, retention, schedules, SQL, per-user isolation
node tests/page-smoke.js  # the page script loads and every selector it wires at load exists
node tests/native-bridge.js  # the page hands playback to a native shell
npx playwright test       # a real browser against a real daemon on fixture feeds
node tests/load/run.js    # k6: many people at once against a daemon with 1,500 generated feeds

The Rust tests cannot see a wrong selector, a handler that runs and does nothing, or a page that renders empty — which is what has actually reached users. Each Playwright case maps to a bug that did.

The suite starts its own daemon and database under /tmp/ipx-ui-test, wiped once per run. Tests share that daemon and run in order, so a test that marks something read changes what later tests see — make assertions that do not depend on earlier ones.