rays
c0f4b0bcb2
- A Popular button beside + Feed opens the popular list directly; the Add feed dialog keeps it too. - The list counts every subscriber, you included. Your own feeds stay on it, marked Subscribed, and clicking one opens it. Private feeds and feeds inside an OPML are still never listed, for anyone. - GET /api/popular rows carry `subscribed`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016HdTEWQNrzyFULijigkmMn
138 lines
7.5 KiB
Markdown
138 lines
7.5 KiB
Markdown
# How it works
|
|
|
|
One binary, `ipx`. `ipx daemon` runs three things in one process: a scheduler, a Unix-socket
|
|
control server, and the web UI. Everything else is a CLI that either does the work itself or hands
|
|
it to a running daemon.
|
|
|
|
## Modules
|
|
|
|
| File | Responsibility | What it replaced in the Python |
|
|
|---|---|---|
|
|
| `src/main.rs` | CLI, dispatch, scan loop, download policy | `iPXAgent.py` |
|
|
| `src/config.rs` | TOML load/save, `General`/`Feed`/`Web`, intervals, slugs | `iPXSettings.py`, `feeds.plist` |
|
|
| `src/db.rs` | SQLite schema, migrations, every query | `.ipxd` plists, `history.dat`, `qmcache.dat` |
|
|
| `src/feed.rs` | Conditional GET, RSS/Atom/OPML parsing | `FeedData.__getFeed/__getEntries` |
|
|
| `src/download.rs` | Streaming download, naming, type sniffing, placement | `iPXDownloader.getFile` |
|
|
| `src/torrent.rs` | librqbit session, seeding limits, stall abort | vendored BitTorrent 4.2.1 |
|
|
| `src/retention.rs` | Quota and age sweeps | `iPXQuotaManager.py` |
|
|
| `src/ipc.rs` | Event and command types, the socket server | `printMSG` on stdout |
|
|
| `src/auth.rs` | Argon2id hashing, session tokens, header names | — |
|
|
| `src/web.rs` | axum: HTTP API, auth, SSE, media streaming | — |
|
|
| `src/logbuf.rs` | Ring buffer behind the UI's Log view | — |
|
|
| `web/index.html` | The whole front end, `include_str!`d into the binary | — |
|
|
|
|
The page is compiled in, so **editing `web/index.html` needs a rebuild**.
|
|
|
|
## A scan
|
|
|
|
1. Skip the feed unless `last_checked + max(schedule, ttl)` has passed (`--force` ignores this).
|
|
2. Conditional GET with the stored `ETag` / `Last-Modified`. `304` ends it there.
|
|
3. Sniff the body: RSS, then Atom, then OPML. An OPML is a live subscription — its feeds are
|
|
re-derived into the database each scan, never written to config.toml.
|
|
4. Record entries. A changed title or description flips the item back to unread.
|
|
5. Record enclosures. `enclosures.url` is `UNIQUE`, which is the dedupe key and subsumes the
|
|
original's `history.dat` pickle: a reaped file keeps its row so it is never fetched twice.
|
|
6. Apply the merged policy (see [users.md](users.md)) and mark anything rejected as `skipped` with
|
|
a reason.
|
|
7. Download what is still pending, newest first, up to the per-scan cap. A `.torrent` body goes to
|
|
the torrent path whatever its advertised type; an HTML body is a failed download — a login wall
|
|
or an error page — and is deleted.
|
|
|
|
## Data model
|
|
|
|
```
|
|
feeds id, url, title, image, etag, last_modified, last_checked, ttl_mins,
|
|
last_error, orphaned, group_id, managed
|
|
entries feed_id, guid, title, link, published, description, first_seen,
|
|
image, duration, episode, season PK (feed_id, guid)
|
|
enclosures id, feed_id, guid, url UNIQUE, mime, length, path, state,
|
|
bytes_done, downloaded_at, last_error
|
|
users id, name, pass_hash, is_admin, created
|
|
sessions token, user_id, created, seen
|
|
subscriptions user_id, feed_id, keywords, auto_download, allow_explicit,
|
|
max_new_per_check, created PK (user_id, feed_id)
|
|
entry_state user_id, feed_id, guid, read, flagged, position
|
|
PK (user_id, feed_id, guid)
|
|
```
|
|
|
|
`entries` still has `read`, `flagged` and `position` columns from before accounts existed. They are
|
|
**dead** — the migration copied them into `entry_state` and nothing reads them now. Anything found
|
|
querying them is a bug; two were.
|
|
|
|
Schema changes: add the table or column to `SCHEMA`, and for a column also to the list in
|
|
`migrate()`, which does `PRAGMA table_info` then `ALTER TABLE ADD COLUMN`. `Db::memory()` runs the
|
|
same path as `Db::open`, so a migration-only column cannot pass tests while missing in production.
|
|
|
|
## Control socket
|
|
|
|
Newline-delimited JSON, both directions, over `$XDG_RUNTIME_DIR/ipx.sock`.
|
|
|
|
```sh
|
|
printf '{"cmd":"fetch","force":true}\n' | socat - UNIX-CONNECT:$XDG_RUNTIME_DIR/ipx.sock
|
|
{"ev":"feed_start","feed":"atp"}
|
|
{"ev":"progress","feed":"atp","enclosure":42,"file":"ep1.mp3","done":8192,"total":3000000}
|
|
{"ev":"download_done","feed":"atp","enclosure":42,"path":"…","bytes":3000000}
|
|
{"ev":"feed_done","feed":"atp","new":1,"downloaded":1,"failed":0,"torrents":0}
|
|
{"ev":"scan_done","feeds":1}
|
|
```
|
|
|
|
**Commands** — `fetch` (optional `feed`, `force`), `reap` (optional `dry_run`), `download`
|
|
(`enclosure`), `status`.
|
|
|
|
**Events** — `feed_start`, `feed_skip`, `feed_done`, `feed_error`, `progress`, `download_done`,
|
|
`download_error`, `torrent_deferred`, `reaped`, `reap_done`, `scan_done`, `status`, `error`.
|
|
`scan_done`, `reap_done` and `status` are terminal: a client that asked for work stops reading
|
|
there.
|
|
|
|
Progress carries the enclosure id, without which a UI cannot tell one download from another and
|
|
ends up animating every pending row. It is throttled to whole percents. The stream is a broadcast,
|
|
so a client attached to a busy daemon also sees that daemon's other work.
|
|
|
|
Inside the process the same events go over a `tokio::broadcast`; commands arrive on an `mpsc` and
|
|
are handled by a single worker, so nothing races over the same download. Shutdown is a `watch`
|
|
channel raced *inside* each job — `tokio::select!` only races branches at the point of selection,
|
|
so a long download had to be able to notice the signal itself.
|
|
|
|
## HTTP API
|
|
|
|
Everything below `/api` needs a signed-in user; the browser gets a redirect to `/login`, anything
|
|
else a `401`.
|
|
|
|
| Route | |
|
|
|---|---|
|
|
| `GET /` | the app |
|
|
| `GET /login`, `POST /api/login`, `POST /api/logout`, `GET /api/me` | sign-in |
|
|
| `GET /api/feeds`, `POST /api/feeds` | your subscriptions; subscribe |
|
|
| `PATCH /api/feeds/{id}`, `DELETE /api/feeds/{id}` | your settings or (admin) the feed's; unsubscribe |
|
|
| `GET /api/feeds/{id}/entries` | paged, filtered, searchable |
|
|
| `POST /api/feeds/{id}/read-all`, `POST /api/feeds/{id}/download-latest` | |
|
|
| `POST /api/entries/{feed}/{guid}/flags`, `…/position` | your read, starred, position |
|
|
| `POST /api/enclosures/{id}/download`, `DELETE /api/enclosures/{id}` | `?force=true` overrides the shared-file warning |
|
|
| `POST /api/fetch` | |
|
|
| `GET /api/opml`, `POST /api/opml` | export your subscriptions; subscribe to every feed in an OPML |
|
|
| `GET /api/popular`, `POST /api/popular/{id}` | what everyone here subscribes to, you included (id, title, art, count, whether it is yours; never a URL, never a private feed); subscribe by id |
|
|
| `GET /api/settings`, `PATCH /api/settings` | admin-only to write |
|
|
| `GET /api/users`, `POST /api/users`, `PATCH /api/users/{id}`, `DELETE /api/users/{id}` | admin-only; the only admin cannot be demoted or removed |
|
|
| `GET /api/events` | SSE, the same broadcast the socket carries |
|
|
| `GET /api/logs` | admin-only; the ring buffer, with a sequence cursor |
|
|
| `GET /media/{id}` | the file, with Range support so seeking works |
|
|
|
|
Show notes are feed-supplied HTML from an untrusted source, sanitized with `ammonia` server-side
|
|
before they reach the page.
|
|
|
|
## Testing
|
|
|
|
```sh
|
|
cargo test # parsing, filters, retention, schedules, SQL, per-user isolation
|
|
node tests/page-smoke.js # the page script loads and every selector it wires at load exists
|
|
npx playwright test # a real browser against a real daemon on fixture feeds
|
|
```
|
|
|
|
The Rust tests cannot see a wrong selector, a handler that runs and does nothing, or a page that
|
|
renders empty — which is what has actually reached users. Each Playwright case maps to a bug that
|
|
did.
|
|
|
|
The suite starts its own daemon and database under `/tmp/ipx-ui-test`, wiped once per run. Tests
|
|
share that daemon and run in order, so a test that marks something read changes what later tests
|
|
see — make assertions that do not depend on earlier ones.
|